One of the themes I keep coming back to is how much of the tedious, rigorous work that we should have been doing all along but didn't always have the incentive or the budget for is now necessary for AI to work properly. Quotas, documentation, lineage, authority, sensitivity labelling: they might be nice to have for humans who will be responsible enough to probably figure things out if they're not there, but they're imperative if you want what you spend on the flashier bits of AI to be worthwhile.
Data for AI normally gets annotated (manually or automatically) when you make a training dataset; while I was researching this, I wondered how many of the AI pilots that failed in production were failing because that extra information just wasn't there in the data they get fed in production. Here's another thing we should shift left, automate as much as possible and turn into a virtuous cycle.
I actually got the idea for this piece when I asked @ironyuppie.com to explain to @sbisson.com what @expanso.io actually does; the idea of annotating data as you create or capture it rather than trying to fix it later by cleaning it really struck me, because that's where you have the most context
so I started digging into how useful it is to annotate data with gathered or generated metadata as soon as possible and what tools there are to do that, like data contracts in DataHub, SurrealDB and the proposed Makoto SLSA-style DBOM
Alas, not everyone I talked to for this piece is on Bluesky; the founder of SurrealDB talked to me about how they attach so much extra information to unstructured data that it gets some structure. there's one school of though that says any document is at least semi-structured and XML creator Jean Paoli told me about the DGML spec Docugami is open sourcing, which lets you tag not just a document but an object inside a document with metadata. And Ulik Hansen gave me some great examples of where this is already routine, because it's so useful.
I love talking to @ironyuppie.com because he will straight out say your bronze tier is toxic waste; even as data literate a company as LinkedIn found their data was a swamp when they had to get ready for GDPR, which is why DataHub was created
that happens when it's too hard to annotate data properly, or there are no incentives to do it - or more likely that you're not surfacing the right incentives, because doing it earlier is always going to be more efficient, and it's what will make answers from AI agents much more accurate
If you find this piece interesting, I've written about the dangers of overcleaning data and removing context before.
I've also looked at synthetic data: often useful, but sometimes just too clean and sometimes not clean at all.
data cleaning
shift left
context
metadata governance
annotation and labelling
data provenance and lineage
data quality
IoT digital twins
LinkedIn
DataHub
DGML
context engineering
fine tuning