A data catalog answers three questions someone on a growing data team asks daily: what tables exist, what do they mean, and can I trust them. Below a certain size, those questions get answered in Slack threads and tribal knowledge. Past that size, the same questions get asked and re-answered constantly, and the person who knows the answer becomes a bottleneck. A catalog is the attempt to make that knowledge queryable instead of remembered.
What a catalog actually does
At the core, a data catalog indexes metadata — table and column names, types, row counts, owners, descriptions — and makes it searchable. On top of that base, most catalogs add a business glossary (mapping technical column names to the terms the business actually uses, so "cust_lifetime_val" is discoverable by someone searching "LTV"), lineage (which tables and jobs produced a given table, and what consumes it downstream), and usage signals like query frequency or last-accessed time that help people judge whether a table is still alive.
The common thread across tools in this space — DataHub, Amundsen, Alation, Collibra, and cloud-native options like AWS Glue Data Catalog — is the same core loop: crawl or ingest metadata from your warehouses and pipelines, index it, and give people a search interface plus a place to add human context (descriptions, tags, ownership) that the source systems don't carry on their own. They differ in how open source vs. commercial they are, how deep their lineage and governance features go, and how much setup they demand, but the underlying job is the same.
Why catalogs matter as platforms scale
At a handful of tables, nobody needs a catalog — you just ask the person who built the pipeline. That doesn't scale. Once you have dozens of engineers, hundreds of tables, and pipelines feeding each other across teams, "ask the person who knows" breaks down in two ways: the person leaves or forgets, and the same question gets asked by five different people who each independently arrive at a guess. Multiple guesses about what a column means is how you get a dashboard and a finance report disagreeing about the same number, discovered in a meeting instead of a code review.
The deeper issue a catalog addresses is trust. Data consumers — analysts, other engineers, ML practitioners — need to know not just what a table contains but whether it's safe to build on: is it still actively maintained, is it the canonical source or a deprecated copy, has anyone flagged quality issues with it. Without a catalog, that trust signal lives nowhere and gets re-derived from scratch by every new consumer.
Lineage: table-level vs. column-level
Lineage tracking comes in two granularities that solve different problems. Table-level lineage answers "what feeds this table, and what does it feed" — useful for impact analysis when you're about to change a schema or deprecate a source. Column-level lineage goes further and tracks a specific field through every transformation, so you can answer "does this specific column in the gold table trace back to that raw source field, and through which transformations."
Table-level lineage can often be inferred from query logs or job dependency graphs with reasonable accuracy. Column-level lineage generally requires parsing SQL or transformation code to trace individual field references, which is a much harder problem and where most catalog tools are noticeably less mature. Don't assume column-level lineage will be complete or trustworthy without a track record of checking it against reality.
Ownership and stewardship models
Metadata without an owner rots. A table description that made sense two schema changes ago is worse than no description, because it's actively misleading. The organizational piece that makes a catalog stay accurate is assigning ownership — usually at the table or domain level — to a specific team or individual who's expected to keep descriptions and tags current as the underlying data changes.
Two ownership models show up in practice: centralized stewardship, where a data platform or governance team owns cataloging as a job, and federated (or "data mesh"-style) stewardship, where each domain team owns the metadata for the tables they produce. Federated models scale better with organization size but only work if updating the catalog is built into each team's normal workflow — schema change PRs, deployment checklists — rather than being a separate chore layered on top.
How catalog adoption fails
The most common failure mode is a one-time population followed by abandonment: someone runs the crawler, gets an initial index of every table, declares victory, and nobody touches it again. Six months later half the tables listed no longer exist, the descriptions are wrong, and the catalog is trusted less than searching Slack — so people stop opening it, which removes the only feedback loop that would have caught the drift.
The fix is less about picking a better tool and more about making catalog updates part of a workflow people already do, rather than an extra step they have to remember. A few patterns that hold up:
- Auto-sync metadata from the warehouse or orchestrator on a schedule, so schema and freshness data never goes stale even if humans do nothing.
- Require a table description and owner as part of the PR or migration that creates a new production table, not as a follow-up task.
- Surface the catalog inside tools people already use — a link from the BI tool, a lookup in the SQL editor — instead of a separate destination they have to remember to visit.
If the catalog isn't part of anyone's daily workflow, the crawler still runs and the metadata still exists, but nobody notices when it's wrong. That's worse than not having a catalog, because the illusion of documentation discourages the manual verification people would otherwise do.
Picking a starting point
For a small team, the pragmatic starting point is often whatever ships closest to your existing stack — AWS Glue Data Catalog if you're already on Glue and Athena, or a lightweight open-source deploy like Amundsen or DataHub if you want something self-hosted and are willing to run it. Commercial platforms like Alation or Collibra earn their cost at larger organizations that need formal governance workflows — approval chains, compliance reporting, access request tracking — on top of discovery. Whichever you pick, budget for the ongoing process of keeping it fed, not just the initial rollout; that's the part that determines whether it's still useful a year later.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.