Python runs the modern data stack. It powers the ingestion jobs, the transformation logic, and nearly every AI and machine learning workload in production today. GitHub's 2025 Octoverse report found Python now powers nearly half of all new AI repositories.
But the moment data enters a Python script, lineage goes dark. You can see what went in and what came out. What happened in between is anyone's guess. That blind spot is where trust breaks down, and it is why so many AI initiatives stall before they reach production
What’s new: Python lineage
Collibra now automatically extracts data lineage from Python scripts, so the logic living inside your code shows up in your lineage graph alongside everything else.
This works in two complementary ways. Run-time lineage captures what is actually executed, pulled directly from the platforms where your Python runs, including Snowflake, Databricks and Google Knowledge Catalog or streamed in through OpenLineage plug-ins. Design lineage goes a step further. Collibra reads the script itself and uses AI to interpret the transformation logic, so you get lineage even for code that has not run yet, code that runs outside a supported platform, or code that never touches a warehouse at all.
Both feed the same graph. The result is end-to-end lineage that no longer stops at the edge of a Python file, covering the full path from source system through transformation to the AI models and outputs that depend on it.
How lineage from Python scripts helps
Ask most data teams to produce end-to-end lineage for a single AI model and you'll get a diagram that stops halfway. Warehouse SQL is covered. The BI layer is covered. Then there's a Python script somewhere in the middle that reads three tables, joins them, engineers a dozen features and writes a training set, and nobody outside the team that wrote it knows what happens inside. Every gap like that breaks the chain. Impact analysis gets fuzzy, audits get manual and the model at the end of the pipeline can't be explained back to its sources.
Problems it solves
- Lineage that stops at the script. Python transformations show up as a black box between two known points, leaving gaps in your graph exactly where the most complex logic lives.
- No traceability for AI training data. Regulators and internal risk teams want to know which source columns fed a model. Design lineage from Python scripts gives you that path, column by column.
- Impact analysis that misses the Python layer. Change an upstream table and you find out what broke when the notebook fails, not before.
- Manual documentation that goes stale immediately. Teams have been hand-mapping Python transformations into spreadsheets and diagrams. That work is obsolete the moment someone merges a pull request.
- Lineage that only exists after the fact. Waiting for a job to run means lineage always trails development. CLI-based extraction from a repo lets you generate it at commit time instead.
How the new Python lineage works
Collibra approaches Python from both ends: what the code did, and what the code says.
Run-time lineage comes from execution. When a Python script runs on Snowflake, Databricks or Google Knowledge Catalog, Collibra's integrations with those platforms pick up the operations it performed and turn them into lineage. For anything running elsewhere, OpenLineage plug-ins emit the same information from the orchestration layer. This is the most accurate picture of what happened, but it depends on the job running somewhere Collibra can observe.
Design lineage comes from the script. Collibra's AI-enabled extraction parses the Python source, identifies reads and writes, follows assignments and transformations through the code, and resolves the column-level relationships between input and output datasets. It handles the messy reality of production Python: pandas operations, Spark DataFrames, SQL embedded in strings, multi-step feature engineering. Because it works on the code rather than the execution, it covers scripts that run on platforms we don't integrate with, scripts that run on a schedule you don't want to wait for, and scripts that haven't shipped yet.
Technical lineage diagram showing an end-to-end Python ML pipeline, from eight source tables through raw staging and feature engineering to published model performance metrics.
Shift left with the CLI. The Collibra CLI runs the same Python processing against a repository. Point it at a GitHub repo, wire it into CI, and lineage gets regenerated as part of your build. Developers see the downstream impact of a change in the pull request rather than in a governance review two sprints later. Lineage stops being a periodic catalog refresh and becomes part of how the code ships.
Where it lands in the platform. Extracted Python lineage doesn't sit in a separate view. It stitches into the same technical lineage graph as your SQL, ETL and BI lineage, so a single trace runs from a source system column, through a Python transformation, into a feature table, into a model, and out to a report. From there it connects to the rest of Collibra: data quality results on the assets in the path, ownership and stewardship, policies, and the AI use case registry in AI Governance.
Why you should be excited
- AI and ML engineers can trace every model input back to its original source, feature by feature, and show exactly how training data was constructed whenever someone asks. They also gain a clear view of which upstream changes will affect model performance, so problems get caught before they cause drift.
- Data engineers no longer have to hand-document pipelines, since lineage generates itself from the code they already wrote. They can run impact analysis before changing a source column rather than after someone's dashboard breaks, debug faster by tracing a bad value back through every transformation that touched it, and get up to speed on unfamiliar pipelines by reading the graph instead of reverse-engineering the script.
- Data architects get a complete view of how data actually moves, with no gaps where Python happens to be involved. That means migrations and platform changes can be planned against real dependency maps rather than assumptions, and redundant pipelines or duplicated transformation logic become easy to spot across teams.
Use cases
Here's what that looks like in practice:
- Proving where your training data came from. When a model comes up for review, someone has to account for which source fields fed it and what was done to them along the way. Feature engineering usually lives in Python, which is exactly the stretch that has historically gone undocumented. Design lineage extracted from those scripts puts the full path from source columns to features to model in the graph, so the answer is already there when the question gets asked.
- Lineage as part of the pull request. Running the Collibra CLI in CI means every merge regenerates lineage for the scripts that changed. Governance stops chasing the code, and the catalog reflects what is actually in production this week rather than what was true at the last scan.
- Impact analysis before a breaking change. Deprecating or altering a column has always been a guessing game once the dependency chain leaves the warehouse. With Python lineage in place, the graph shows the scripts that consume it, the feature tables they build, the models trained on those tables, and the reports downstream of all of it. Teams know the blast radius and who to notify before they ship.
Key takeaways
Lineage is only useful when it is complete. A graph with a Python-shaped hole in the middle cannot support the impact analysis, the trust or the AI explainability that organizations need right now. By combining run-time lineage from the platforms where code executes with AI-extracted design lineage from the code itself, Collibra closes that gap without asking engineers to change how they work. Everything lands in one connected graph alongside your SQL, BI and AI assets, governed by the same policies and enriched by the same quality signals. One place to see how data moves, no matter what language moved it.
Where to learn more about lineage from Python scripts
See how run-time and design lineage close the gap in your own pipelines. Sign up for our preview and be among the first to try it: https://preview.collibra.com/key/pythonlineage
- Meredith BaileyMeredith BaileyProduct Marketing DirectorCollibra
Meredith Bailey
Meredith Bailey
Product Marketing Director
Collibra
- Přemysl VitovecPřemysl VitovecSoftware Engineering ManagerCollibra
Přemysl Vitovec
Přemysl Vitovec
Software Engineering Manager
Collibra
- Vojtěch TalášekVojtěch TalášekStaff Product ManagerCollibra
Vojtěch Talášek
Vojtěch Talášek
Staff Product Manager
Collibra











