Data & lineage
Data lineage for research claims: from vendor drop to client sentence
Lineage is the receipt for intellectual work. Without it, ‘the model said so’ collapses into politics—whoever speaks last or loudest wins. With it, you replay the same inputs and either reproduce the claim or isolate the delta in minutes. This note treats lineage as an engineering contract: what identifiers must exist, how handoffs append without stripping parents, and how to triage when two honest teams see different numbers.
The question that ends bad meetings
‘Which pull produced this?’ is the grown-up version of ‘who moved my cheese?’ If the room goes quiet, you do not have lineage—you have confidence that scales with seniority.
The productive follow-up is not blame; it is a replay recipe: start URL or file id, parameter block, expected headline figure. If you cannot write that recipe on one screen, you are not ready for external challenge.
Mental model: manifest as receipt
Think of every material output as carrying a folded receipt: what was bought (dataset), when (pull timestamp), how paid for (transform chain), and who ran the transaction (runner id). Consumers should unfold the receipt without calling the author.
Receipts can be compact—JSON in a footer, a stable id that resolves server-side—but they must be machine-checkable, not prose in a footnote.
Minimum viable identifiers
Per material series: vendor id, dataset id, as-of timestamp, corporate action rev if equity, FX policy pointer, and transform chain version (even if that chain is ‘none’).
Hash stable inputs after normalization so two analysts cannot silently fork semantics with the same ticker.
If your organization cannot agree on ‘as-of’ versus ‘received-at’ versus ‘displayed-at’, stop shipping tables until the glossary is one page and signed.
From spreadsheet to service
Excel lineage usually lives in a name and a prayer. Service-backed routes can stamp manifests automatically—but only if engineers treat manifests as part of the API contract, not a logging afterthought.
The migration pattern that works: pick one high-friction grid, wrap it behind a route that logs manifests, then starve the forked spreadsheet of legitimacy—not by policy edict, by making the logged path faster.
Corporate actions and restatements
Splits and spins rewrite history. Your lineage must flag restated windows and point at the vendor announcement used to choose the restatement version—otherwise replay reproduces fiction.
Document how you handle ‘point-in-time’ versus ‘fully adjusted’ series in one place; mix them in the same workbook and you will relitigate every earnings season.
Client deliverables
Exports that leave the building should carry a compact provenance footer: not secrets—just enough for a counterparty to request the same vintage under NDA without guessing.
If lawyers push back on ‘too technical’ footers, negotiate a redacted manifest id that resolves only to entitled readers—half-metadata beats zero-metadata.
Legal hold and deletion
Automated purges must respect legal hold tags; lineage systems fail loudest when deletion pipelines ignore matter ids.
Engineers should treat hold objects like foreign keys: if a hold applies, purges must no-op or quarantine, never silently drop.
Sampling for audit
Pick random trades or memos monthly; trace end-to-end under time pressure. Failures become engineering tickets, not personnel drama.
Rotate reviewers so the same hero cannot smooth gaps with memory—fresh eyes catch stale assumptions.
Score each drill: time-to-first-manifest, percent of chain machine-readable, count of human interventions. Trend the score, not anecdotes.
Cross-team lineage contracts
Research hands to risk, risk to finance, finance to client reporting—each hop must append identity and timestamp without stripping parent manifest ids. Ambiguous ‘final’ without pointer is where firms lose multi-week investigations.
Write the contract as a one-page ‘append rules’ doc: allowed transformations, forbidden fields to drop, escalation when someone forwards a PNG.
Inter-book reconciliation as lineage QA
When two books disagree, align manifests before arguing models. Mismatch often traces to pulls minutes apart or parallel spreadsheets—both fixable once visible.
Teach risk and research to paste manifest hashes into a shared compare view before opening P/L diffs—you will kill a class of useless meetings.
Difficulty classes (know what you are solving)
Trivial: deterministic replay from logged route—should succeed in one sitting.
Moderate: legacy workbook with named tabs but missing version—expect a half-day archaeology pass.
Forensic: ad-hoc CSV + email chain + who-knows-which vendor revision—schedule explicit time or refuse external reliance.
Mislabeling difficulty is how teams burn credibility: call forensic work forensic.
Failure modes
Silent re-pulls that change numbers without bumping revision. Ad-hoc CSVs that bypass ingestion. ‘Helpful’ exports that strip metadata. All three should trigger alarms, not shrugs.
Another quiet killer: dual ‘official’ spreadsheets with unclear precedence—track them like incidents until one dies.
Public desk routes referenced above
Same URL discipline applies after you authenticate—bookmarkable paths, no tenant data on these pages.
- Public calculator hub GET-shareable patterns you can mirror for every material desk run; use as the pattern for replay links.
- Product atlas Route-level map so reviewers can open the exact module that produced a figure.
- Operating rhythm Template language for stages, owners, and completion rules in a mandatory morning loop.
- Guides Pair public guides with internal SOPs so training points at one narrative spine.
- Service status Public narrative around uptime; align its claims with what engineering will defend in postmortems.
- Readiness JSON Coarse liveness contract—use it to rehearse what is safe to expose anonymously vs behind auth.
Operator toolkits
Checklists and scripts you can lift into runbooks, vendor RFPs, or board appendices.
Operational data governance note; align to your counsel and regulatory regime.