Data migration and three-way reconciliation

A migration has exactly one load order that works: every child finds its parent on the first pass. Map it before writing a loader.

1. VideoLoad order: why every child has to find its parent on the first passPreview6 min
2. VideoMapping sixteen archive objects before writing a single loader7 min
3. ReadingWhat to migrate, what to archive, and what to leave behind6 min
4. VideoDependency graphs, and the load order they force7 min
5. VideoStaging tables, and why the target org is not the place to debug a mapping6 min
6. LabLab: sequence a twelve-object load order from a dependency map6 min

The same external-id discipline as course 1.9, now under the load volumes a real retirement produces.

1. VideoKeying every record on an external id before the first load7 min
2. VideoUpsert at scale, and the re-run that costs nothing6 min
3. ReadingComposite keys, and the record that needed more than one field to be unique7 min
4. VideoA synthetic external id, and the hand-entered rows it had to skip6 min
5. VideoIdempotency as a test, not an assumption7 min
6. LabLab: run the same load twice and prove nothing duplicated6 min

The load itself: sixteen archive objects, roughly fifty thousand records, and the batch discipline that kept every run stable.

1. VideoSixteen archive objects, fully populated, zero failures: the load itselfCompleted orders, purchase orders, invoices, inventory, stock movements and more, all sixteen objects loaded with zero failures.6 min
2. VideoRoughly fifty thousand records, and the batch size that kept every run stable7 min
3. ReadingError handling that stops a batch instead of half-loading it6 min
4. VideoValidating a completed load against its own row counts7 min
5. LabLab: load a multi-object archive and verify zero failures6 min
6. VideoWhat zero failures does and does not prove7 min

The same forensic pass as course 1.6, this time as a migration decision: what a retiring system hands you is not automatically true.

1. VideoA reported lifetime invoicing total, and the number that did not sit right6 min
2. VideoTwo mistake invoices, inherited from the source systemTwo mistake invoices and their phantom parent order, all inherited bad data from the retiring system, inflated the reported figure by most of its value.6 min
3. ReadingIsolating a phantom parent order without touching a live one7 min
4. VideoThe corrected figure, pre-tax across 2,860 invoices6 min
5. LabLab: find a phantom-data pair in a seeded migration dataset7 min
6. VideoProving no live order was ever polluted6 min

4,506 transactions from 2010 to 2026, reconciled across the old system, the new one and the accounting ledger. 97% keyed, and the residue itemised.

1. VideoReconciling across the old system, the new one and the accounting ledger7 min
2. Video4,506 transactions from 2010 to 2026, and the span that made this hard6 min
3. Reading97 percent keyed: what a keying rate actually measures6 min
4. VideoThe the residue residue, itemised for the controller instead of hidden7 min
5. VideoMatching on amount and date when no shared key exists6 min
6. LabLab: run a three-way reconciliation on a seeded three-system dataset7 min
7. VideoReporting a keying rate honestly when it is not 100 percent6 min

A total is a claim. A per-order file that either agrees or explains why not is the evidence behind the claim.

1. VideoBuilding a per-order file that either agrees or explains why not7 min
2. Video141 of 141: what it took to get every order to a stated outcome6 min
3. ReadingAn explained mismatch versus an unexplained one, and why the label matters7 min
4. VideoRounding, timing and currency: the three reasons a match looks like a mismatch6 min
5. LabLab: build a per-order reconciliation file and clear every exception7 min
6. VideoHanding the per-order file to the controller, not just the total6 min

Not everything in a retiring system deserves a place in the new one. Deciding what to leave behind is a migration decision, not a shortcut.

1. VideoDeciding what not to migrate, and saying so in writing6 min
2. VideoArchiving in place instead of carrying dead data into a live org7 min
3. ReadingWhere the archive lives, and how someone finds it in five years6 min
4. VideoThe record that looked live but had not moved in six years6 min
5. LabLab: sort a dataset into migrate, archive and discard, with reasons7 min
6. VideoThe cost of migrating something nobody will ever query again6 min

The moment two systems stop being simultaneously authoritative. Plan the rollback before the cutover, not during it.

1. VideoCutover without a window where two systems are both authoritative7 min
2. VideoA cutover checklist: what freezes, what switches, and in what order6 min
3. ReadingThe rollback plan for a cutover, written before the cutover, not during it7 min
4. VideoReconciling one more time immediately after cutover, not a week later6 min
5. LabLab: write a cutover plan for a two-system retirementThe capstone lab for the course. Plan a cutover including the freeze window, the rollback trigger and the immediate post-cutover reconciliation.38 min
6. VideoTelling the business the migration is done, and what that claim covers6 min
7. QuizCheck: data migration and reconciliation14 min