Provenance & Checks¶
Every value in the panels comes from a printed page: a census volume, a Government Gazette, a delimitation report, or a TEBA/NRC memorandum. This page sets out how each value got from the page into the panels, who did each step, how it was checked, and what remains open. Where the record does not say who did something, the page says so.
From page to panel¶
Source scans (raw_archives/, read-only)
→ 2012–13 contract transcription (oDesk freelancers; Rao's Infocube)
→ agricultural census 1961–83: hand corrections by Christopher Boone's research assistants (2023–26)
→ corrections ledgers (shared_resources/corrections/), applied by the panel builders
→ panels (shared_resources/data/panels/)
Source scans. The scanned PDFs in shared_resources/raw_archives/ are never edited. When two transcriptions disagree, the printed page decides.
Contract transcription, 2012–13. Laurence Wilse-Samson commissioned two channels. Rao's Infocube, a data-entry firm in India engaged through Columbia University, delivered about 1,100 pages in March–April 2013: manufacturing and mining tables, Principal Statistics on a Regional Basis, the agricultural census of 1953–54 and 1963–64, election returns, and the strike tables from Work in Progress. It said every page was typed twice, but no record of its checks survives. oDesk freelancers transcribed the agricultural census from 1949–50 on, in May–June 2013, with no evidence of double entry. Neither channel left an error report.
Contract transcriptions contain errors
The 2013 spreadsheets have digit misreadings (most often 3↔5), shifted rows, copied total rows, totals typed as formulas, and blank cells. No value should be used on the strength of the contract transcription alone.
Hand corrections of the agricultural census. Christopher Boone's research assistants (Daniel Rinner from 2023, Li Sutherland in 2024) corrected the oDesk spreadsheets for 1961–83 by hand against the PDFs. Boone's notes record who checked which table and give error rates before correction, up to 36–42% for some 1976 tables, but do not list every changed cell. The corrected sheets (ag_census/_boone_cleaned/) supply the 1961–83 rows of the panel. The project's own converted CSVs are the oDesk transcription with a few logged fixes, most copied from the corrected sheets, so they are not an independent transcription.
Corrections ledgers. Every correction is a row in a ledger in shared_resources/corrections/, giving the old and new value, the page evidence, the method, who checked it, the date and the status. The builders apply the rows and refuse any row whose old value does not match the file. Nothing is edited by hand.
| Ledger | Rows | Corrects |
|---|---|---|
ag_census_boone_panel.csv |
68 (65 applied) | Agricultural panel cells read from page images (see below) |
district_crosswalk_master.csv |
8 | Wrong pre-1949 district name matches |
electoral_division_geocoding.csv, …_additions.csv |
23 | Divisions geocoded to a namesake town, or missing |
teba_panel_coordinates.csv |
4 | Two namesake district centroids |
forced_removals.csv |
122 | Destination districts of Surplus People Project entries |
manpower_survey.csv |
16 | Manpower Survey sector-years held out or flagged at survey breaks |
ag_census_2013_tables.csv |
71 (proposed) | Cells in five 2013 agricultural spreadsheets outside the panel, read from the page images by AI |
mining_census.csv |
3 (proposed) | Two 1966 ore-milled cells; 1966 gold ounces flagged as unreliable |
The 1977 election returns were re-transcribed in full from the Government Gazette rather than corrected (see Elections). 99_validate_panel.py runs checks on the built panels.
Status vocabulary¶
| Status | Meaning |
|---|---|
| Transcribed | Typed or extracted from the source, with no systematic check |
| Internally checked | Passes arithmetic checks within the source: rows add to totals, districts sum to printed regional and national totals |
| Machine-read against the page | Compared with page images by an AI model (Claude), with no person involved |
| Checked against printed source | Compared with the printed page by a person |
| Reproduced in analysis | Feeds a result that a script regenerates from the panel. This says nothing about accuracy |
Internal checks catch most digit errors, but not two values exchanged between rows. A machine reading can misread a degraded digit in the same way a typist did.
Agricultural census, 1972–80¶
This work was done in October 2026 by AI agents (Claude); the scripts and page renders are in ops_admin/data_program_2026_10/.
For each variable and year that the 1974-shock paper uses, a script compared every district cell across the 2013 original, the project's working copy and the hand-corrected sheet. All three agree in 17,350 cells; in 650 one or both corrected versions changed the original. The versions are not independent, so their agreement is not evidence of accuracy. The comparison also found two loader errors: the 1980 Cape and Wynberg wage rows, and the 1975 Heidelberg (Cape) and Heidelberg (Transvaal) rows, had been exchanged.
Every cell where the versions disagreed was then read from page images, and the 1976 employment table (Table 3.1) was read in full: 2,904 cells. The readings make up the first 62 ledger rows, 30 of them cell values and 32 loader fixes; 61 change the panel, and one was later set aside on a person's reading (below). With those changes, every district row of 1976 Table 3.1 adds to its total. Every column matches the printed national total except domestic employment, which is 20 short after the readings described below.
Second reading. A second AI reading of all 62 rows, made in a separate session and not blind, agrees with 60 of them. The other two are the 1976 casual-wage cells for Barkly West. I read that row on the page as 1,609 (whites) and 224,303 (total), which do not add: the parts sum to 224,503. The panel keeps both figures as printed, so the row is 200 off; the total agrees with the printed national total. Apart from that row, no person has re-read any of the 62 rows.
Cells read by a person. I read the 1976 casual-wage rows for New Hanover and Mtonjaneni and five 1976 domestic-employment cells (Middelburg (Cape), Albany, Indwe, Mooi River, Lydenburg) on the page in October 2026. For New Hanover the scan is unclear between 3 and 5 in the leading digit of the total and of black wages; I have kept Boone's 320,832 and 315,832, which add up. Mtonjaneni's total and black casual wages are 226,045, not 224,045 (the last two digits are from the AI reading). Domestic employment is 205, not 203, in Indwe and 561, not 361, in Lydenburg; the other three cells read as transcribed. Four further ledger rows change the panel; two proposed changes for New Hanover were not applied.
Status. The 1972–80 variables used in the paper are machine-read against the page wherever the versions disagreed, and in full for 1976 Table 3.1. The remaining cells, and the other years from 1961 to 1983, rest on the hand corrections and on internal checks. Before 1978, province totals were checked against district sums by script; the failure rate runs from 1–6% (1949) to about 23% for some tables in 1956 and 1961.
Flags in 1974, 1975 and 1979. Row checks on the 2013 transcription (grand total against regular, casual and domestic; each total against its race columns) fail in 40 district rows for 1974, 42 for 1975 and 45 for 1979 (and in three printed total rows). In the panel, every district row from 1972 to 1980 passes these checks. Every cell in those district rows where the panel differs from the 2013 value has been read from the page image by an AI model, and each reading agrees with the panel (ops_admin/data_program_2026_10/adjudication/flag_disposition_1974_75_79.csv).
Tables outside the panel. In five 2013 spreadsheets (1972–73 Part 14; 1983 Parts 16, 17, 30 and 33) most region totals in the 1983 files were typed as formulas over the district rows, so district-sum checks on them could not fail; the 1972–73 totals had not been checked against the page. The 425 printed total rows in these tables were read from the scans by an AI model (one 1972–73 cell, region 38 electricity, could not be read with confidence), and the district rows were checked against them. 71 typed cells differ from the page; they are proposed rows in ag_census_2013_tables.csv, not yet applied. None of these tables (equipment purchases, horticulture, livestock, sales, farm debt) feeds the panel.
Other datasets¶
| Dataset | Status | Basis |
|---|---|---|
| Election returns, 1977 | Machine-read; internally checked | Transcribed by AI from Government Gazette 5851. Contested returns add up, apart from six Gazette misprints |
| Election returns, other years | Transcribed | Rao's Infocube (2013); unopposed seats coded |
| Electoral ↔ district crosswalks | See Geographic Crosswalks | Map reading by computer vision, AI readings of the commission reports, and one research assistant's map check |
| Mining census | National totals match an independent series | Rao's Infocube. 19 of 66 files fail internal checks; 1966 gold ounces are unreliable |
| Manufacturing census | Internally checked; failures open | Rao's Infocube. 14 of 23 files fail internal checks, probably because of aggregate rows |
| Manpower Survey | Internally checked; benchmarked | A copy of the data tapes from DataFirst (2013), benchmarked against Chamber of Mines and manufacturing census totals (Manpower Survey) |
| Strike panel | Transcribed | Rao's Infocube typed the first four columns. Research assistants added the rest in 2013–14 (Strike Panel) |
| Forced removals | Transcribed; destinations audited | Partly typed by hand, partly extracted by AI. Destinations were audited by AI in October 2026 |
| AI extractions (TEBA UJ books, NRC wages, 1988/1990 and pre-1949 agriculture, wage agreements, rural property, public sector employment) | Transcribed | Extracted by Claude or Gemini. None has been page-checked yet |
Open items¶
- 1976 domestic employment. The column is 20 short of the printed national total. The five cells that could not be read as 3 or 5 have now been read by a person (above); the remaining gap has not been traced.
- 1976 casual wages, national total. The district rows fall about R640,000 short of the printed national total for casual cash wages (R54,170,886), after the corrections above, and by about the same in the black column. Part of this may be districts missing from the transcription; it has not been traced.
- Unchecked columns. From 1972 to 1980, total employment adds to its parts in every district row of the panel, but the record does not say whether the corrected sheets took those totals from the page or computed them. The ratio columns have not been checked.
- Manpower Survey printed reports. DataFirst holds scans of the printed reports: a report or a summary of results for every survey from No. 6 to No. 16, and one report a year for 1988–94. They are in the "Related materials" of its Manpower Survey 1965–1994 entry (catalogue 315) and need a free login. They have not yet been used to check the tape file year by year.
Known limits¶
- Kranskop, 1975. The page prints 15,696 regular Black farm workers, about six times the counts for 1974 and 1976. The Natal province total, as transcribed, is larger than the sum of the other Natal districts by exactly the Kranskop row, so the printed total seems to include the row. The figure is therefore the census's own, not a transcription error, but having looked at the page I think it is an error in the census itself. It is excluded from the panel.
- 1988 and 1990 agriculture. The 1988 census covers only one development region (Natal, Region E). The 1990 survey is published by statistical region, so it does not join the district panel without a region-to-district concordance.
- 2013 Stata importers. Five of the 2013 importers (1976 Parts 1, 2, 5 and 6; 1983 Part 11) drop the first row of the table, which is the Cape magisterial district. Two others drop a heading or units row, and the 1970–71 importer drops only total rows. Legacy
.dtafiles built with those five lack the Cape district. The current loaders keep it.
Provenance index¶
PROVENANCE_INDEX.csv (in shared_resources/data/docs/inventories/) links 1,808 CSVs to a source publication and digitization method, 277 of them to the file and the rest only to the domain. CSVs created since March 2026 are not in it.