Agricultural Census¶
District-level agricultural production data from South African agricultural census publications.
Analysis-Ready Panel¶
| File | Rows | Description |
|---|---|---|
panels/ag_census_district_panel.csv |
3,971 | Employment/wages/mechanization panel (16 census years, 1949–1990); district rows only |
panels/ag_census_aggregate_rows.csv |
59 | SA and province rows, kept separately for total checks |
panels/master_district_panel.csv |
8,928 | Joined panel (ag + TEBA + population + removals + geography) |
_boone_cleaned/data_census_ag_panel.csv |
3,097 | Hand-corrected transcription (Boone's research assistants); the source of the 1961–83 rows |
geography_panel_1948_1994.csv |
13,113 | Cleaned geography panel |
long_run_district_panel_1918_1994.csv |
15,525 | Extended 55-year panel |
COMPLETE_YEAR_COVERAGE.csv |
25 | Year coverage inventory |
The panel builder (shared_resources/scripts/panel_builders/01_ag_census_panel.py) takes the 1961–83 rows from the transcription corrected by Christopher Boone's research assistants and applies the corrections ledger shared_resources/corrections/ag_census_boone_panel.csv (62 cells read from page images by an AI model; a second AI reading agrees with 60). Other years come from this project's converted CSVs, which are the oDesk transcription with a few logged corrections. Kranskop 1975 is excluded as a census reporting anomaly (see Provenance & Checks).
Year-Specific Data¶
141 CSV files organized by census year:
| Folder | Years | Files | Notes |
|---|---|---|---|
agriculture_1949_converted/ |
1949-50 | 5 | Internally checked (1-6% of checks fail) |
agriculture_1951_converted/ |
1951-52 | 6 | |
agriculture_1956_converted/ |
1956 | 2 | Internally checked (0-23% of checks fail) |
agriculture_1961_converted/ |
1961-62 | 11 | Internally checked (0-23% of checks fail) |
agriculture_1967_converted/ |
1967-68 | 3 | |
agriculture_1970_converted/ |
1970-71 | 2 | |
agriculture_1972_converted/ |
1972-73 | 16 | Internally checked (86% pass); paper variables machine-read where versions disagree |
agriculture_1974_converted/ |
1974 | 6 | Paper variables machine-read where versions disagree |
agriculture_1975_converted/ |
1975 | 6 | Paper variables machine-read where versions disagree |
agriculture_1976_converted/ |
1976 | 9 | Employment table (3.1) machine-read in full |
agriculture_1978_converted/ |
1978 | 5 | Wide format trimmed |
agriculture_1979_converted/ |
1979 | 4 | |
agriculture_1980_converted/ |
1980 | 4 | |
agriculture_1981_converted/ |
1981 | 2 | Wide format trimmed |
agriculture_1983_converted/ |
1983 | 19 | Wide format trimmed |
Post-1983 Extractions¶
8 CSV files extracted from scanned bilingual PDFs via the Claude Vision API (status: transcribed). They extend the agricultural series beyond 1983. The 1988 census covers Natal Development Region E only, and the 1990 survey is published by statistical region, so its rows do not join the district panel without a region-to-district concordance.
1988 Census of Agriculture (Natal Development Region E, 119 pages):
| File | Rows | Contents |
|---|---|---|
agriculture_1988_principal_farmers_units.csv |
353 | Principal stats, farmer demographics, unit sizes |
agriculture_1988_employment_extracted.csv |
169 | Employment by race and worker type |
agriculture_1988_employment_clean.csv |
169 | Column names corrected |
agriculture_1988_remuneration_income.csv |
215 | Wages + gross income from products |
agriculture_1988_production_animals.csv |
494 | Production volume/value + livestock |
agriculture_1988_expenditure_assets_debts.csv |
291 | Expenditure, asset values, farming debts |
1990 Agricultural Survey (95 pages):
| File | Rows | Contents |
|---|---|---|
agriculture_1990_principal_units.csv |
355 | Principal stats + unit sizes |
agriculture_1990_employment_extracted.csv |
524 | Employment by race (mixed tables — use clean version) |
agriculture_1990_employment_clean.csv |
219 | Column names corrected; employment only |
agriculture_1990_remuneration_income_expenditure.csv |
937 | Wages, income, expenditure, assets |
Pre-1949 Extractions¶
pre_1949_extracted/ — 16 markdown extractions from pre-1949 agricultural census PDFs via Gemini Vision (status: transcribed). The OCR layer of these scans is too poor for text extraction.
| Census Year | Parts | Contents |
|---|---|---|
| 1921 | 1 | Agricultural-pastoral production |
| 1923-24 | 1 | Agricultural-pastoral production |
| 1926-27 | 1 | Agricultural-pastoral production |
| 1929-30/1934 | 1 | Agricultural-pastoral production |
| 1936-37 | 8 | Full census — livestock, crops, land use by district |
| 1938-39 | 3 | Agricultural-pastoral production |
| 1945-46 | 1 | Agricultural-pastoral production |
The folder also holds two parsed CSVs (pre_1949_districts_improved.csv, pre_1949_districts_parsed.csv) built from the OCR text; they are not used in the panel.
Variables¶
Typical variables include:
- District identification (name, code, province)
- Employment: regular employees by race and gender, casual/seasonal workers
- Wages: annual wage bill by race category
- Machinery: tractors, combines, other equipment counts
- Land: area farmed, area irrigated
- Production: output by crop type
Provenance¶
- Source: South African agricultural census publications; PDFs in
raw_archives/01_production_censuses/agriculture/source_pdfs/ - 2013 transcription: oDesk freelancers (Rao's Infocube for 1953–54 and 1963–64), Excel format
- Corrected transcription: Christopher Boone's research assistants, by hand against the PDFs, 2023–26 (1961–83 sheets). This project's converted CSVs (
agriculture_YYYY_converted/) are the oDesk transcription with a small number of logged corrections, most of them copied from the hand-corrected sheets. - Conversion:
ops_admin/scripts/digitization/batch_convert.py
Corrections¶
Corrections to the panel are rows in the ledger shared_resources/corrections/ag_census_boone_panel.csv, applied by the panel builder. Its 62 rows (30 cell values and 32 fixes to loader errors) come from readings of page images by AI subagents (Claude), each citing the page. A second AI reading agrees with 60 of the 62; two Barkly West cells await a person's reading, and no person has re-read the rest. Corrections to this project's converted CSVs are logged in ag_census/CORRECTIONS_LOG.md, with originals in _originals/.
For 1972–80, a script compared the variables used in the 1974-shock paper across the 2013 original, this project's working copy and the hand-corrected version, and every cell where they differed was read from the page images; the 1976 employment table was read in full (2,904 cells). The working copy took many of its corrections from the hand-corrected version., so the three versions are not independent. The counts and the open cells are on the Provenance & Checks page.
Key Empirical Finding: Specification Sensitivity¶
The agricultural census data supports a district-level difference-in-differences analysis of the 1974 mine labor shock's effect on mechanization (combines adoption in open vs. closed recruiting districts). This result is specification-sensitive:
- Pooled OLS with province FE: +0.373 log-points (p < 0.05) — open districts adopted more combines post-1974
- TWFE with district FE: +0.017 (null) — the result disappears with district fixed effects
- Tractors placebo: +0.795 in pooled OLS (larger than combines), suggesting the pooled OLS captures pre-existing trends rather than a causal effect
The national-level patterns are robust: combines +30%, tractors +13%, Black employment -10% between 1972-1980. The wage shock was real and economically transformative — but it operated as a general equilibrium effect, not a differential one between open and closed districts. The district-level combines DiD is fragile (null under two-way fixed effects; the tractor placebo fails), so the lead empirical finding is the Eiselen electoral divergence. See Data Appendix D.0-D.3.