Data Catalog¶
All datasets live in shared_resources/data/ as UTF-8 CSV files.
District-Level Data¶
These datasets can be joined on magisterial district (279 districts, stable ~1960-1990).
| Dataset | Years | Unit | Key Variables | Page |
|---|---|---|---|---|
| Agricultural Census | 1949-1990 | District × year | Employment by race, wages, mechanization | Details |
| TEBA Mine Recruiting | 1953-1975 | District × year | Mine recruit counts by origin | Details |
| Population | 1904-1996 | District × census year | Population by race, urban/rural | Details |
| Forced Removals | 1960s-1980s | District (cross-section) | Removal events, population moved | Details |
| Elections | 1938-1992 | Constituency × year | Votes by party, turnout | Details |
National & Sectoral Data¶
Not at district level — different units of observation.
| Dataset | Years | Unit | Key Variables | Page |
|---|---|---|---|---|
| Strike Panel | c. 1970-1991 | Incident | Company, union, workers, outcome | Details |
| Manpower Survey | 1965-1994 | Sector-occupation-year | Employment by race and gender (165K rows) | Details |
| Mining Census | 1964-1987 | Mine or mineral type | Gold production, costs, profits | Details |
| Manufacturing Census | 1948-1985 | Industrial region | Output, employment by industry | Details |
| Industrial Wages | 1973-1988 | Industry-occupation-area | Gazetted minimum wage rates (825 files) | Details |
Financial & Macro¶
| Dataset | Years | Key Variables | Page |
|---|---|---|---|
| Sanctions & Macro | 1968-2015 | Gold price, exchange rate, JSE indices | Details |
| Share Prices & Boards | 1973-2015 | JSE market values, director networks | Details |
| Input-Output Tables | 1967-1993 | Sector flow matrices (48 files) | Details |
Supplementary¶
| Dataset | Description | Page |
|---|---|---|
| Geography Reference | State of emergency, labour law dates, ag suitability | Details |
| Rural Property | Farm land transfers 1939-1960 (4,711 rows) | Details |
| CPS 1980 | Household survey microdata (84K records) | Details |
| Public Sector | Government employment 1920-1980 (44 files) | Details |
Total: ~1,900 CSV files across 24 categories.
Analysis-Ready Panels¶
The panel builder pipeline (shared_resources/scripts/panel_builders/) builds these panels; 99_validate_panel.py checks them.
| Panel | Rows | Coverage | Notes |
|---|---|---|---|
| Master District | 8,928 | 279 districts x 32 years | Joins ag census, TEBA, population, removals, geography |
| Ag Census Employment | 3,971 | 16 census years (1949-1990) | Hand-corrected transcription (1961-1983) plus a ledger of AI page readings; project CSVs for other years |
| Manufacturing | 2,054 | 7 years (1963-1985) | 275 districts, 33 industry categories |
| Mining (mine-level) | 696 | 17 years (1964-1987) | Gold mine production, costs, profits |
| Electoral | 1,750 | 10 elections (1943-1987) | One row per division per election; geocoded |
| Strikes | 3,956 | 1960-1991 | 51% geocoded to magisterial districts |
| TEBA Station | ~400 | 23 years (1953-1975) | Station-level recruiting volumes |
| Industry | 1,262 | 1965-1994 | Manpower survey + mining mine-year |
Output location: shared_resources/data/panels/
Research using the archive¶
The book manuscript, The Cost of Apartheid (Wilse-Samson), and the 1974-shock paper (Boone and Wilse-Samson) are built on these datasets. Both are listed on the home page; the paper's results and replication package are described under Replication.
Data Quality¶
Status of the production census data (terms as defined on the Provenance & Checks page):
- Agricultural census: internal-consistency and cross-year checks for all years; hand corrections for 1961–83 by Boone's research assistants; for the 1972–80 variables used in the 1974-shock paper, a comparison of the versions with every disagreement read from page images by an AI model (a second AI reading agrees with 60 of the 62 changed cells; no person has re-read them).
- Mining census: National totals agree with Financial Statistics (a separate publication); 19 of 66 files fail internal checks and are open.
- Manufacturing census: TABLE 1 (industry by district) passes its internal checks best. One national total in TABLE 1.2 is wrong.
- Corrections are ledger rows in
shared_resources/corrections/, applied by the builders; corrections to converted CSVs are logged with backups in_originals/subdirectories.
See Provenance & Checks for error rates, correction logs and how each dataset was checked.