Getting started¶
Accessing the data¶
All analysis-ready datasets are CSV files in shared_resources/data/. They are not tracked in git (too large), but the documentation, codebooks, and quality reports are.
To get the data files, contact Laurence Wilse-Samson at NYU Wagner. Many files are under their holders' terms (TEBA records, DataFirst microdata, Datastream and other licensed financial data) and some cannot be supplied at all; see Licensing and access.
Quick start (Python)¶
import pandas as pd
# Agricultural census — district-level panel
ag = pd.read_csv('shared_resources/data/ag_census/geography_panel_1948_1994.csv')
# TEBA mine recruiting — district panel
teba = pd.read_csv('shared_resources/data/teba_panel/district_recruiting_panel.csv')
# Strike incidents (the 2013 master file; for counts use strikes_clean.csv in strikes-apartheid-jpe)
strikes = pd.read_csv('shared_resources/data/strike_panel/strikes_master_csv.csv')
# Elections: division x election panel (unopposed seats flagged, votes NA)
elections = pd.read_csv('shared_resources/data/panels/electoral_division_panel.csv')
File conventions¶
All CSVs use:
- UTF-8 encoding, comma delimiters, header row
- NA for missing values (never empty strings or zero)
- Metadata columns:
_source_file,_category,_year,_converted_date
Data quality¶
A format audit in March 2026 recovered headers, trimmed wide files and removed duplicates across 1,814 CSV files; it checked file structure, not values. What has been checked against the printed sources, and what remains open, is on Provenance and checks.
Key panels¶
| Panel | File | Rows | Use |
|---|---|---|---|
| Ag census geography | ag_census/geography_panel_1948_1994.csv |
13,113 | District-level agricultural production |
| Long-run districts | ag_census/long_run_district_panel_1918_1994.csv |
15,525 | Extended 55-year panel |
| TEBA recruiting | teba_panel/district_recruiting_panel.csv |
1,674 | Mine labor supply by district |
| Strikes master | strike_panel/strikes_master_csv.csv |
2,678 | Strike incidents from Work in Progress reports, c. 1970–1991 |
| Electoral divisions | panels/electoral_division_panel.csv |
1,750 | Division × election results, 1943–1987 |
| District crosswalk | crosswalks/district_crosswalk_master.csv |
398 | Links naming schemes across datasets |
Geographic crosswalks¶
Linking datasets across different geographic coding schemes is a core challenge. See Geographic crosswalks for the district crosswalks and the electoral-division crosswalks for the 12th, 13th and 14th Delimitations.