Back to all work

SAM Pipeline

An automated Python pipeline that builds Social Accounting Matrices. Each one is a balanced table that records every payment in a country’s economy for a year. This is my current research in Northeastern’s Financial Stress Lab, supported by a PEAK Summit Fellowship.

PEAK Summit Fellowship · Northeastern Undergraduate Research and Fellowships

“A Reproducible Pipeline for Multi-Country Social Accounting Matrix Construction”

Mentor: Prof. Mikhail Oet · Business and Social Sciences

Role: Undergraduate researcher, PEAK Summit Fellow
Advisors: Prof. Mikhail Oet (PI), Prof. Paul Chiou
Dates: Phase 2, Jul 2026–present
Stack: Python, scipy, cvxpy, pandas, pytest

KazakhstanKyrgyzstanUzbekistanTajikistanTurkmenistanAfghanistanIranUnited StatesMongoliaSri Lanka

Each line is trade between two of the 10 economies built so far. The pulse shows what linking the tables is for: following a shock in one country, here Kazakhstan, out to its trading partners.

The goal is to link the country tables into one model. The map centers on Central Asia and its neighbors Iran and Afghanistan, with Mongolia, Sri Lanka and the United States just off the edge.

Summary

A Social Accounting Matrix (SAM) is a square table of every payment made in a country’s economy in one year. Each account gets one row for the money it receives and one column for the money it spends. An account can be an industry, households, the government or the rest of the world. Every row total has to equal its column total, since every dollar received is a dollar spent or saved.

My pipeline builds these tables reproducibly from global input-output databases, which record what each industry buys from every other. Then it balances each table, checks it, and records where every number came from.

18 balanced SAMs so far. 17 cover 10 countries (the United States, Kazakhstan, Sri Lanka, Mongolia, Kyrgyzstan, Uzbekistan, Tajikistan, Turkmenistan, Afghanistan and Iran), and one covers Nigeria, which is shelved
140 accounts in every table, including 59 industries chosen by measurement
Every account balances to better than one part in ten million
Comparing two sources caught a $31.2bn error in a global database that had passed every check on its totals

The United States' 2019 Social Accounting Matrix as a grid of 140 by 140 accounts. Bright squares are large payments: a dense block where industries buy from each other, a diagonal line where each industry sells its own product, a row of value added (wages and profit combined in this build), and columns of final spending. — The United States in 2019, one of the 18 tables built so far, with 140 accounts and 2,393 payments. Data: OECD Inter-Country Input-Output tables, 2025 edition.

Problem

The SAMs are there to calibrate an agent-based model for economic-statecraft research. That model, which is still to be built, will simulate how sanctions, export bans or a supply shock spread through an economy. It needs each economy’s real structure, such as how much of a country’s steel goes to construction, how much household income comes from wages, and how much an industry depends on imports. A SAM records exactly that in a form a model can read.

A small example SAM with two industries, two products, labour, capital, households, savings and the rest of the world. Each row lists what an account receives and each column what it pays, and the row and column totals match. — A SAM in miniature, using the toy economy from the pipeline’s own test file. Read each cell as “the row receives this much from the column”. For example, households receive 70 from labour. Every row total equals its column total.

Building a good one is hard for four reasons.

No single dataset is enough. One source has the trade detail and another splits income into wages and profit. The target region, Central Asia and the Indo-Pacific, is thinly covered by most of them.
Sources disagree, so the raw table never balances. Forcing it to balance can quietly overwrite good data with guesses.
The model needs every country at full detail. Lumping countries into composite regions like “rest of the world” won’t do.
Every number has to be traceable, so that an estimate is never mistaken for a measurement.

This is the second phase of my research. The first, my GTAP trade stress model, exposed three limits in global trade tables, and I designed this pipeline around them.

More detail: how phase 1 shaped this project

The GTAP model pushed a supply cut through global trade tables. Checking its results exposed three limits. The tables lacked the income side of the economy, so the math needed a small correction term, and that term ended up deciding the results. Two countries also showed zero risk because they mine almost no metal ore, and the model only cut a country’s own mining, even though both import all the ore they use.

Diagram in three boxes. Phase 1, the trade stress model of January to June 2026, used GTAP 10 and OECD ICIO data to model the 2010 Japan rare-earths episode. Checking its results exposed three limits: output approximated from intermediate flows only; a Leontief inverse that needed a regularization term, which ended up deciding the results; and zero stress for Germany and the UK, because they mine almost no metal ore and the model only cut a country’s own mining, though both import all the ore they use. These were carried into phase 2, the SAM pipeline: full tables with value added and final demand, cross-entropy balancing, GDP-identity and structural-plausibility gates, and cell-level provenance. — How phase 1 shaped phase 2. Each limit the trade stress model exposed became a requirement for the SAM pipeline.

Design

Each build combines three kinds of input. A backbone database records who buys what from whom. Translator files map each backbone’s sectors onto one shared list. Referees are independent GDP figures that the finished table is checked against but never built from.

Flow diagram: GLORIA, ADB and OECD ICIO data and the concordance files feed five pipeline steps (read and map, assemble the raw SAM, balance, validate, write only if no error), with World Bank and IMF GDP as independent benchmarks, producing balanced country SAMs. — How the data moves through the pipeline: read and map, assemble, balance, validate, and write only if nothing fails.

Three backbones. GLORIA covers 164 regions and is the only one of the three that splits income into labour and capital. The Asian Development Bank’s tables cover countries the OECD folds into “rest of world”. The OECD’s ICIO is the cross-check. Each build uses one of them, and where another source covers the same country, it checks the build.
A balancer that moves guesses before measurements. No maintained Python version of cross-entropy balancing (Robinson, Cattaneo and El-Said, 2001) existed, so I implemented it from the published formulation. Each cell is weighted by how trustworthy its source is, and an imputed cell absorbs about a hundred times the adjustment of an observed one.
Solving the small problem instead of the big one. The direct formulation solves for every cell, a few thousand per country, and it didn’t finish balancing one country in ten minutes. Its dual, an equivalent form of the same problem, solves for one number per account. That’s 140 numbers, and it balances a country in under a second. The two answers agree to 2×10⁻¹¹.
Provenance on every cell. Each number records its source and how it was derived, on a six-level scale from observed to scenario. A build that contains an unflagged value fails validation.

More detail: source coverage, the 59 sectors, and the balancer’s rules

Grid of the ten countries against four sources (GLORIA, ADB, OECD ICIO, GTAP), showing where a SAM was built and where a source can cross-check it. Sri Lanka is marked unusable in GLORIA, and Turkmenistan has no second source. — Which source covers which country. Most countries have one usable backbone, and Turkmenistan has nothing that can cross-check it.

59 sectors, chosen by measurement. A group of related sectors is split apart only if one of them is at least 1% of a country’s output somewhere, and every member is measured reliably, at 2% reconciliation error or less. The extra detail goes where economic statecraft bites. Crude oil is split from natural gas (crude alone is 8.1% of Kazakhstan’s output), pipelines are their own transport mode, and banking is separate from insurance. Copper ore gets its own sector too, after clearing the gate on Mongolian data at 13.38% of output. Uranium didn’t, at 0.11% of Kazakh output, so it stays grouped, and a test makes sure it does.

Bar chart of the largest single metal ore in each country as a share of gross output, on a log scale. Only Mongolian copper ore, at 13.38%, clears the 1% line. — The 1% rule for metal ores. Only Mongolian copper clears it, so copper became its own sector and uranium did not.Computed from GLORIA MRIO release 060 (ielab.info/resources/gloria), CC BY-NC-SA 4.0. Shared under the same licence.

The balancer’s rules. A cell that starts at zero stays zero unless it is deliberately opened, and flows that the accounting structure rules out can never be opened. The six provenance levels run from observed through allocated and imputed to scenario. A test checks that the direct and dual answers match.

The math

Five formulas do the work in this pipeline. For each one, here’s what it says in plain words and why I use it. The last, the Leontief matrix, is also what my trade-stress project ran on.

1. The accounting identity

\sum_{j} T_{i j} = \sum_{j} T_{j i}

\frac{| r_{i} - c_{i} |}{\max (r_{i}, c_{i})} \leq 10^{- 7}

T_ij is what account j pays account i, so row i is what account i receives and column i is what it spends. r_i and c_i are those two totals.

In plain words. Every account’s income has to equal its spending, like a budget that balances to the cent. The check allows a gap of at most one part in ten million of the account’s size.

Why I use it. It’s the rule that makes a table a SAM. A model run on a table that doesn’t balance would create or lose money out of nowhere, so a build that misses it fails.

2. Cross-entropy balancing

min_{T \geq 0} \sum_{i j} w_{i j} KL (T_{i j} ‖ T_{i j}^{0})

KL (x ‖ p) = x \ln \frac{x}{p} - x + p

T⁰ is the raw table from the sources, and w_ij is how much I trust cell ij, high for observed cells and low for imputed ones. KL is the Kullback-Leibler divergence, which measures how far a number has moved in proportion to its size. The table also has to satisfy the identity above.

In plain words. Find the balanced table closest to the raw one, where moving a trusted cell costs more than moving a guessed one.

Why I use it. The sources disagree, so the raw table never balances on its own. RAS, the classic method, treats every total as exact and stops converging when sources conflict. Cross entropy treats them as estimates, reports which ones had to move, and keeps a cell that starts at zero at zero, so it never invents a flow.

3. Solving it through the dual

T_{i j} = T_{i j}^{0} \cdot \exp (\frac{λ_{j} - λ_{i}}{w_{i j}})

λ_i is one number per account, found by the solver.

In plain words. At the best answer, every cell is its starting value times a factor set by just two numbers, one for the account that receives and one for the account that pays.

Why I use it. That shrinks the problem from a few thousand cells to 140 account numbers. The direct version didn’t finish one country in ten minutes, and this one balances a country in under a second. The solver also knows it’s done exactly when every account balances.

4. GDP from the income side

GDP = VA + (taxes - subsidies)

gap = \frac{| {GDP}_{SAM} - {GDP}_{ref} |}{{GDP}_{ref}}

VA is value added, the wages and capital income every industry pays. GDP_ref is an official figure the table is never built from.

In plain words. Add up what the economy pays workers and owners of capital, plus taxes on production and products, minus subsidies. That’s GDP measured from the income side.

Why I use it. A table can balance perfectly and still be the wrong size, and comparing it with an outside figure catches that. A gap up to 0.05% passes, and a gap over 0.5% puts a review flag on the build.

5. The Leontief matrix, and two checks built on it

a_{i j} = \frac{Z_{i j}}{x_{j}}

L = {(I - A)}^{- 1}

a i i must stay below 1

value added over output stays between 0.10 and 0.98

Z_ij is what industry j buys from industry i, x_j is industry j’s output, and VA_j is its value added. a_ii, the diagonal of A, is the share of an industry’s output that it buys back as its own input. The checks apply to industries that are at least 0.5% of the economy.

In plain words. A, the Leontief matrix, says how many cents of each input an industry needs for every dollar it makes. L adds up every round of that buying, which is how an input-output model follows a change through a whole economy. A Leontief system needs a_ii below 1 to work at all, so anything over 0.60 gets a human look. Value added also has to sit between 10 and 98 cents per dollar of output.

Why I use it. A table can balance and match GDP and still be wrong inside. GLORIA’s Sri Lanka table had its government industry buying back 0.884 of its own output, and it passed every check that existed at the time. The pipeline doesn’t run L itself. It checks that A is sound, because any model built on these tables will rely on it. My trade-stress project ran on L directly.

Build

I’m building it in layers, because each layer needs the one before it to be stable. The backbones, the split between wages and profit, and the balancing are done. Government and taxes are in progress this fall, and linking the countries to each other hasn’t started.

One command builds a country end to end, and Sri Lanka takes about 20 seconds. If a build hits an error it writes nothing, so a broken table can never replace a good one.

More detail: the build layers, the 140 accounts, and the tools

Layer	What it adds	Status
1. Backbone	Production, final demand and trade, from GLORIA, ADB or OECD ICIO	Done, with three adapters and 10 countries
2. Income and institutions	Wages versus profit, plus government, taxes and transfers	Wages and profit done. Government and taxes in progress this fall
3. Balance each country	Cross-entropy balancing and validation	Done
4. Link the countries	Reconciled trade between the country tables	Not started
5. Extensions	Energy, employment and finance, plus households by income decile	Not started. Household deciles planned this fall

Block-level SAM for Kazakhstan 2023 in US$ billions: industries sell 722 to products, buy 476 of inputs and pay 247 to factors, which pay 247 to households; the enterprises row and column are empty. — A real SAM summed into blocks, for Kazakhstan 2023 in US$ billions. It has the same shape as the toy. The empty enterprises row and column are institutional detail that’s still to come.Computed from GLORIA MRIO release 060 (ielab.info/resources/gloria), CC BY-NC-SA 4.0. Shared under the same licence.

A full SAM has 140 accounts: 59 industries, 59 products, 8 kinds of income, households, enterprises, government, 8 tax accounts, savings and investment, the rest of the world, and a statistical-discrepancy account that keeps unexplained gaps out of real accounts.
The OECD and ADB downloads are pinned by SHA-256 hash and retrieval date, so the pipeline notices if a source file changes. GLORIA’s full table is about 5.8 GB, so the pipeline reads it once per country and caches that slice.
It’s written in Python, with scipy and cvxpy for the optimization, pandas and Parquet for the data, matplotlib for the figures, and pytest for the tests.

Testing and iteration

Every build runs 13 automated checks, from balance and the GDP identity to two structural-plausibility tests, and 344 automated tests guard the code (passing as of 9 October 2026). The two structural thresholds were measured from real builds, and a threshold never gets widened just to silence a finding.

When every check passes and the data is still wrong. For a month, the Sri Lanka build on GLORIA looked like the best pilot. Its GDP was within 0.6% of the World Bank figure, and every check was green. Building it again on a second source, ADB, exposed the problem. GLORIA records Sri Lanka’s government sector buying $31.2bn of its own output, about 21.6% of the country’s gross output. It also records $10.3bn of pipeline transport in a country with no pipeline network. Each error is an output and an input at the same time, so the two sides cancel out of GDP and out of the balance condition. That’s why nothing caught them.

Dot plot of value added divided by gross output for Sri Lankan industries, GLORIA 2023 against ADB 2022, with a shaded plausible band from 0.10 to 0.98. GLORIA puts government at 0.07 and several service sectors near 1.0; the ADB values all sit in the band. — The same Sri Lankan industries from two sources. GLORIA’s government sector sits at 0.07, which means it buys nearly everything it produces. ADB’s values are all plausible. The chart reads the finished, balanced table, where that one cell comes to $31.4bn. GLORIA’s raw table records $31.2bn.Computed from GLORIA MRIO release 060 (CC BY-NC-SA 4.0), ADB MRIO (CC BY 3.0 IGO) and OECD ICIO tables. Shared under CC BY-NC-SA 4.0.

So matching totals don’t prove the structure is right. Sri Lanka now builds on ADB, and GLORIA supplies only the split between wages and profit. On ADB, apparel, the country’s largest export, is 10.2% of output, against 3.6% on GLORIA. Two new checks now catch this kind of error from a single source, which matters because most of the target region has no second source.

A second check. On 30 September an AI tool (Codex) audited and revised the code, and on 1 October a second AI tool (Claude Code) checked that work. The audit found a real error in the GDP formula, which counted relocated subsidies as tax revenue. Every reported GDP gap was recalculated with the fixed formula, and all 18 saved tables still balance below 10⁻⁷ with no negative cells.

Two big GDP gaps turned out to come from the data, not the pipeline. Nigeria came out 28% low because a 2025 rebasing of its national accounts raised its 2019 GDP by about 42%. Against the old accounts the gap is 1.8%, and Nigeria stays shelved. Turkmenistan came out +53.7% against the World Bank and −4.5% against the IMF, which converts at a different exchange rate.

More detail: the 13 checks, the new structure checks, and the second benchmark

The 13 checks cover balance, the GDP identity, factor payments, control totals, the external account, how much balancing moved, non-finite values, negative cells, unlabelled provenance, empty accounts, the second GDP benchmark, and two structural-plausibility tests. The test suite includes stress tests that feed real pilot data with deliberately broken inputs.

Dot plot for Kazakhstan 2023: each account’s gap between row and column total before balancing (scattered up to about 2%) and after (all below the 1e-7 tolerance line). — Kazakhstan 2023, with one dot per account showing the gap between its row and column totals before and after balancing. Every account ends below the tolerance of 1×10⁻⁷.Computed from GLORIA MRIO release 060 (ielab.info/resources/gloria), CC BY-NC-SA 4.0. Shared under the same licence.

On ADB, the Sri Lanka table needed almost no balancing. Of the two new checks, one sets a plausible range for value added (an industry’s output minus the inputs it buys) over output, and the other caps how much any sector can buy back of its own output.

The same rebasing mechanism later explained the gap from Uzbekistan’s 2024 revision. After Turkmenistan, every build reports a second GDP benchmark next to the World Bank figure. Only the World Bank figure decides whether a build passes, though, so a gap can’t be argued away after the fact.

Results

GDP is the headline check. The income a SAM records should match the official figure for that country and year, and any gap over 0.5% puts a review flag on the build so someone has to explain it.

Build	Gap to World Bank GDP	What explains it
Kazakhstan 2023, GLORIA	+0.76%	Close match
Sri Lanka 2022, ADB	−2.82%	Within 3%. This is the Sri Lanka build in use
United States 2019, OECD ICIO	−0.05%	Close match
United States 2023, GLORIA	+3.33%	Product taxes are probably counted twice, since GLORIA’s US value added alone matches GDP to 0.001%. Inferred, not yet confirmed
Uzbekistan 2023, GLORIA	−15.71%	−0.28% against the accounts before Uzbekistan’s 2024 revision
Turkmenistan 2023, GLORIA	+53.69%	−4.53% against the IMF, which uses the official exchange rate
Kyrgyzstan 2023, GLORIA	−8.92%	Not yet explained. ADB also covers Kyrgyzstan, so it can be cross-checked

The GDP chart for every build, how much of each table comes straight from data, and who earns the value added are in the gallery below. The charts use the 14 current builds. The 18 saved tables also include superseded and shelved ones, such as the GLORIA Sri Lanka table and Nigeria.

What I’d change, and what’s next

Start version control on day one. The project has no Git history, so the record of every change is file timestamps and dated recovery copies. Now every substantive change gets a recovery copy and a tested rollback first.
Check the structure from the start, as well as the totals. The ratio checks that would have flagged Sri Lanka right away only arrived after a second source exposed the problem.

Next, under the PEAK Summit Fellowship (fall 2026), I’m adding government, tax and transfer accounts, starting with Sri Lanka 2022, and splitting households by income decile using World Bank PIP data. The fall ends with a final report and abstract for RISE, Northeastern’s research expo. Later, the country tables get linked through their trade into one global web, with every country at full detail.

More detail: the plan for government accounts

The research for the government accounts is done. The central bank’s and statistics department’s 2022 tables are downloaded and checked, and four build options were compared. The recommended option, which isn’t adopted yet, reconciles the fiscal and national-accounts totals first and then fits the ADB detail to them.

Gallery

More charts (6): GDP for every build, data provenance, who earns value added, and more

Dot plot of SAM GDP minus benchmark GDP, in percent, for 14 builds, against the World Bank and a second benchmark, with a shaded band of plus or minus 3%. Most builds sit inside the band. Against the World Bank, Uzbekistan, Iran, Kyrgyzstan and Turkmenistan sit well outside it, and the United States 2023 GLORIA build sits just outside at +3.3%. — Every build’s GDP against the World Bank, whose figure decides whether a build passes, and against a second benchmark that helps explain the gaps. The shaded band marks ±3%.Computed from GLORIA MRIO release 060 (CC BY-NC-SA 4.0), ADB MRIO (CC BY 3.0 IGO) and OECD ICIO tables. Shared under CC BY-NC-SA 4.0. GDP benchmarks: World Bank World Development Indicators (CC BY 4.0) and IMF World Economic Outlook.

Stacked bars showing, for each of the 14 builds, the share of total value that is source-derived, allocated by assumption, or an imputed placeholder. Source-derived shares run from 74% to 84%, except Sri Lanka at 58%. — How much of each SAM comes straight from data. In every build except Sri Lanka, 74–84% of the value is read from a source, and the rest is flagged as allocated or placeholder.Computed from GLORIA MRIO release 060 (CC BY-NC-SA 4.0), ADB MRIO (CC BY 3.0 IGO) and OECD ICIO tables. Shared under CC BY-NC-SA 4.0.

Stacked bars of labour, capital and mixed-income shares of factor income for ten countries. The United States is 54% labour, 37% capital and 9% mixed income. — Who earns the value added in each SAM. The US labour share, 54%, is in line with published estimates of 53–55%.Computed from GLORIA MRIO release 060 (CC BY-NC-SA 4.0), ADB MRIO (CC BY 3.0 IGO) and OECD ICIO tables. Shared under CC BY-NC-SA 4.0.

Heatmap of the full 140 by 140 SAM for Kazakhstan 2023 on a log scale, with 3,467 non-zero cells: a diagonal of industries selling their own products and a dense block of industries buying inputs from each other. — The full 140 × 140 matrix for Kazakhstan 2023, with 3,467 non-zero cells. Computed from GLORIA MRIO release 060 (ielab.info/resources/gloria), CC BY-NC-SA 4.0. Shared under the same licence.

Bar chart on a log scale of the total amount the balancer moved in each of 14 builds, as a percent of benchmark GDP. The nine GLORIA builds sit above the 0.5% review threshold, from 1.11% for the United States 2023 to 19.01% for Afghanistan 2023. The three OECD ICIO builds moved 0.001% to 0.003%, and the two ADB builds about zero. — How much the balancer had to move in each build, as a share of GDP. The OECD and ADB builds needed almost no adjustment. Every GLORIA build moved more than 0.5% of GDP, which adds a review flag to its build report. Computed from GLORIA MRIO release 060 (CC BY-NC-SA 4.0), ADB MRIO (CC BY 3.0 IGO) and OECD ICIO tables. Shared under CC BY-NC-SA 4.0.

Timeline from 2000 to 2024. ADB national tables cover 2000 to 2022, and a build uses 2022. The OECD ICIO 2025 file covers 2016 to 2022, with builds for 2019 and 2022. GLORIA release 060 covers 2023, which is built. GTAP 12 has reference year 2023, but the national tables beneath it date from 2003 for Kyrgyzstan to 2018 for Uzbekistan. — The years each source covers, and the years actually built. GTAP 12 is dated 2023, but the national tables underneath it are 5 to 20 years older.