Local area calibration used only 1 variable (household_count) at CD level — 588 total targets

[MaxGhenis]

Discovery

While building a pipeline comparison diagram, I traced through the code and calibration logs that produced the published HuggingFace district files (e.g. CA-04.h5). The production local area calibration used far fewer targets than expected.

Evidence

1. The target filter is extremely restrictive

fit_calibration_weights.py lines 105-115:

targets_df, X_sparse, household_id_mapping = builder.build_matrix(
    sim,
    target_filter={
        "stratum_group_ids": [4, 7],  # 4=SNAP households, 7=state income tax
        "variables": [
            "health_insurance_premiums_without_medicare_part_b",
            "snap",
            "state_income_tax",  # Census STC state income tax collections
        ],
    },
)

This uses OR logic (sparse_matrix_builder.py line 166), selecting only targets that match stratum groups 4 or 7 OR have those three variable names.

2. Production calibration log confirms 588 targets

calibration_log_20260131_153028.csv (the 5,000-epoch production run) contains 588 targets broken down as:

Geo level	Variable	Count
CD	household_count	350
State	household_count	51
State	snap	51
State	state_income_tax	44
National	person_count	4
National	snap	1
National	health_insurance_premiums_without_medicare_part_b	1
Sub-state	household_count	86
Total		588

3. The ONLY CD-level variable is household_count

Out of 436 congressional districts, only 350 have a target at all, and that target is exclusively household_count. No CD-level SNAP, income, employment, demographics, or any other variable is being calibrated.

4. The weight file is confirmed byte-identical

w_district_calibration.npy (the file uploaded to HuggingFace and used by modal_app/local_area.py to publish district H5 files) is byte-identical to calibration_weights_20260131_153028.npy. This confirms the 588-target, 5,000-epoch run is the one that produced all published district files.

5. The database has 37,758 targets available

The policy_data.db contains 37,758 targets across 27 stratum groups. Only 588 were used due to the restrictive filter. 86 CDs have zero targets.

Impact

Every published district-level file on HuggingFace was produced by a calibration that only matched household counts at the CD level. CD-level income distributions, SNAP participation, tax collections, etc. are entirely uncalibrated — they reflect whatever the stratified sampling + geography assignment happened to produce.

Context

The Arnold 2026 slides (CalibrationSlide.tsx) claimed 20,994 targets (34 target groups). No evidence supports this number in any actual calibration run. Maria's PR #489 ("Supporting all calibration targets") would have expanded coverage but hasn't been merged.

The new unified pipeline (PR #516) addresses this by using all 37,749+ DB-driven targets in a single solve.

@baogorek — does this match your understanding? Am I missing a different calibration run that used more CD-level targets? The only logs and weight files I found locally point to this 588-target run.

[MaxGhenis]

Correction: local files ≠ production weights

The w_district_calibration.npy on HuggingFace (uploaded Feb 4 in commit "new data and weights") is NOT the same as my local files from Jan 31.

	HuggingFace (production)	Local (Jan 31)
Elements	5,231,564	5,667,564
HH per CD	12,000	12,999
Nonzero weights	2,221,961	5,414,654
Weight sum	117,573,808	59,985,068

The production weights came from a different calibration run — likely executed on Modal on or before Feb 4. The Modal container is ephemeral, so no calibration log survives for that run. There's no log on HuggingFace either.

The core code path is the same — remote_calibration_runner.py clones the branch (default: main) and runs fit_calibration_weights.py unchanged. So the target filter (stratum_group_ids: [4, 7], variables: [hip, snap, state_income_tax]) would still apply, producing at most ~588 targets. But I can't verify the exact target count without the log.

The 12,000 vs 12,999 HH/CD difference suggests a different stratified_extended_cps.h5 was used too (the stratified CPS is also on HF and may have been regenerated with different parameters between runs).

@baogorek — do you have the calibration log from the Feb 4 Modal run? Or do you know if a different branch/filter was used?

[baogorek]

Here is the Calibration log from the Feb 4th wieghts cd_calibration_log.csv.

@MaxGhenis the process I currently use to update the Local Area H5s is to

1. Check in policy_data.db and the base (stratified) h5 data set in Hugging Face (the calibration directory)
2. Create a "calibration package" from "the junkyard" which uses the inputs in 1, resulting in the X matrix and the targets
3. Upload the calibration package to Kaggle and get a weights vector and a calibration log
4. Check in the weights to Hugging face (the calibration directory)
5. Run the Publish Local Area H5 Github action, which will result in the H5s being stored in a "staging" area on Hugging Face and GCP
6. Run a Promote Github action, which will move the files in the staging directory to their production equivalent and clean up staging.

We have code that can fit the model with a GPU on Modal. I know the above is clunky but we're not too far off from automating the rest of the pieces. I still like like the idea of a staging area and one final button press to send the H5s to production, but that's your call.

Let me know if you want an example prompt of how I tell Claude to "go into the junkyard" when I need it to know how the calibration package is created.

[MaxGhenis]

Unified calibration pipeline: initial findings

@baogorek Thanks for the calibration log — very helpful for comparison.

Target database

After running all legacy ETL scripts: 40,292 targets (525 national, 6,195 state, 33,572 CD), up from the initial 9,460. Removed 20 SPM threshold decile targets (now calculating SPM thresholds by metro), converted state_code constraints to state_fips, and fixed total_income_tax → income_tax.

Experiment results: clones matter, epochs don't

Config	Records	HH/CD	Targets	Epochs	Mean Error
1 clone, 100 ep	111K	~256	40,292	100	61.8%
1 clone, 1000 ep	111K	~256	40,292	1,000	61.2%
Bao Feb 4	5.2M	~12,000	18,435	1,000	5.7%
100 clones (running)	11.2M	~25,600	40,292	500	TBD

With only 1 clone (~256 HH per CD), the optimizer converges quickly but plateaus at ~61% mean error — not enough degrees of freedom per CD. Your production run with ~12K HH/CD achieved 5.7% at 1000 epochs. We're now running 100 clones (~25.6K HH/CD) which should be well above that threshold.

Error patterns (1 clone)

• Tax expenditure targets (SALT, medical, interest deductions): worst performers (400%+ error) — these may need a different approach
• SOI AGI-band targets: 84% mean error, but some specific bands hit <1%
• SNAP/household counts: best category at 32-51% mean error
• National vs CD: similar error rates (~75%), suggesting the issue is systematic, not just CD-level

What's different from your pipeline

Your workflow (DB → calibration package → Kaggle GPU → weights) produces the stratified_extended_cps.h5 with ~12K HH/CD pre-assigned. Our unified pipeline clones the base 111K HH and assigns geography dynamically. Same idea, different packaging.

Full findings saved to policyengine_us_data/storage/calibration/CALIBRATION_FINDINGS.md. Will update once the 100-clone run completes.

[baogorek]

@MaxGhenis the fit falling apart with all the targets is consistent with what I've seen. If you're not under 10% mean error, then it's a red flag. With the full set of targets, I've also noticed that the training will stop improving very early, like after a couple hundred epochs. When I'm unconstrained on the final sparsity and working with an easy set of targets, it will continue improving past 10K epochs.

Note that we can fit with around 20K targets can get the ~5% error (with some L2) and probably more than that with a bit of trial and error. But I think after that it's going to have to be a medicaid-style investigation about exactly why the equation can't do its job.

BTW, you're also suffering from lack of target upratings (let me know if you have an implementation), which I'm handling in #489. That PR is also handing a problem where targets that have multiple years are just all thrown in the pot (e.g., ssn_card_type), so that's probably hurting your fit too.

I am planning to rebase on top of this PR, so I would recommend being aggressive about merging it. I can clean up anything you missed.

FYI on the junkyard, which is in the brand new-cd-var. Here is a template if you want Claude to go look at it for any reason:

I have two clones of policyengine-us-data:
- Junkyard (~/devl/policyengine-us-data): an experimental clone with lots of scratch work, diagnostic scripts, and
in-progress changes. Not everything here is meant to be committed.
- Dev clone (~/devl/sep/policyengine-us-data): my clean development clone where I make PRs. I'm on a feature branch here.

The junkyard has working functionality I need to bring over. Trace what's required to run this command from the junkyard:

python create_calibration_package.py \
  --db-path ~/devl/huggingface/policyengine-us-data/calibration/policy_data.db \
  --dataset-uri ~/devl/sep/policyengine-us-data/policyengine_us_data/storage/stratified_extended_cps_2024.h5 \
  --mode Stratified \
  --local-output .

<you may want to reference the target_groups_report.md that's in the same folder, which
has groups we know we can successfully fit>
<custom instructions about the feature that you want to bring over from the junkyard>

[MaxGhenis]

Update: 100-clone run in progress + DB improvements

100-clone calibration

Matrix build completed: 40,292 targets × 11,152,400 columns, 1.07B nonzero entries. L0 optimization running now (500 epochs). Each CD gets ~25,600 HH — double Bao's 12K — so this should have plenty of degrees of freedom. Waiting on first epoch results.

Target filtering via DB flag

Per @baogorek's observation that the full 40K target set causes the fit to fall apart, I've added a calibrate column to the targets table (boolean, default=1). The DB stays exhaustive — all targets remain — but the matrix builder now respects calibrate=0 to exclude problematic targets without deleting them.

Also added --stratum-groups CLI flag to filter by stratum group ID:

# Only calibrate to known-good groups
python -m policyengine_us_data.calibration.unified_calibration \
    --stratum-groups 1,2,3,4,5,6,7,8,9

Auto-saved diagnostics

main() now auto-saves after each run:

• unified_diagnostics.csv — per-target errors (target, true_value, estimate, rel_error, abs_rel_error, achievable)
• unified_run_config.json — full config + summary metrics + wall-clock timing

Next steps

• If 100-clone run converges well → great, we know clones were the bottleneck
• If not → use calibrate=0 or --stratum-groups to narrow to a fittable subset per @baogorek's suggestion (~20K targets at ~5% error), then investigate remaining targets individually
• Still need target upratings from Supporting all calibration targets with SparseMatrixBuilder #489