Skip to content

Fix robustness bugs in gmd(): clean errors, flag coercion, year filtering, fetch hardening#6

Open
Yicheng-Zhao1999 wants to merge 1 commit into
KMueller-Lab:mainfrom
Yicheng-Zhao1999:fix/python-robustness-bugs
Open

Fix robustness bugs in gmd(): clean errors, flag coercion, year filtering, fetch hardening#6
Yicheng-Zhao1999 wants to merge 1 commit into
KMueller-Lab:mainfrom
Yicheng-Zhao1999:fix/python-robustness-bugs

Conversation

@Yicheng-Zhao1999

Copy link
Copy Markdown

Summary

Fixes the batch of robustness issues reported for the Python global_macro_data package. Each fix was developed step-by-step with an independent review pass, and the whole set was re-verified against the live backend.

Test suite: 130 passed (99 baseline + 31 new regression tests), fixture-backed and deterministic.

Bugs fixed

# Symptom (before) Behavior (after)
1 (P0) A listed version whose .dta is missing failed with a generic "raise an issue" Clear GMDCommandError naming the version & path: The data file for version X could not be retrieved (distribute/GMD_X.dta)…
2 (P1) country=840 (scalar int) leaked TypeError: 'int' object is not iterable Clean GMDCommandError: country must be a string or a list of strings…; lists containing ints still accepted
3 (P1) get_available_versions() leaked a raw RuntimeError when offline Wrapped in GMDCommandError; cache/local fallbacks preserved
4 (P1) raw="no" / iso="off" were truthy and activated; fast only matched "yes" New _coerce_flag: accepts bools + a whitelist (yes/no/true/false/on/off/1/0); ambiguous values raise; fast now consistent
5 (P2) start_year= leaked TypeError; no year filtering at all start_year/end_year are real params doing inclusive row filtering; genuinely-unknown kwargs still raise TypeError
6 (P2) Wrong setup.py author; unpinned deps; no User-Agent/retry; cache could be poisoned by a partial write Correct author/email; pinned deps; _fetch_from sends a User-Agent and retries with backoff (no retry on 4xx); _atomic_to_stata writes the cache atomically

Extra: GitHub-mirror fallback for helper tables

Remote reads now try the S3 release bucket first and fall back to the GitHub mirror. Note (documented in-code): the GitHub mirror only hosts the helpers/* tables — it does not host the versioned datasets (distribute/GMD_<ver>.dta) or per-variable raw CSVs, so those remain S3-only and cannot fall back. Verified live: with S3 forced down, helpers/versions.csv still loads from GitHub while a dataset fetch fails cleanly.

Notes for maintainers

  • The original Bug 1 S3 404s (2025_01/03/05/06/08) no longer reproduce on the live bucket — all nine versions return 200 and load end-to-end, so the missing vintages appear to have been re-uploaded already. The code-side improvement (clearer message) is still worth keeping. Suggest a release-pipeline pre-assertion that every version in versions.csv has a reachable distribute/GMD_<ver>.dta.
  • _fetch_primary is now unused by production code (kept for test stability).

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant