Fix robustness bugs in gmd(): clean errors, flag coercion, year filtering, fetch hardening#6
Open
Yicheng-Zhao1999 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the batch of robustness issues reported for the Python
global_macro_datapackage. Each fix was developed step-by-step with an independent review pass, and the whole set was re-verified against the live backend.Test suite: 130 passed (99 baseline + 31 new regression tests), fixture-backed and deterministic.
Bugs fixed
.dtais missing failed with a generic "raise an issue"GMDCommandErrornaming the version & path:The data file for version X could not be retrieved (distribute/GMD_X.dta)…country=840(scalar int) leakedTypeError: 'int' object is not iterableGMDCommandError: country must be a string or a list of strings…; lists containing ints still acceptedget_available_versions()leaked a rawRuntimeErrorwhen offlineGMDCommandError; cache/local fallbacks preservedraw="no"/iso="off"were truthy and activated;fastonly matched"yes"_coerce_flag: accepts bools + a whitelist (yes/no/true/false/on/off/1/0); ambiguous values raise;fastnow consistentstart_year=leakedTypeError; no year filtering at allstart_year/end_yearare real params doing inclusive row filtering; genuinely-unknown kwargs still raiseTypeError_fetch_fromsends a User-Agent and retries with backoff (no retry on 4xx);_atomic_to_statawrites the cache atomicallyExtra: GitHub-mirror fallback for helper tables
Remote reads now try the S3 release bucket first and fall back to the GitHub mirror. Note (documented in-code): the GitHub mirror only hosts the
helpers/*tables — it does not host the versioned datasets (distribute/GMD_<ver>.dta) or per-variable raw CSVs, so those remain S3-only and cannot fall back. Verified live: with S3 forced down,helpers/versions.csvstill loads from GitHub while a dataset fetch fails cleanly.Notes for maintainers
2025_01/03/05/06/08) no longer reproduce on the live bucket — all nine versions return 200 and load end-to-end, so the missing vintages appear to have been re-uploaded already. The code-side improvement (clearer message) is still worth keeping. Suggest a release-pipeline pre-assertion that every version inversions.csvhas a reachabledistribute/GMD_<ver>.dta._fetch_primaryis now unused by production code (kept for test stability).🤖 Generated with Claude Code