FIX: Correct dataset reference metadata#2262
Open
romanlutz wants to merge 2 commits into
Open
Conversation
Align dataset authors, affiliations, citation details, and YAML provenance with authoritative paper and source metadata. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: dd1ad5b0-f48b-4a55-b75e-0b875d438047
Contributor
There was a problem hiding this comment.
Pull request overview
This PR corrects drift in dataset/bibliography provenance metadata across PyRIT’s seed dataset loaders and related YAML metadata, aligning author/group attribution and references with authoritative sources while keeping lineage distinctions (e.g., DecodingTrust vs. RealToxicityPrompts).
Changes:
- Updated multiple remote seed dataset loaders to correct/normalize
authors,groups, and reference metadata, including moving some provenance constants onto the owning loader classes. - Refined DecodingTrust toxicity provenance to attribute the pinned subset to DecodingTrust while keeping RealToxicityPrompts as upstream lineage.
- Normalized affiliation strings across dataset YAML metadata and updated BibTeX author formatting for DarkBench.
Reviewed changes
Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/datasets/test_decoding_trust_toxicity_dataset.py | Adjusts assertions around DecodingTrust toxicity per-seed provenance metadata. |
| pyrit/datasets/seed_datasets/remote/wildguardmix_dataset.py | Moves provenance constants onto the loader class and updates group metadata. |
| pyrit/datasets/seed_datasets/remote/transphobia_awareness_dataset.py | Corrects publication year in the referenced paper description. |
| pyrit/datasets/seed_datasets/remote/sosbench_dataset.py | Adds an additional author to the SOSBench provenance list. |
| pyrit/datasets/seed_datasets/remote/simple_safety_tests_dataset.py | Updates an author name in the SimpleSafetyTests provenance list. |
| pyrit/datasets/seed_datasets/remote/salad_bench_dataset.py | Normalizes a group/affiliation string (“The Chinese University of Hong Kong”). |
| pyrit/datasets/seed_datasets/remote/promptintel_dataset.py | Updates group attribution for PromptIntel seeds. |
| pyrit/datasets/seed_datasets/remote/msts_dataset.py | Expands/normalizes group affiliations for MSTS provenance metadata. |
| pyrit/datasets/seed_datasets/remote/mm_safetybench_dataset.py | Moves provenance constants onto the class and ensures groups are attached to produced seeds. |
| pyrit/datasets/seed_datasets/remote/harmbench_dataset.py | Switches docstring reference to cite key and expands the authors list for seed provenance. |
| pyrit/datasets/seed_datasets/remote/garak_dataset.py | Replaces generic “garak Team” provenance with specific author/group attribution. |
| pyrit/datasets/seed_datasets/remote/figstep_dataset.py | Expands group affiliations for FigStep provenance. |
| pyrit/datasets/seed_datasets/remote/decoding_trust_toxicity_dataset.py | Refines docstring references and adjusts per-seed authors/groups to reflect DecodingTrust-only attribution. |
| pyrit/datasets/seed_datasets/remote/darkbench_dataset.py | Updates an author name formatting in dataset provenance. |
| pyrit/datasets/seed_datasets/remote/dangerous_qa_dataset.py | Adds an additional affiliation to the DangerousQA provenance groups list. |
| pyrit/datasets/seed_datasets/remote/coconot_dataset.py | Expands/normalizes group affiliations for CoCoNot provenance. |
| pyrit/datasets/seed_datasets/remote/beaver_tails_dataset.py | Normalizes group affiliation strings for BeaverTails provenance. |
| pyrit/datasets/jailbreak/templates/multi_parameter/many_shot_template.yaml | Normalizes institution names for Many-shot jailbreaking template provenance. |
| pyrit/datasets/jailbreak/templates/cipher_chat.yaml | Normalizes affiliation string for CipherChat template provenance. |
| pyrit/datasets/executors/question_answer/wmdp_dataset.py | Corrects an author name in the embedded WMDP metadata string. |
| pyrit/datasets/converters/persuasion/misrepresentation.yaml | Normalizes “UC Davis” to “University of California, Davis” in converter provenance. |
| pyrit/datasets/converters/persuasion/logical_appeal.yaml | Normalizes “UC Davis” to “University of California, Davis” in converter provenance. |
| pyrit/datasets/converters/persuasion/expert_endorsement.yaml | Normalizes “UC Davis” to “University of California, Davis” in converter provenance. |
| pyrit/datasets/converters/persuasion/evidence_based.yaml | Normalizes “UC Davis” to “University of California, Davis” in converter provenance. |
| pyrit/datasets/converters/persuasion/authority_endorsement.yaml | Normalizes “UC Davis” to “University of California, Davis” in converter provenance. |
| pyrit/datasets/converters/morse_description.yaml | Adds missing group attribution for CipherChat-derived converter metadata. |
| pyrit/datasets/converters/decomposition/word_game_preamble.yaml | Normalizes “University of Maryland, College Park” affiliation formatting. |
| pyrit/datasets/converters/decomposition/reconstruction_prompt.yaml | Normalizes “University of Maryland, College Park” affiliation formatting. |
| pyrit/datasets/converters/decomposition/decomposition_system_prompt.yaml | Normalizes “University of Maryland, College Park” affiliation formatting. |
| pyrit/datasets/converters/caesar_description.yaml | Adds missing group attribution for CipherChat-derived converter metadata. |
| pyrit/datasets/converters/atbash_description.yaml | Adds missing group attribution for CipherChat-derived converter metadata. |
| doc/references.bib | Updates DarkBench BibTeX author formatting to match the dataset provenance naming. |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: dd1ad5b0-f48b-4a55-b75e-0b875d438047
ValbuenaVC
approved these changes
Jul 23, 2026
| assert seed.authors is not None and "Boxin Wang" in seed.authors # DecodingTrust lead | ||
| assert "Samuel Gehman" in seed.authors # RealToxicityPrompts lead | ||
| assert seed.groups is not None and "University of Illinois Urbana-Champaign" in seed.groups | ||
| assert seed.groups is not None and "University of Illinois at Urbana-Champaign" in seed.groups |
Contributor
There was a problem hiding this comment.
Nit: Many of the fixes in this PR seem like they could be automated as linting or regression tests. E.g., for all references to universities in tests or datasets, match University of {State} {at} {Location} and not University of {State} {Location} etc. Do you think it's worth tracking as a follow-up item? Non-blocking so I've approved but curious if you think it's worth the effort
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Corrects bibliography and dataset provenance metadata that had drifted from authoritative papers and first-party sources. The audit aligns author names, affiliations, publication details, and contribution YAML metadata while preserving curated citation overrides and source-specific attribution.
Loader provenance constants now live on their owning classes where applicable, and the DecodingTrust toxicity loader attributes its pinned subset only to DecodingTrust while retaining RealToxicityPrompts as upstream lineage.
Tests and Documentation