gh-154719: Preserve trailing whitespace in t-string interpolation expressions

[lpyu001]

Problem

Interpolation.expression for t-string literals did not preserve all whitespace inside the replacement field. In particular, whitespace immediately before }, !, :, or a debug = was removed, contrary to the documented behavior.

A related issue occurred while reconstructing interpolation metadata containing comments. For an expression such as:

template = t'''{(
    1,  # comment
    "\"#")}'''

the expression metadata was truncated at the # inside the escaped-quote string. This could cause ast.unparse() to produce invalid Python code that could not be parsed again.

Root cause

The parser unconditionally stripped trailing whitespace from every interpolation expression while processing metadata intended for debug expressions.

Additionally, the lexer reconstructs expression metadata in two passes when comments are present:

• The first pass correctly skips escaped characters while locating comments.
• The second pass did not skip escaped characters.

As a result, an escaped quote such as \" was incorrectly treated as the end of a string. A following # was then interpreted as the beginning of a comment, causing the remainder of the expression metadata to be discarded.

Solution

Preserve lexer metadata unchanged for regular t-string interpolations.

For debug interpolations:

• Remove only the debug = and whitespace following it.
• Preserve whitespace before the debug =.
• Return the original metadata unchanged if the debug marker is absent.

Make the lexer's second metadata-processing pass handle escaped characters consistently with the first pass, so escaped quotes and # characters inside strings are preserved.

Changes

• Preserve trailing whitespace in Interpolation.expression.
• Preserve whitespace before }, !, :, and debug =.
• Handle escaped characters correctly while removing comments from reconstructed interpolation metadata.
• Add exact regression tests for:
  • Leading and trailing whitespace.
  • Conversions and format specifications.
  • Debug expressions.
  • Comparison expressions followed by debug =.
  • Multiline expressions.
  • Comments combined with escaped quotes and # inside strings.
• Add an AST parse/unparse/parse round-trip regression test.
• Update the multiline t-string expectation in test_annotationlib.

Testing

• test_tstring and test_unparse: 94 tests passed.
• test_fstring, test_grammar, test_ast, test_compile, test_tokenize, test_annotationlib, and test_string: 883 tests passed.
• Debug build completed successfully.

• Issue: Interpolation.expression drops trailing whitespace despite documentation saying it includes any whitespace #154719

[pablogsal]

I am not sure this comment is accurate: after the lexer fix, removing comments can never drop the debug = (the marker cannot be inside a comment and the metadata always extends past it), so IIUC this branch is only reachable via things like a line continuation right after the =. Can you adjust the comment or add a test that reaches this branch?

[lpyu001]

Thanks, I will address it.

[lpyu001]

A valid case that reached
this branch is an explicit line continuation after the debug marker:

t'''{value =\
}'''

The first patch returned "value =\\\n" as the expression. This conflicts with the [Interpolation.expression documentation(https://docs.python.org/3.16/library/string.templatelib.html#string.templatelib.Interpolation.expression),which says the expression ends before =. The expected expression is "value ".

I updated the code to strip explicit line continuations after the debug marker and added regression tests.

[pablogsal]

The lexer fix also changes the debug text for f-strings (_PyLexer_set_ftstring_expr is shared when in_debug is set), e.g. a debug expression combining a comment with an escaped quote was truncated before this. Can you add a test for the f-string case in test_fstring as well? Maybe worth mentioning f-strings in the NEWS entry too.

[lpyu001]

I have added the tests and updated the NEWS file