Bug description:
The iconv-backed codecs start a fresh conversion on every call, so an incremental decode of
a stateful encoding loses the shift state and silently returns wrong text. iso-2022-cn has
no built-in codec, so the plain name reaches the iconv codec:
import codecs
blob = 'ABC中文DEF'.encode('iso-2022-cn')
print('one-shot: ', blob.decode('iso-2022-cn'))
d = codecs.getincrementaldecoder('iso-2022-cn')()
print('incremental:', ''.join(d.decode(bytes([b])) for b in blob) + d.decode(b'', True))
one-shot: ABC中文DEF
incremental: ABCVPNDDEF
codecs.iterdecode() and StreamReader.read(1) are wrong in the same way. A stateful
encoding forced with the iconv: prefix is also affected, even when it has a built-in
codec.
Either the codec objects need to keep one conversion across calls, or the iconv search
function should refuse stateful encodings, which would make iso-2022-cn a LookupError.
The iconv codecs are new in 3.16 (gh-152997), so no released version is affected.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Linked PRs
Bug description:
The iconv-backed codecs start a fresh conversion on every call, so an incremental decode of
a stateful encoding loses the shift state and silently returns wrong text.
iso-2022-cnhasno built-in codec, so the plain name reaches the iconv codec:
codecs.iterdecode()andStreamReader.read(1)are wrong in the same way. A statefulencoding forced with the
iconv:prefix is also affected, even when it has a built-incodec.
Either the codec objects need to keep one conversion across calls, or the iconv search
function should refuse stateful encodings, which would make
iso-2022-cnaLookupError.The iconv codecs are new in 3.16 (gh-152997), so no released version is affected.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Linked PRs