diff --git a/peps/pep-0828.rst b/peps/pep-0828.rst index dfaf0b48fa2..1958f7f58d3 100644 --- a/peps/pep-0828.rst +++ b/peps/pep-0828.rst @@ -15,8 +15,7 @@ Abstract ======== This PEP introduces support for :keyword:`yield from ` in an -:ref:`asynchronous generator function ` -through a new ``async yield from`` construct: +:ref:`asynchronous generator function `: .. code-block:: python @@ -26,7 +25,7 @@ through a new ``async yield from`` construct: return 3 async def main(): - result = async yield from agenerator() + result = yield from agenerator() assert result == 3 @@ -39,8 +38,7 @@ This is not to be confused with an :term:`asynchronous generator iterator`, which is the object *returned* by an asynchronous generator. This PEP also uses the term "subgenerator" to refer to a generator, synchronous -or asynchronous, that is used inside of a ``yield from`` or ``async yield from`` -expression. +or asynchronous, that is used inside a ``yield from``. Motivation @@ -104,38 +102,18 @@ item. This comes with a few drawbacks: Specification ============= - -Syntax ------- - - Compiler changes -^^^^^^^^^^^^^^^^ +---------------- The compiler will no longer emit a :exc:`SyntaxError` for -:keyword:`return` statements inside asynchronous generators. - - -Grammar changes -^^^^^^^^^^^^^^^ - -The ``yield_expr`` and ``simple_stmt`` rules need to be updated for the new -``async yield from`` syntax: - -.. code-block:: peg - - yield_expr[expr_ty]: - | 'async' 'yield' 'from' a=expression - - simple_stmt[stmt_ty] (memo): - | &('yield' | 'async') yield_stmt +:keyword:`return` or ``yield from`` statements inside asynchronous generators. Changes to ``StopAsyncIteration`` --------------------------------- The :class:`StopAsyncIteration` exception will gain a new ``value`` attribute -to be used as the result of ``async yield from`` expressions. +to be used as the result of ``yield from`` expressions in asynchronous generators. This attribute can be supplied by passing a positional argument to ``StopAsyncIteration``. For example: @@ -156,18 +134,18 @@ If no argument is supplied, ``value`` will be ``None``. In the body of an asynchronous generator function, the statement ``return expression`` is roughly equivalent to ``raise StopAsyncIteration(expression)``. However, similar to implicit -``StopIteration`` exceptions raised inside of synchronous generators, +``StopIteration`` exceptions raised inside synchronous generators, the exception cannot be caught in the body of the asynchronous generator. -``async yield from`` semantics ------------------------------- +``yield from`` semantics in an asynchronous generator +----------------------------------------------------- -The statement +In an asynchronous generator, the statement .. code-block:: python - RESULT = async yield from EXPR + RESULT = yield from EXPR is roughly equivalent to the following: @@ -216,26 +194,24 @@ Rationale ========= -Relation to ``yield from`` --------------------------- +Relation to synchronous generators +---------------------------------- -This PEP aims to be very similar to the semantics of ``yield from``, with the -exception that asynchronous generator methods are used instead of synchronous +This PEP aims to be very similar to the semantics of synchronous ``yield from``, +with the exception that asynchronous generator methods are used instead of synchronous generator methods when delegating. This is a very intuitive design and furthers symmetry with synchronous generators. -Choice of ``async yield from`` as the syntax --------------------------------------------- +Choice of ``yield from`` as the syntax +-------------------------------------- -This PEP uses ``async yield from`` as the syntax to ensure that the behavior -of the syntax is immediately clear to the user. - -However, it is acknowledged that this is somewhat verbose. There is not any -great solution to this problem; see :ref:`pep-828-rejected-ideas` for -discussion about proposed alternatives. In short, ``async yield from`` was -chosen as the best choice of syntax because, while verbose, it is very clear -and readable. +This PEP uses ``yield from`` as the syntax, but it is acknowledged that +this is somewhat controversial, as this is an asynchronous context switch +without an explicit ``async`` or ``await`` marker; see +:ref:`pep-828-rejected-ideas`. In short, ``yield from`` was chosen +as the best choice of syntax because it is symmetric with synchronous +generators and easy to type. Backwards Compatibility @@ -244,7 +220,7 @@ Backwards Compatibility This PEP introduces a backwards-compatible syntax change. The addition of the ``value`` attribute to :exc:`StopAsyncIteration` is a -minor semantic change to an existing builtin exception, but is unlikely +minor semantic change to an existing built-in exception, but is unlikely to affect existing code in practice, as it mirrors the existing ``value`` attribute on :exc:`StopIteration` and does not affect any other behavior on ``StopAsyncIteration`` or the asynchronous iterator protocol. @@ -262,10 +238,8 @@ How to Teach This The details of this proposal will be located in Python's canonical documentation, as with all other language constructs. However, this PEP intends to be very intuitive; users should be able to naturally reach -for ``async yield from`` given their own background knowledge about generators -in Python. This can be encouraged further by suggesting -``async yield from`` in the error message when a user attempts to use -``yield from`` in an asynchronous generator. +for ``yield from`` given their own background knowledge about generators +in Python. Reference Implementation @@ -281,74 +255,74 @@ Rejected Ideas ============== -Using ``yield from`` to delegate to asynchronous generators ------------------------------------------------------------ +Using ``async yield from`` as the syntax +---------------------------------------- -Due to the verbosity of ``async yield from``, it was proposed to overload -the existing ``yield from`` syntax to perform asynchronous subgenerator -delegation when used inside of an asynchronous generator. +There are two primary counterarguments against this proposal as it is currently +written: -For example: +1. It adds behavior that does asynchronous work without having ``async`` or + ``await`` as its first non-whitespace token. +2. It changes the meaning of ``yield from`` depending on the type of generator + it is used in. -.. code-block:: python +The solution to these issues was to introduce a new ``async yield from`` +statement that would perform asynchronous subgenerator delegation. However, +this new syntax came with its own set of problems. - async def asubgenerator(): - yield 1 - yield 2 +First and foremost, ``async yield from`` is very verbose. There are no other +syntax constructs in Python that use three keywords in a row. - async def agenerator(): - yield from asubgenerator() +Second, there is no ``async yield`` as a counterpart; CPython already emits +significantly different bytecode for ``yield`` statements inside asynchronous +generators, so it's not far-fetched for ``yield from`` to do the same. +Third, ``yield`` (as well as ``yield from``) is, technically speaking, already +an asynchronous context switch, because the generator will be suspended and can +only be resumed via an ``await`` (on :meth:`~agen.asend`) from the caller. -This has the benefit of being more concise than ``async yield from``, but also -has a few downsides. +Finally, plain ``yield from`` is more symmetric and intuitive to users of +Python. To visualize, take these two examples: -Most importantly, this makes the asynchronous context switches necessary for -delegation implicit, which has no precedent in Python; all syntax that may -execute an ``await`` is prefixed with ``async``. It has been argued that one -of the upsides of ``async``/``await`` over threads is the explicit switch -points, so hiding ``await``\ s behind a ``yield from`` hurts this benefit. +.. code-block:: python -Second, many were uncomfortable with ``yield from`` being context-dependent. -It felt like a potential footgun for ``yield from`` to mean something -different based on the type of generator it was used in. In practice, this -may come up in a scenario where one wants to convert a synchronous generator -into an asynchronous generator. + def gen(): + yield 1 -For example, imagine a developer is writing a function for streaming data -to the caller: + async def agen(): + yield 1 .. code-block:: python - def stream_data(): - yield ... - yield from something_else() - yield ... + def gen(): + yield from subgen() -Now, imagine that the developer wants to add an ``await`` call somewhere in -this function; the ``yield from something_else()`` statement would suddenly -become a runtime :class:`TypeError` (as opposed to a compile-time -:class:`SyntaxError`). With the current proposal, the existence of -``async yield from`` (which would ideally be included in the error message) -would make it much clearer that ``something_else`` must also be asynchronous -in order to delegate to it. + async def agen(): + yield from asubgen() + + +``async yield from`` would break this symmetry. -Finally, this would preclude the introduction of support for synchronous -subgenerator delegation inside asynchronous generators (see -:ref:`pep-828-synchronous-delegation`), because the ``yield from`` -syntax would already be overloaded. However, the author of this proposal -does acknowledge that the issues with synchronous subdelegation may preclude -the introduction of this anyway -- it is not entirely clear whether the issues -are solvable given time. +However, all in all, the difference between ``yield from`` and +``async yield from`` is primarily up to the taste of the reader. Even the +author of this PEP is not completely convinced for one way or the other. +During discussion, the difference between ``yield from`` and ``async yield from`` +was compared to `deciding what 0^0 should be in mathematics +`_; it depends on the +context and the point of view. The alternative was to not add support for +delegation to asynchronous subgenerators, which was a lose-lose scenario for +all parties involved. A decision had to be made, and it was clear that consensus +was never going to be reached. ``async from``, ``await from``, and similar spellings ------------------------------------------------------ +***************************************************** -As an alternate solution to the verbosity of ``async yield from``, some have +As an alternative solution to the verbosity of ``async yield from``, some have suggested using spellings such as ``async from`` in order to cut down on the verbosity. Unfortunately, changes in the spelling will likely hurt the -readability of the syntax as a whole. +readability of the syntax as a whole, and thus would not improve the argument +for ``async yield from``. The benefit of ``async yield from`` is that it specifies each of the three important parts without introducing new keywords. In particular: @@ -367,10 +341,10 @@ exists. Allowing delegation to synchronous subgenerators ------------------------------------------------ -In an earlier revision of this proposal, the synchronous ``yield from`` -construct was allowed in an asynchronous generator, which would delegate to a -synchronous generator from an asynchronous one. This had a number of hidden -issues. +In an earlier revision of this proposal, ``yield from`` in an asynchronous +generator would delegate to a synchronous generator (and ``async yield from`` +was the counterpart for asynchronous subgenerator delegation). +This had a number of hidden issues. In particular, the mixing of asynchronous frames with synchronous frames had a layer of complexity unfit for Python. In the implementation, there would have @@ -379,7 +353,7 @@ asynchronous generator methods: :meth:`~agen.asend` to :meth:`~generator.send`, :meth:`~agen.athrow` to :meth:`~generator.throw`, and :meth:`~agen.aclose` to :meth:`~generator.close`. -It's trivial for anyone that needs to delegate to a subgenerator to write +It's trivial for anyone who needs to delegate to a subgenerator to write the wrapper class to upgrade a synchronous :class:`~collections.abc.Iterable` or :class:`~collections.abc.Generator` to an async one before calling ``async yield from``. @@ -427,8 +401,8 @@ async one before calling ``async yield from``. async def agen(): - async yield from AsAsyncIterator([1, 2, 3]) - async yield from AsAsyncGenerator(subgen()) + yield from AsAsyncIterator([1, 2, 3]) + yield from AsAsyncGenerator(subgen()) To quote Brandt Bucher (paraphrased): @@ -457,6 +431,11 @@ collecting outside opinions about the design and implementation. Change History ============== +- 18-Jul-2026 + + - Switched from ``async yield from`` to ``yield from`` as the choice of + syntax. + - 26-May-2026 - Removed support for delegating to a synchronous subgenerator (via