PEP 823: None-aware access operators

• Abstract
• Terminology
• Motivation Nested objects with optional attributes Parsing structured data Other common patterns
• Nested objects with optional attributes
• Parsing structured data
• Other common patterns
• Specification The None-aware access operators Short-circuiting Parenthesized expressions - groupings Assignments Await expressions AST changes Grammar changes Multiline formatting
• The None-aware access operators Short-circuiting Parenthesized expressions - groupings Assignments Await expressions
• Short-circuiting
• Parenthesized expressions - groupings
• Assignments
• Await expressions
• AST changes
• Grammar changes Multiline formatting
• Multiline formatting
• Backwards Compatibility
• Security Implications
• How to Teach This
• Reference Implementation
• Deferred Ideas Coalesce ?? and coalesce assignment operator ??= None-aware function calls None-assertion operator Add list.get(key)
• Coalesce ?? and coalesce assignment operator ??=
• None-aware function calls
• None-assertion operator
• Add list.get(key)
• Rejected Ideas Exception-aware operators Add a maybe keyword Remove short-circuiting ? Unary Postfix operator Builtin function for traversal Maybe function Result object No-Value Protocol Use existing syntax or keyword Defer None-aware indexing operator Ignore groups for short-circuiting Change primary rule to be right-recursive
• Exception-aware operators
• Add a maybe keyword
• Remove short-circuiting
• ? Unary Postfix operator
• Builtin function for traversal
• Maybe function
• Result object
• No-Value Protocol
• Use existing syntax or keyword
• Defer None-aware indexing operator
• Ignore groups for short-circuiting
• Change primary rule to be right-recursive
• Common objections Difficult to read Easy to get ?. wrong Not obvious what ?. and ?[ ] do ?. and ?[ ] should handle missing attributes Short circuiting is difficult to understand Just use … … a conditional expression … a match statement … try … except … … a traversal library Proliferation of None in code bases None is not special enough ? last available ASCII character
• Difficult to read
• Easy to get ?. wrong
• Not obvious what ?. and ?[ ] do
• ?. and ?[ ] should handle missing attributes
• Short circuiting is difficult to understand
• Just use … … a conditional expression … a match statement … try … except … … a traversal library
• … a conditional expression
• … a match statement
• … try … except …
• … a traversal library
• Proliferation of None in code bases
• None is not special enough
• ? last available ASCII character
• Footnotes
• Copyright

Abstract

This PEP proposes adding two new operators.

• The “None-aware attribute access” operator ?.
• The “None-aware indexing” operator ?[ ]

The general idea is to provide access operators which can traverse None values without raising exceptions.

Both operators evaluate the left-hand side, check if it is not None and only then evaluate the full expression. They are roughly equivalent to:

Terminology

Motivation

First officially proposed over ten years ago in (the now deferred) PEP 505, the idea to add None-aware access operators has been along for some time now, discussed at length in numerous threads, most recently in [1] and [2]. This PEP aims to capture the current state of discussion and proposes a specification for addition to the Python language. In contrast to PEP 505, it will only focus on the two access operators. See the Deferred Ideas section for more details.

None-aware access operators are not a new invention. Several other modern programming languages have so called “null-aware” or “optional chaining” operators, including TypeScript [3], ECMAScript (a.k.a. JavaScript) [5], C# [7], Dart [9], Swift [10], Kotlin [11], Ruby [13], PHP [14] and more.

The general idea is to provide access operators which can traverse None values without raising exceptions.

Nested objects with optional attributes

When writing Python code, it is common to encounter objects with optional attributes. Accessing attributes, subscripts or function calls can raise AttributeError or TypeError at runtime if the value is None. To ensure these operations will not raise, a common option is to add conditional is not None checks. Even for fairly simple objects, this often adds nesting and code duplication. Consider the following simplified example:

This could also be written using assignment expressions:

While both options are correct, it requires more effort than necessary to understand the simple checks being added here. Additionally, even writing these statements and expressions can become quite cumbersome, to a point that it is not uncommon to skip certain parts: be it the explicit is not None and instead defaulting to implicit boolean checks or the assignments to temporary variables and instead repeating the previous expressions.

The goal for ?. and ?[ ] is to make reading and writing these kinds of expressions much simpler while being predictable and doing the correct things intuitively. Using these operators, the function could instead be written as:

Here ? are inserted after each optional subexpression to change the attribute access to “None-aware attribute access” operators. In contrast to before, it is not necessary to add additional if statements, assignments or nesting, so starting from “normal” attribute access operators and changing these to ?. where necessary afterwards, is an easy way to write these expressions.

Parsing structured data

The ?. and ?[ ] operators can also aid in the traversal of structured data, oftentimes coming from JSON and parsed as nested dicts and lists. It is worth noting though that the operators do NOT handle missing attributes / data. In these cases, at least for dictionaries a useful helper method is dict.get(key).

Writing it using ?. and ?[ ] would look like this:

Other common patterns

A collection of additional patterns which could be improved with ?. and ?[ ]. It is not the goal to list every foreseeable use case but rather to help recognize these patterns which often hide in plain sight. Attribute and function names have been shortened.

Note

Most patterns below are not fully identical. As mentioned earlier, it is common to use boolean expressions to filter out None values. Other falsy values, e.g. False, "", 0, [], {} or custom objects which overwrite __bool__, are filtered out too though. If code relied on this property, the expression cannot necessarily be replaced with ?. or ?[ ].

Specification

The None-aware access operators

Two new operators are added, ?. and ?[ ]. Both operators first evaluate the left-hand side (the base). The result is cached, so that the expression is not evaluated again. It is checked if the result is not None and only then is the remaining expression (the tail) evaluated as if normal attribute or subscript access were used.

The base can be replaced with any number of expressions, including Parenthesized ones while the tail is limited to attribute access, subscript, their None-aware variants and call expressions.

Short-circuiting

If the left-hand side (the base) for ?. or ?[ ] evaluates to None, the remaining expression (the tail) is skipped and the result will be set to None instead. This comprises everything in the tail part, including the evaluation of function arguments or subscripts. The AttributeError for accessing a member of None or TypeError for trying to get a subscript of None are omitted. It is therefore not necessary to change subsequent . or [ ] on the right-hand side just because a ?. or ?[ ] is used prior.

The None-aware access operators will only short-circuit expressions containing primary expressions (name, attribute access, subscript, their None-aware counterparts, and call expressions). As a rule of thumb, short-circuiting is broken once an operator other than ., [ ], ?., ?[ ] is reached.

Another way to look at this is to ask whether a part of an expression could be extracted and defined as a variable without changing the meaning of it. If that is the case, short-circuiting will likely be broken. For example function arguments or subscripts are evaluated on their own and would not short-circuit the remaining tail of the outer expression.

Parenthesized expressions - groupings

Using ?. and ?[ ] inside groups is possible. In addition to the rules laid out in the previous section, short-circuiting will also be broken at the end of a group. For example the expression (a?.b).c will raise an AttributeError on .c if a is None. This is conceptually identical to extracting the group contents and storing the result in a temporary variable before substituting it back into the original expression.

Common use cases for None-aware access operators in groups are boolean or conditional expressions which can provide a fallback value in case the first part evaluates to None.

Assignments

None-aware expressions may only be used in a Load context. Assignments are not permitted and will raise a SyntaxError.

This does not apply if the None-aware expressions is only part of a larger expression and evaluated on its own, for example as a function argument.

Await expressions

None-aware access operations are permitted in await expressions. It is up to the developer to make sure they do not evaluate to None at runtime, otherwise a TypeError is raised. This behavior is similar to awaiting any other variable which can be None.

AST changes

Two new AST nodes are added NoneAwareAttribute and NoneAwareSubscript. They are the counterparts to the existing Attribute and Subscript nodes. Notably there is no expr_context attribute because the new nodes do not support assignments themselves and thus the context will always be Load. Furthermore, an optional group attribute is added for all expression nodes. It is set to 1 if an expression is the topmost node in a group, 0 otherwise.

Grammar changes

A new ? token is added. In addition the primary grammar rule is updated to include none_aware_attribute and none_aware_subscript.

Multiline formatting

Using two separate tokens to express ?. and ?[ allows developers to insert a space or line break as needed. For multiline expressions it enables that ? is appended to the optional subexpression whereas . or [ could be moved to the next line. This is intended merely as an option for developers. Everyone is free to choose a style that fits their needs, especially code formatters might prefer a style which conforms better to their existing preferences. An example of what is possible:

Backwards Compatibility

Existing programs will continue to run as is. So far code which used either ?. or ?[ ] raised a SyntaxError.

Security Implications

There are no new security implications from this proposal.

How to Teach This

After students know how the attribute access . and subscript [ ] operators work, they may learn about the “None-aware” variants for both.

Students may find it helpful to think of ?. and ?[ ] as a combination of two different actions. First the ? postfix represents an is not None check on the subexpression with short-circuiting if the check fails. If it succeeds, the attribute and subscript access are performed like normal.

Experienced developers may find that, after learning about the PEP, they start to notice the patterns described in the Motivation section in their own code bases.

Reference Implementation

A reference implementation is available at https://github.com/cdce8p/cpython/tree/pep823-none-aware-access-operators. An online demo can be tested at https://pep823-and-pep824-demo.pages.dev/.

Deferred Ideas

Coalesce ?? and coalesce assignment operator ??=

PEP 505 also suggested the addition of a “None coalescing” operator ?? and a “None coalescing assignment” operator ??=. As the None-aware access operators have their own use cases, the coalescing operators were moved into a separate document, see PEP 824. Both proposals can be adopted independently of each other.

None-aware function calls

The None-aware access operators work for attribute and index access. It seems natural to ask if there should be a variant which works for function invocations. It might be written as a.foo?() which would be equivalent to:

This has been deferred on the basis that the proposed operators are intended to help for nested objects with optional attributes and the parsing of structured data, not the traversal of arbitrary class hierarchies.

A workaround would be to write a.foo?.__call__(arguments).

None-assertion operator

Several programming languages with “null-aware” operators also have a “not-null assertion” operator !, for example TypeScript [4], C# [8], Dart [9] and Kotlin [12]. It was proposed to include a “None-assertion” operator ! here. This would be especially useful in typed Python code where type checkers are not able to statically deduce that a variable, with an optional (can be None) value, cannot be None in some contexts.

This PEP focuses on the None-aware access operators and therefore this proposal is out of scope.

Add list.get(key)

It was suggested to add a .get(key) method to list and tuple objects, similar to the existing dict.get method. This could further make parsing of structured data easier since it would no longer be necessary to check if a list or tuple is long enough before trying to access the n-th element avoiding a possible IndexError. While potentially useful, the idea is out of the scope for this PEP.

Rejected Ideas

Exception-aware operators

Arguably, the reason to short-circuit an expression when None is encountered is to avoid the AttributeError or TypeError that would be raised under normal circumstances. Instead of testing for None, it was suggested that ?. and ?[ ] could instead handle AttributeError and TypeError and skip the remainder of the expression. Similar to nested try-except blocks.

While this would technically work, it’s not at all clear what the result should be if an error is caught. Furthermore, this approach would hide genuine issues like a misspelled attribute which would have raised an AttributeError. There are also already established patterns to handle these kinds of errors in the form of getattr and dict.get(key).

As catching exceptions would be unexpected and hide potential errors, it is rejected.

Add a maybe keyword

The None-aware access operators only check for None in the place there are used. If multiple attributes in an expression can return None, it might be necessary to add them multiple times a?.b.c?[0].d?.e(). It was suggested to instead add a new soft-keyword maybe to prefix the expression: maybe a.b.c[0].d.e(). A None check would then be added for each attribute and item access automatically.

While this might be easier to write at first, it introduces new issues. When using explicit ?. and ?[ ] operators, the input space is well defined. Only a, .c and .d are expected to possibly be None. If .b all of the sudden is also None, it would still raise an AttributeError since it was unexpected. That would not happen for maybe. This behavior is problematic since it can subtly hide real issues. As the expression output can already be None, the space of potential outputs did not change and as such no error would appear.

If it is the intent to catch all AttributeError and TypeError, a try-except block can be used instead.

As the ?. and ?[ ] would allow developers to be more explicit in their intent, this suggestion is rejected.

Remove short-circuiting

It was suggested to remove the Short-circuiting behavior completely because it might be too difficult to understand. Developers should instead change any subsequent attribute access or subscript to their None-aware variants.

The idea has some of the same challenges as Add a maybe keyword. By forcing the use of ?. or ?[ ] for attributes which are not-optional, it will be difficult to know if the not-optional attributes .c or .d suddenly started to return None as well. The AttributeError would have been silenced.

Another issue especially for longer expressions is that all subsequent attribute access and subscript operators need to be changed as soon as just one attribute in a long chain is optional. Missing just one can instantly cause a new AttributeError or TypeError.

? Unary Postfix operator

To generalize the None-aware behavior and limit the number of new operators introduced, a unary, postfix operator ? was considered. ?. or ?[ ] could then be considered to be two separate operators.

While this might have made teaching the operators a bit easier, just one instead of two new operators, it may also be too general, in a sense that it can be combine with any other operator. For example it is not clear what the following expressions would mean:

Even if a default meaning of is not None else None is assumed, the expressions are likely to raise errors at some point.

This degree of generalization is not useful. The None-aware access operators where intentionally chosen to make it easier to access values in nested objects with optional attributes.

If future PEPs want to introduce new operators to access attributes or call methods, e.g. a chaining operator, it would be advisable to consider if a None-aware variant for it could be useful, at that time.

Builtin function for traversal

There are a number of libraries which provide some kind of object traversal functions. The most popular likely being glom [15]. Others include jmespath [16] and nonesafe [17]. The idea is usually to pass an object and the lookup attributes as string to a function which handles the evaluation. It was suggested to add a traverse or deepget function to the stdlib.

While these libraries do work and have its use cases, especially glom provides an excellent interface to extract and combine multiple data points from deeply nested objects, they do also have some disadvantages. Passing the lookup attributes as a string means that often times there are no more IDE suggestions. Type checking these expressions is also limited. Furthermore, normal function calls can not provide short-circuiting, so they would still need to be combined with assignment and conditional expressions.

Maybe function

Another suggestion was to add a maybe function which would return either an instance of Something or an instance of Nothing. Nothing would override the dunder methods in order to allow chaining on optional attributes.

A Python package called pymaybe [18] provides a rough approximation. An example could look like this:

While this could work, Something and Nothing are only wrapper classes for the actual values which adds its own challenges. For example to filter out None in a subsequent operation an is not None check would always return True and instead .is_some() would need to be used. This would make adopting it across a large codebase difficult and limit its usefulness. Additionally any pure Python implementation can not really short-circuit the expression. The best it can do is to implement no-ops on the wrapper classes.

As such a builtin maybe function to support accessing nested objects with optional attributes is rejected.

Result object

It was suggested to introduce a Result object similar to how asyncio.Future works today. Expressions marked with a special keyword or syntax would then return an instance of Result instead of the evaluated expression. The actual value could then be retrieved by calling .result() or .exception() on it. With that it could be possible to gracefully handle None-aware expression as well.

While this is an interesting idea, it would be a disruptive change how expressions need to be written and evaluated today.

An advantages of the ?. and ?[ ] operators is that they do not change the result much aside from adding None as a possible return value of an expression. As such they are a better solution for the use cases outlined in the Motivation section.

No-Value Protocol

The None-aware access operators could be generalized to user-defined types by defining a protocol to indicate when a value represents “no value”. Such a protocol may be a dunder method __has_value__(self) that returns True if the value should be treated as having a value and False if the value should be treated as no value.

In the specification section, all uses of x is not None would be replaced with x.__has_value__().

There are a few obvious candidates like math.nan and NotImplemented. However, while these could be interpreted as representing no value, the interpretation is domain specific. For the language itself they should still be treated as values. For example math.nan.imag is well defined (it is 0.0) and so short-circuiting math.nan?.imag to return None would be incorrect.

As None is already defined by the language as being the value that represents “no value” the idea is rejected.

Use existing syntax or keyword

Some comments suggested to use existing syntax like -> for the None-aware access operators, e.g. a->b.c.

Though possible, the -> operator is already used in Python for something completely different. Additionally, a majority of other languages which support “null-aware” or “optional chaining” operators use ?.. Some exceptions being Ruby [13] with &. or PHP [14] with ?->. The ? character does not have an assigned meaning in Python just yet. As such it makes sense to adopt the most common spelling for the None-aware access operators. Especially considering that it also works well with the “normal” . and [ ] operators.

Defer None-aware indexing operator

A point of discussion was the ?[ ] operator. Some thought it might be missed to easily in an expression a.b?[c]. To move the discussion forward, it was suggested to defer the operator for later.

Though it is often helpful to reduce the scope to move forward at all, the ?[ ] operator is necessary to efficiently get items from optional objects. While for dictionaries a suitable alternative is to use d?.get(key), for general objects developers would have needed to defer to o?.__getitem__(key).

Furthermore, any future PEP just for a ?[ ] would likely have needed to included a lot of the arguments and objections listed in this one again. As such it makes sense to include both operators in the same PEP.

While adding list.get(key) as suggested in Add list.get(key) would reduce the need for ?[ ] for lists and tuples and as such would be a valuable addition to the language itself, it does not remove the need for arbitrary objects which implement a custom __getitem__ method.

Ignore groups for short-circuiting

An earlier version of this PEP suggested the short-circuiting behavior should be indifferent towards grouping. It was assumed that short-circuiting would be broken already for more complex group expressions like (a?.b or c).d by the behavior outline in the Short-circuiting section, while for simpler ones like (a?.b).c the grouping was considered trivial and the expression would be equal to a?.b.c. The advantage being that developers would not have to look for groupings when evaluating simpler expressions. As long as any None-aware access operator was used and the expression was not broken by any other unrelated operator, it would return None instead of raising an AttributeError or TypeError.

This suggestion was rejected in favor of the specification outline in the Parenthesized expressions - groupings section since it violates the substitution principle. An expression (a?.b).c should behave the same whether or not a?.b is written inline inside a group or defined as a separate variable.

Furthermore, defining the short-circuiting behavior that way would have been a deviation from the already established behavior in languages like JS [6] and C# [7].

Change primary rule to be right-recursive

The primary grammar rule as it is defined [19] is left-recursive. This can make it difficult to reason about especially with regards to the Short-circuiting and Parenthesized expressions behavior.

It was therefore proposed to make the None-aware access operators part of the primary rule right-recursive instead. The expression a.b?.c[0].func() would then roughly be parsed as:

In comparison the proposed Grammar changes are intentional kept to a minimum. The None-aware access operators should behave more or less like a drop-in replacement for . and [ ], only with the behavior outline in this PEP.

Common objections

Difficult to read

Easy to get ?. wrong