Goals for Improving Python

Python is a great language. It's simple enough for learning/scratch projects; and powerful enough to make complex solutions.

However, nothing is perfect. Our worlds are evolving at an exponential rate. The Javascript community has an answer to everything, introducing features + new concepts as they shoulder the responsibility of being a language for every stack. To rub more salt in, key python tools are now being written in Javascript, due to its size, flexibility; but most importantly, the insane power that Typescript provides.

There are 3 main areas in Python that I'd like to improve, and I think they're features the community will appreciate.

The goal is to have PEPs prepared where needed in the next few months, with everything detailed here finished ready for the Python 3.12 release.

Disclaimer: While I've been an avid follower of Python Core, it's rare I get directly involved. Know, though, it has been a lack of time + motivation that has kept me away.

Tagged Templates

We all rejoiced with the introduction of f-strings; however they don't go far enough.

For those that don't know what a Tagged Template is, it's effectively a custom f-string. The main use case here is for providing a more native form of escaping.

Uses

Escaping

>>> evil_href = '#"><s>exit();</s><"'
>>> f'<a href="{evil_href}">Link</a>'
'<a href="#"><s>exit();</s><"">Link</a>'
>>> f'<a href="{html.escape(some_link)}">Link</a>'
'<a href="&quot;&gt;&lt;s&gt;exit();&lt;/s&gt;&lt;&quot;">Link</a>'

# Proposed syntax (backticks)
>>>  html`<a href="{evil_href}">Link</a>`
'<a href="&quot;&gt;&lt;s&gt;exit();&lt;/s&gt;&lt;&quot;">Link</a>'

New gen logging

>>> amount = 10
>>> LOG.warning(f"{amount=}")
>>> LOG.warning`{amount=}`
# logs `amount=10

consider the potential for lazy logging by deferring analysis. In the first version, the string is evaluated before warning is called. however in the second, there's potential to defer the evaluation until the LOG.warning actually tries to emit the message, as LOG.warning uses the expression itself.

Implementation details

Disclaimer: while I have walked the CPython codebase, I'm not familiar with all the related workings

The structure could be similar to that of string.Formatter.parse; however to align with the newer f-string syntax (which has an extra layer ontop of the mini-language).

An alternate is an iterator that emits strings + fields. The iterator determines the underlying grammar (usually f-string or the Format Mini-Language).

# Simplest form, provide utilities to guide development.
# Migration for user is simple, `html("")` to `html`` `

class Element:
  is_expression: bool
  code: Code  # reference to code object of the literal or expression
  evaluate: (self, ns) -> str  # evaluates the code expression


def html(template: str, ns={}):
  string_parts = []
  for element in process(template):
    evaluated = element.evaluate(ns=ns)
    if element.is_expression:
      final_elements.append(escape(evaluated))
    else:
       final_elements.append(evaluated)
  return ''.join(string_parts)

Without knowing the optimisations taken for f-string, this "dumb" approach might not be feasible.

Structural Types + Manipulations

Python is rife with uses of meta-programming to subtly adjust how classes can be used. Django, dataclasses, attrs, pydantic. Just because a metaclass isn't used, doesn't mean it isn't meta-programming.

For something so conceptually simple, not many understand how the data model in python works. This is fine, you really don't need to know it. For whatever reason, Metaclasses don't have much support in mypy or pyright. Instead, they fallback onto using Quirks within their implementations for popular libraries.

My aim is 2 fold:

document how the objects used to build a class tie in together, and write a mypy plugin to codify the assumptions. The goal here is to get to a point where the class body written by the user is exposed to mcls.__new__, which can then output a class with a different body
mypy plugin to demo using Dictionaries directly to do structure manipulation (like Typescript Mapped Types)

After those 2 aims are done, I will work with members of Pyright + Mypy to see if these are appropriate solutions, and how we can iterate from there.

Structure Remapping

To aid with documenting the shape of data, be it class structures or your domain data, we need to be able to preserve context.

A new Typing library

Much of how the typing library works is actually somewhat voodoo; a lot is hidden from us, and exists only as Documentation due to much of the usage being within the type-analysers (mypy, pyright) themselves. This makes deviating from the basics pretty difficult when compared to a feature-rich language, such as Typescript.

A new library will allow for a different take on Typing within Python:

The typing library was designed not designed for much runtime use, primarily for debugging
A focus on "Structures" to define shapes/behaviour will allow features to be iterated on without being tied to Python/mypy/pyright (Avoid SpecialForm)
A focus on building tools for use within the runtime, with an aim for doing more with Macros down the road
It's a chance to fully learn + document how types actually function, and how implementations differ

Summary

It's going to be a busy year over at OpenDIRO; but I hope that once the Discussions are opened up, we can iterate quickly to get community feedback.

Have a good one, and stay tuned for what we'll be doing to extend Pulumi.

Command Palette