Migrating to Stable ABI for Free Threading (`abi3t`)¶

Starting with the 3.15 release, CPython supports a variant of the Stable ABI that supports free-threaded Python: Stable ABI for Free-Threaded Builds, or abi3t for short. This document describes how to adapt C API extensions to support free threading.

Why do this¶

The typical reason to use Stable ABI is to reduce the number of artifacts that you need to build and distribute for each version of your library.

Without the Stable ABI, you must build a separate shared library, and typically a wheel distribution, for each feature version of CPython you wish to support. For example, each tag in the following table represents a separate library/wheel:

CPython version	Non-free-threaded	Free-threaded
3.12	`cpython-312`	—
3.13	`cpython-313`	`cpython-313t`
3.14	`cpython-314`	`cpython-314t`
3.15	`cpython-315`	`cpython-315t`
3.16	`cpython-316`	`cpython-316t`
Future versions	(etc.)	(etc.)

That’s a lot of builds, especially when multiplied by the number of supported platforms.

With the Stable ABI (abi3, introduced in CPython 3.2), a single extension (per platform) can cover all non-free-threaded builds of CPython:

CPython version	Non-free-threaded	Free-threaded
3.12	`abi3`	—
3.13		`cpython-313t`
3.14		`cpython-314t`
3.15		`cpython-315t`
3.16		`cpython-316t`
Future versions		(etc.)

The Stable ABI for free-threaded builds (abi3t), introduced in CPython 3.15, does the same for free-threaded builds. And it’s compatible with non-free-threaded ones as well:

CPython version	Non-free-threaded	Free-threaded
3.12	`abi3` *	—
3.13		`cpython-313t`
3.14		`cpython-314t`
3.15	`abi3t`
3.16
Future versions

* (As above, the abi3 extension is compatible with all non-free-threaded builds; even the 3.15+ ones that this table “attributes” to abi3t.)

Why not do this¶

There are two main downsides to Stable ABI.

First, you extension may become slower, since Stable ABI prioritizes compatibility over performance. The difference is usually not noticeable, and often can be mitigated by using the same source to build both a Stable ABI build and a few version-specific ones for “tier 1” CPython versions.

Second, not all of the C API is available. Extensions need to be ported to build for Stable ABI, which may be difficult or, in rare cases, impossible.

Specifically, abi3t requires on API added in CPython 3.15. If you want to build your extension for older versions of CPython from the same source, you have two main options:

Use preprocessor conditionals.

When following this guide, use #ifdef Py_TARGET_ABI3T blocks whenever you are told to do a change that breaks the build on CPython versions you care about. Keep the pre-existing code in #else blocks.

For hand-written C extensions, this approach is reasonable down to CPython 3.12, due to additions introduced in PEP 697. Keeping compatibility with 3.11 and below may be worth it for code generators (for example, Cython).
Do not port to abi3t, and continue building separate extensions for each version of CPython, until you can drop support for the older versions.

This is a valid approach. Not all extensions need to switch to abi3t right now.

Prerequisites¶

This guide assumes that you have an extension written directly in C (or C++), which you want to port to abi3t.

If your extenstion uses a code generator (like Cython) or language binding (like PyO3), it’s best to wait until that tool has support for abi3t. If you maintain such a tool, you might be able to adapt the instructions here for your tool.

Non-free-threaded Stable ABI¶

Your extension should support the Stable ABI (abi3t). If not, either port it first, or follow this guide but be prepared to fix issues it does not mention.

Free-threading support¶

While it’s technically not a hard prerequisite, you will most likely want to prepare your extension for free threading before you port it to abi3t. See C API Extension Support for Free Threading for instructions.

Isolating Extension Modules¶

Your module should use multi-phase initialization, and it should either be isolated or limit itself to be loaded at most once per process. If it is not your case, follow Isolating Extension Modules first. (See the opt-out section for a shortcut.)

Avoiding variable-sized types¶

If your extension defines variable-sized types (using Py_tp_itemsize or PyTypeObject.tp_itemsize), it cannot be ported to abi3t 3.15.

Setting up the build¶

If you use a build tool (such as setuptools, meson-python, scikit-build-core), search its documentation for a way to select abi3t. At the time of writing, not all of them have this; but if your tool does, use it. You may want to verify that it set the right flag by temporarily adding the following just after #include <Python.h>:

#if Py_TARGET_ABI3T+0 <= 0x30f0000
#error "abi3t define is not set!"
#endif

This should result in a different error than “abt3t define is not set”.

Note

If your build tool doesn’t support abi3t yet, set the following macro before including Python.h:

#define Py_TARGET_ABI3T 0x30f0000

or specify it as a compiler flag, for example:

-DPy_TARGET_ABI3T=0x30f0000

Once your extension builds with this setting, it will be compatible with CPython 3.15 and above.

If you set this macro manually, you will later need to name and tag the resulting extension manually as well. This is covered in Tagging and distribution below.

This guide will ask you to do a series of changes. After each one, verify that your extension still builds in the original (non-abi3t) configuration, and ideally run tests on all Python versions you support. This will ensure that nothing breaks as you are porting.

Module export hook¶

Unless you’ve done this step already, your extension module defines a module initialization function named PyInit_<module_name>. You will need to port it to module export hook, PyModExport_<module name>, a feature added in CPython 3.15 in PEP 793.

Your existing init function should look like this (with your own names for <modname> and <moddef>):

PyMODINIT_FUNC
PyInit_<modname>(void)
{
    return PyModuleDef_Init(&<moddef>);
}

If there is some code before the return, move it to a Py_mod_create or Py_mod_exec slot function. See PyInit documentation for related information.

The function references a PyModuleDef object (<moddef> in the code above). Its definition should be similar to the following, with different values and perhaps some fields unnnamed or left out:

static PyModuleDef <moddef> = {
    PyModuleDef_HEAD_INIT,
    .m_name = "my_module",
    .m_doc = "my docstring",
    .m_size = sizeof(my_state_struct),
    .m_methods = my_methods,
    .m_slots = my_slots,
    .m_traverse = my_traverse,
    .m_clear = my_clear,
    .m_free = my_free,
};

Remove this definition and the PyInit function (or put them in an #ifndef Py_TARGET_ABI3T block, to retain backwards compatibility), and replace them with the following:

PyABIInfo_VAR(abi_info);

static PySlot my_slot_array[] = {
    PySlot_STATIC_DATA(Py_mod_abi, &abi_info),
    PySlot_STATIC_DATA(Py_mod_name, "my_module"),
    PySlot_STATIC_DATA(Py_mod_doc, "my docstring"),
    PySlot_SIZE(Py_mod_state_size, sizeof(my_state_struct)),
    PySlot_STATIC_DATA(Py_mod_methods, my_methods),
    PySlot_STATIC_DATA(Py_mod_slots, my_slots),
    PySlot_FUNC(Py_mod_state_traverse, my_traverse),
    PySlot_FUNC(Py_mod_state_clear, my_clear),
    PySlot_FUNC(Py_mod_state_free, my_free),
    PySlot_END
};

PyMODEXPORT_FUNC
PyModExport_<modname>(void)
{
    return my_slot_array;
}

Leave out any fields that were missing (excexpt the new Py_mod_abi), and substitute your own values.

See PySlot and export hook documentation for details on this API.

Associated `PyModuleDef`¶

Since the new API does not use a PyModuleDef structure, a definition will not be associated with the resulting module. This changes the behavior of the following functions:

Check your code for these. If you do not use them, you skip the rest of this section.

These functions are typically used for two purposes:

To get the definition the module was created with. This is no longer possible using the new API. Modules no longer keep a reference to the definition, so you will need to figure out a different way to pass the relevant data around.

To check if a given module object is “yours”. This use case is now served by module tokens – opaque pointers that identify a module. To use a token, declare (or reuse) a unique static variable, for example:
```
static char my_token;
```
and add a pointer to it in a new entry to your module’s PySlot array:
```
static PySlot my_slot_array[] = {
   ...
   PySlot_STATIC_DATA(Py_mod_token, &my_token),
   PySlot_END
}
```
Then, switch from PyModule_GetDef() calls such as:
```
PyModuleDef *def = PyModule_GetDef(module);
```
to PyModule_GetToken() (which uses an output argument and may fail with an exception):
```
void *token;
if (PyModule_GetToken(module, &token) < 0) {
   /* handle error */
}
```
and from PyType_GetModuleByDef() calls such as:
```
PyObject *module = PyType_GetModuleByDef(type, my_def);
/* handle error; use module */
```
to PyType_GetModuleByToken() (which returns a strong reference):
```
PyObject *module = PyType_GetModuleByToken(type, my_token);
/* handle error; use module */
Py_XDECREF(module);
```

`PyObject` opaqueness¶

The PyObject and PyVarObject structures are opaque in abi3t.

Accessing their members is prohibited. If you do this, switch to getter/setter functions mentioned in their documentation:

Also, the size of the PyObject structures is unknown to the compiler. It can – and does – change between different CPython builds.

Note

While the size is available at runtime (for example as sys.getsizeof(object()) in Python code), you should resist the temptation to calculate pointer offsets from it. The object memory layout is subject to change in future abi3t implementations.

Custom type definitions¶

Since PyObject is opaque, the traditional way of defining custom types no longer works:

typedef struct {
   PyObject_HEAD  // expands to `PyObject ob_base;` which has unknown size

   int my_data;
} CustomObject;

static PyType_Spec CustomType_spec = {
   ...
   .basicsize = sizeof(CustomObject),
   ...
};

Most likely, all your class definitions, and all code that accesses your classes’ data, will need to be rewritten. This will probably be the biggest change you need to support abi3t.

For each such type:

Instead of defining a struct for the entire instance, define one with only the “additional” fields – ones specific to your class, not its superclasses:

typedef struct {
   int my_data;
} CustomObjectData;

Change the name. Almost all code that uses the struct will need to change (notably, pointers to the new structure cannot be cast to/from PyObject*), and changing the name will highlight the usages as compiler errors. (If you use typeof, C++ auto, or similar ways to avoid typing the type name, this won’t work. Be extra careful, and consider running tools to detect undefined behavior.)

Then, to create the class, use negative basicsize to indicate “extra” storage space rather than total instance size:

static PyType_Spec CustomType_spec = {
   ...
   .basicsize = -sizeof(CustomObjectData), /* note the minus sign */
   ...
};

If you use Py_tp_members, set the Py_RELATIVE_OFFSET flag on each member and specify the offset relative to your new struct.

Custom type data access¶

Then comes the hard part: in all code that needs to access this struct, you will need an additional PyObject_GetTypeData() call to retrieve a CustomObjectData * pointer from PyObject *:

PyObject *obj = ...;
CustomObjectData *data = PyObject_GetTypeData(obj, cls);

Note that this call requires the type object for your class (cls).

If your class is not subclassable (that is, it does not use the Py_TPFLAGS_BASETYPE flag), cls will be Py_TYPE(obj). Otherwise, DO NOT USE Py_TYPE with PyObject_GetTypeData(): it might return memory reserved to an unrelated subclass! For example, if a user makes a subclass like this:

class Sub(YourCustomClass):
   __slots__ = ('a', 'b')

then Py_TYPE(obj) is YourCustomClass, and the underlying memory may look like this:

╭─ PyObject *obj
│              ╭─ the pointer you want
│              │                    ╭─ PyObject_GetTypeData(obj, Py_TYPE(obj))
▼              ▼                    ▼
┌──────────┬───┬────────────────┬───┬─────────────┬───┬─────────────┐
│ PyObject │...│ CustomTypeData │...│ PyObject *a │...│ PyObject *b │
└──────────┴───┴────────────────┴───┴─────────────┴───┴─────────────┘

(Ellipses indicate possible padding. Note that this memory layout is not guaranteed: future versions of Python may add different padding or even switch the order of the structures.)

There are two main ways to get the right class:

In instance methods, your implementation may use the PyCMethod signature (and the METH_METHOD bit in PyMethodDef.ml_flags), and get the class as the defining_class argument.

Otherwise, give your class a unique static token using the Py_tp_token slot, and use:

PyTypeObject cls;
if (PyType_GetBaseByToken(Py_TYPE(obj), my_tp_token, &cls) < 0) {
    /* handle error */
}
CustomObjectData *data = PyObject_GetTypeData(obj, cls);

Type tokens work similarly to module tokens covered earlier in this guide.

Avoid build-time conditionals¶

Check your code for API that identifies the version of Python used to build your extension. This no longer corresponds to the Python your extension runs on, so code that uses this information often needs changing. The macros to check for are:

PY_VERSION_HEX, PY_MAJOR_VERSION, PY_MINOR_VERSION:
- to get the run-time version, use Py_Version;
- to determine what C API is available, use Py_TARGET_ABI3T. This macro is set to the minimum supported version.
Py_GIL_DISABLED: under abi3t, this macro is always defined. Code that works with free-threaded Python should also work with the GIL enabled (since the GIL can be enabled at run time), and usually does (unless it, for some reason, requires more than one attached thread state at one time).

Further code changes¶

If you are still left with compiler errors or warnings, find a way to fix them. Alas, this guide is limited, and cannot cover all possible code changes extensions may need.

If you find a problem that other extension authors might run into, consider submitting an issue (or pull request) for this guide.

It is possible your issue cannot be fixed for the current version of abi3t. In that case, reporting it may help it get prioritized for the next version of CPython.

Tagging and distribution¶

If you are using a build tool with abi3t support, your extension is ready, but you might want to check that it was built correctly.

Extensions built with abi3t should have the following extension:

On Windows: .pyd (like any other extension);
Linux, macOS, and other systems that use the .so suffix: .abi3t.so (not .cpython-315t.so or .abi3.so). Note that both free-threaded and non-free-threaded builds will load .abi3t.so extensions;
Other systems: consult your distributor, and perhaps update this guide.

If you distribute the extension as a wheel, use the following tags:

Python tag: cp3YY, where YY is the minimum Python version the extension is built for. (For example, cp315 if you set Py_TARGET_ABI3T to 0x30f0000. See Compiling for Stable ABI for more values.)
ABI tag: abi3.abi3t. This is a compressed tag set that indicates support for both non-free-threaded and free-threaded builds.

For example, the wheel filename may look like this:

myproject-1.0-cp315-abi3.abi3t-macosx_11_0_arm64.whl

Testing¶

Note that when you build an extension compatible with multiple versions of CPython, you should always test it witch each version it supports (for example, 3.15, 3.16, and so on). Stable ABI only guarantees ABI compatibility; there may also be behavior changes – both intentional ones (covered by PEP 387) and bugs.

Be sure to run tests on both free-threaded and non-free-threaded builds of CPython.

If they pass, congratulations! You have an abi3t extension.

Migrating to Stable ABI for Free Threading (`abi3t`)¶

Why do this¶

Why not do this¶

Prerequisites¶

Non-free-threaded Stable ABI¶

Free-threading support¶

Isolating Extension Modules¶

Avoiding variable-sized types¶

Setting up the build¶

Module export hook¶

Associated `PyModuleDef`¶

`PyObject` opaqueness¶

Custom type definitions¶

Custom type data access¶

Avoid build-time conditionals¶

Further code changes¶

Tagging and distribution¶

Testing¶

Table of Contents

Previous topic

Next topic

This page

Migrating to Stable ABI for Free Threading (abi3t)¶

Why do this¶

Why not do this¶

Prerequisites¶

Non-free-threaded Stable ABI¶

Free-threading support¶

Isolating Extension Modules¶

Avoiding variable-sized types¶

Setting up the build¶

Module export hook¶

Associated PyModuleDef¶

PyObject opaqueness¶

Custom type definitions¶

Custom type data access¶

Avoid build-time conditionals¶

Further code changes¶

Tagging and distribution¶

Testing¶

Migrating to Stable ABI for Free Threading (`abi3t`)¶

Associated `PyModuleDef`¶

`PyObject` opaqueness¶