Allow building in free-threaded python - #2286
Conversation
fbb512a to
48ae917
Compare
e16603d to
cc3a8c0
Compare
cc3a8c0 to
b882b77
Compare
maresb
left a comment
There was a problem hiding this comment.
I had my agent give it a try on conda-forge and here's the report.
I did an independent check of this from the conda-forge side — built pytensor for free-threaded CPython 3.14 (python_abi 3.14.* *_cp314t, OpenBLAS instead of MKL, linux-64) and probed the GIL state in fresh processes. This was against released 3.2.3 rather than this branch, but since the PR carries no runtime changes the behavior should be identical.
The premise checks out. The default path really does stay GIL-free:
| action | sys._is_gil_enabled() after |
|---|---|
import pytensor |
False ✅ |
default-mode pytensor.function(...) |
False ✅ |
mode="NUMBA" |
False ✅ |
mode="FAST_COMPILE" |
True ← imports lazylinker_ext |
mode="C" |
True (expected) |
scan, default / FAST_RUN |
False ✅ |
scan, mode="C" |
True ← imports scan_perform |
Importing PyTensor itself never flips it — the flip only happens at graph-compile time, or on a direct import of one of the compiled extensions. Nice result, and worth knowing it's confirmed on a real free-threaded build rather than just in principle.
Three things I ran into that might be worth addressing:
1. pytest_sessionfinish can't currently see PyTensor's own C extensions
The smoke-test job sets PYTENSOR_FLAGS=…,cxx=, which means lazylinker_ext / cutils_ext can never be compiled or imported in the first place. So the hook can't fail on the thing its comment describes ("some test pulled in PyTensor's own C extensions") — the configuration structurally prevents it.
It isn't doing nothing: it still verifies that numpy / scipy / numba stay GIL-free on 3.14t, which is a genuinely useful regression check on the dependency stack. But the PyTensor-side regression — the default backend starts touching a C extension again — would pass silently.
Running the suite without cxx= isn't the answer, since tests that deliberately use mode="C" would trip the hook. A cheap alternative would be a standalone smoke script run in the default configuration (no cxx= override):
import sys, sysconfig
assert sysconfig.get_config_var("Py_GIL_DISABLED") == 1
import pytensor, pytensor.tensor as pt
x = pt.vector("x")
pytensor.function([x], pt.exp(x).sum())([1.0, 2.0, 3.0])
assert not sys._is_gil_enabled(), "default backend re-enabled the GIL"That's exactly the user-facing contract this PR is asserting, and it's the part currently untested.
2. FAST_COMPILE re-enabling the GIL is surprising
It's a py-linker mode, so I'd expect it to stay GIL-free — but with a compiler available it pulls in lazylinker_ext and the GIL comes back. Someone debugging on a free-threaded build would likely reach for FAST_COMPILE and silently lose free-threading with no indication beyond the interpreter's warning.
Worth noting that the cibuildwheel test-command here uses FAST_COMPILE, and stays GIL-free only because it also disables the C backend. Might be worth either a documented caveat or making the py linker skip the CVM when Py_GIL_DISABLED is set.
3. scan_perform — intended to stay GIL-enabling, or eventually declare compatibility?
pytensor/scan/scan_perform.cpython-314t-*.so builds fine for 3.14t, but importing it re-enables the GIL. It's currently unreachable in CI: it's only imported on the C/CVM path in Scan.perform (pytensor/scan/op.py), which cxx= disables, and the smoke test runs only tests/tensor/test_math_scipy.py.
Since it's a Cython extension, declaring compatibility is mechanically small — # cython: freethreading_compatible=True plus Cython ≥ 3.1 — but obviously only after deciding the code is actually safe without the GIL. Given the PR description's position that the C extensions don't claim thread safety, permanently accepting "C path ⇒ GIL on" is perfectly coherent; I just wanted to check which it is, since it determines whether scan is usable at all in a free-threaded context.
Bonus: the mkl-service exclusion may not be permanent
Your workaround matches what I found independently — mkl-service has no cp3*t builds anywhere. FWIW IntelPython/mkl-service#213 is open and does the right thing (adds freethreading_compatible=True to the .pyx and PyUnstable_Module_SetGIL(m, Py_MOD_GIL_NOT_USED) to _mklinitmodule.c). Its stated blocker, IntelPython/mkl-service#174, merged on 2026-07-21, so it's just awaiting review. If it lands, the mkl/mkl-service exclusion here could eventually be dropped.
For context on the conda-forge side: none of this blocks packaging. mkl-service is only pulled in for an obsolete MKL-2018 check in check_mkl_openmp() whose exception is caught and debug-logged, so dropping it for the free-threaded variant costs nothing. The build itself needed no recipe changes beyond swapping the BLAS provider.
Now that C-backend is not the default, there's no good reason to prevent free-threaded builds.
Our C extensions are not thread-safe, but they don't claim to be, so if imported python will just re-enable the gil with a warning.
For users, care must be taken to split the memory and shared variables of a compiled function
fn.copy(share_memory=False, swap={})before threading, but that's not really new. See pymc-devs/pymc#8355 for an exampleAlso helped by #2285