feat: Embeded sub-interpreters (#5666)

* First draft a subinterpreter embedding API

* Move subinterpreter tests to their own file

* Migrate subinterpreter tests to use the new embedded class.

* Add a test for moving subinterpreters across threads for destruction

And find a better way to make that work.

* Code organization

* Add a test which shows demostrates how gil_scoped interacts with sub-interpreters

* Add documentation for embeded sub-interpreters

* Some additional docs work

* Add some convenience accessors

* Add some docs cross references

* Sync some things that were split out into #5665

* Update subinterpreter docs example to not use the CPython api

* Fix pip test

* style: pre-commit fixes

* Fix MSVC warnings

I am surprised other compilers allowed this code with a deleted move ctor.

* Add some sub-headings to the docs

* Oops, make_unique is C++14 so remove it from the tests.

* I think this fixes the EndInterpreter issues on all versions.

It just has to be ifdef'd because it is slightly broken on 3.12, working well on 3.13, and kind of crashy on 3.14beta.  These two verion ifdefs solve all the issues.

* Add a note about exceptions.

They contain Python object references and acquire the GIL, that means they are a danger with subinterpreters!

* style: pre-commit fixes

* Add try/catch to docs examples to match the tips

* Python 3.12 is very picky about this first PyThreadState

Try special casing the destruction on the same thread.

* style: pre-commit fixes

* Missed a rename in a ifdef block

* I think this test is causing problems in 3.12, so try ifdefing it to see if the problems go away.

* style: pre-commit fixes

* Document the 3.12 constraints with a warning

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* ci: add cpptest to the clang-tidy job

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

* noexcept move operations

* Update include/pybind11/subinterpreter.h

std::memset

Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>

---------

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>

.github/workflows/format.yml

@@ -55,3 +55,6 @@ jobs:
 
     - name: Build
       run: cmake --build build -j 2 -- --keep-going
+
+    - name: Embedded
+      run: cmake --build build -t cpptest -j 2 -- --keep-going

CMakeLists.txt

@@ -223,6 +223,7 @@ set(PYBIND11_HEADERS
     include/pybind11/operators.h
     include/pybind11/pybind11.h
     include/pybind11/pytypes.h
+    include/pybind11/subinterpreter.h
     include/pybind11/stl.h
     include/pybind11/stl_bind.h
     include/pybind11/stl/filesystem.h

docs/advanced/embedding.rst

@@ -237,31 +237,259 @@ global data. All the details can be found in the CPython documentation.
 
     Creating two concurrent ``scoped_interpreter`` guards is a fatal error. So is
     calling ``initialize_interpreter`` for a second time after the interpreter
-    has already been initialized.
+    has already been initialized. Use :class:`scoped_subinterpreter` to create
+    a sub-interpreter.  See :ref:`subinterp` for important details on sub-interpreters.
 
     Do not use the raw CPython API functions ``Py_Initialize`` and
     ``Py_Finalize`` as these do not properly handle the lifetime of
     pybind11's internal data.
 
 
-Sub-interpreter support
-=======================
+.. _subinterp:
+
+Embedding Sub-interpreters
+==========================
+
+A sub-interpreter is a separate interpreter instance which provides a
+separate, isolated interpreter environment within the same process as the main
+interpreter.  Sub-interpreters are created and managed with a separate API from
+the main interpreter. Beginning in Python 3.12, sub-interpreters each have
+their own Global Interpreter Lock (GIL), which means that running a
+sub-interpreter in a separate thread from the main interpreter can achieve true
+concurrency.
+
+pybind11's sub-interpreter API can be found in ``pybind11/subinterpreter.h``.
+
+pybind11 :class:`subinterpreter` instances can be safely moved and shared between
+threads as needed. However, managing multiple threads and the lifetimes of multiple
+interpreters and their GILs can be challenging.
+Proceed with caution (and lots of testing)!
+
+The main interpreter must be initialized before creating a sub-interpreter, and
+the main interpreter must outlive all sub-interpreters. Sub-interpreters are
+managed through a different API than the main interpreter.
+
+The :class:`subinterpreter` class manages the lifetime of sub-interpreters.
+Instances are movable, but not copyable. Default constructing this class does
+*not* create a sub-interpreter (it creates an empty holder).  To create a
+sub-interpreter, call :func:`subinterpreter::create()`.
+
+.. warning::
+
+    Sub-interpreter creation acquires (and subsequently releases) the main
+    interpreter GIL. If another thread holds the main GIL, the function will
+    block until the main GIL can be acquired.
+
+    Sub-interpreter destruction temporarily activates the sub-interpreter. The
+    sub-interpreter must not be active (on any threads) at the time the
+    :class:`subinterpreter` destructor is called.
+
+    Both actions will re-acquire any interpreter's GIL that was held prior to
+    the call before returning (or return to no active interpreter if none was
+    active at the time of the call).
+
+Each sub-interpreter will import a separate copy of each ``PYBIND11_EMBEDDED_MODULE``
+when those modules specify a ``multiple_interpreters`` tag. If a module does not
+specify a ``multiple_interpreters`` tag, then Python will report an ``ImportError``
+if it is imported in a sub-interpreter.
+
+pybind11 also has a :class:`scoped_subinterpreter` class, which creates and
+activates a sub-interpreter when it is constructed, and deactivates and deletes
+it when it goes out of scope.
+
+Activating a Sub-interpreter
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Once a sub-interpreter is created, you can "activate" it on a thread (and
+acquire its GIL) by creating a :class:`subinterpreter_scoped_activate`
+instance and passing it the sub-intepreter to be activated.  The function
+will acquire the sub-interpreter's GIL and make the sub-interpreter the
+current active interpreter on the current thread for the lifetime of the
+instance. When the :class:`subinterpreter_scoped_activate` instance goes out
+of scope, the sub-interpreter GIL is released and the prior interpreter that
+was active on the thread (if any) is reactivated and it's GIL is re-acquired.
+
+When using ``subinterpreter_scoped_activate``:
+
+1. If the thread holds any interpreter's GIL:
+   - That GIL is released
+2. The new sub-interpreter's GIL is acquired
+3. The new sub-interpreter is made active.
+4. When the scope ends:
+    - The sub-interpreter's GIL is released
+    - If there was a previous interpreter:
+        - The old interpreter's GIL is re-acquired
+        - The old interpreter is made active
+    - Otherwise, no interpreter is currently active and no GIL is held.
+
+Example:
+
+.. code-block:: cpp
+
+    py::initialize_interpreter();
+    // Main GIL is held
+    {
+        py::subinterpreter sub = py::subinterpreter::create();
+        // Main interpreter is still active, main GIL re-acquired
+        {
+            py::subinterpreter_scoped_activate guard(sub);
+            // Sub-interpreter active, thread holds sub's GIL
+            {
+                py::subinterpreter_scoped_activate main_guard(py);
+                // Sub's GIL was automatically released
+                // Main interpreter active, thread holds main's GIL
+            }
+            // Back to sub-interpreter, thread holds sub's GIL again
+        }
+        // Main interpreter is active, main's GIL is held
+    }
+
+
+GIL API for sub-interpreters
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+:class:`gil_scoped_release` and :class:`gil_scoped_acquire` can be used to
+manage the GIL of a sub-interpreter just as they do for the main interpreter.
+They both manage the GIL of the currently active interpreter, without the
+programmer having to do anything special or different. There is one important
+caveat:
+
+.. note::
+
+    When no interpreter is active through a
+    :class:`subinterpreter_scoped_activate` instance (such as on a new thread),
+    :class:`gil_scoped_acquire` will acquire the **main** GIL and
+    activate the **main** interpreter.
+
+
+Full Sub-interpreter example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Here is an example showing how to create and activate sub-interpreters:
+
+.. code-block:: cpp
+
+    #include <iostream>
+    #include <pybind11/embed.h>
+    #include <pybind11/subinterpreter.h>
+
+    namespace py = pybind11;
+
+    PYBIND11_EMBEDDED_MODULE(printer, m, py::multiple_interpreters::per_interpreter_gil()) {
+        m.def("which", [](const std::string& when) {
+            std::cout << when << "; Current Interpreter is "
+                    << py::subinterpreter::current().id()
+                    << std::endl;
+        });
+    }
+
+    int main() {
+        py::scoped_interpreter main_interp;
+
+        py::module_::import("printer").attr("which")("First init");
+
+        {
+            py::subinterpreter sub = py::subinterpreter::create();
+
+            py::module_::import("printer").attr("which")("Created sub");
+
+            {
+                py::subinterpreter_scoped_activate guard(sub);
+                try {
+                    py::module_::import("printer").attr("which")("Activated sub");
+                }
+                catch (py::error_already_set &e) {
+                    std::cerr << "EXCEPTION " << e.what() << std::endl;
+                    return 1;
+                }
+            }
+
+            py::module_::import("printer").attr("which")("Deactivated sub");
+
+            {
+                py::gil_scoped_release nogil;
+                {
+                    py::subinterpreter_scoped_activate guard(sub);
+                    try {
+                        {
+                            py::subinterpreter_scoped_activate main_guard(py::subinterpreter::main());
+                            try {
+                                py::module_::import("printer").attr("which")("Main within sub");
+                            }
+                            catch (py::error_already_set &e) {
+                                std::cerr << "EXCEPTION " << e.what() << std::endl;
+                                return 1;
+                            }
+                        }
+                        py::module_::import("printer").attr("which")("After Main, still within sub");
+                    }
+                    catch (py::error_already_set &e) {
+                        std::cerr << "EXCEPTION " << e.what() << std::endl;
+                        return 1;
+                    }
+                }
+            }
+        }
+
+        py::module_::import("printer").attr("which")("At end");
+
+        return 0;
+    }
+
+Expected output:
+
+.. code-block:: text
+
+    First init; Current Interpreter is 0
+    Created sub; Current Interpreter is 0
+    Activated sub; Current Interpreter is 1
+    Deactivated sub; Current Interpreter is 0
+    Main within sub; Current Interpreter is 0
+    After Main, still within sub; Current Interpreter is 1
+    At end; Current Interpreter is 0
+
+.. warning::
+
+    In Python 3.12 sub-interpreters must be destroyed in the same OS thread
+    that created them.  Failure to follow this rule may result in deadlocks
+    or crashes when destroying the sub-interpreter on the wrong thread.
+
+    This constraint is not present in Python 3.13+.
+
+
+Best Practices for sub-interpreter safety
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- Never share Python objects across different interpreters.
+
+- :class:`error_already_set` objects contain a reference to the Python exception type,
+  and :func:`error_already_set::what()` acquires the GIL. So Python exceptions must
+  **never** be allowed to propagate past the enclosing
+  :class:`subinterpreter_scoped_activate` instance!
+  (So your try/catch should be *just inside* the scope covered by the
+  :class:`subinterpreter_scoped_activate`.)
+
+- Avoid global/static state whenever possible. Instead, keep state within each interpreter,
+  such as within the interpreter state dict, which can be accessed via
+  ``subinterpreter::current().state_dict()``, or within instance members and tied to
+  Python objects.
+
+- Avoid trying to "cache" Python objects in C++ variables across function calls (this is an easy
+  way to accidentally introduce sub-interpreter bugs). In the code example above, note that we
+  did not save the result of :func:`module_::import`, in order to avoid accidentally using the
+  resulting Python object when the wrong interpreter was active.
 
-Creating multiple copies of ``scoped_interpreter`` is not possible because it
-represents the main Python interpreter. Sub-interpreters are something different
-and they do permit the existence of multiple interpreters. This is an advanced
-feature of the CPython API and should be handled with care. pybind11 does not
-currently offer a C++ interface for sub-interpreters, so refer to the CPython
-documentation for all the details regarding this feature.
+- Avoid moving or disarming RAII objects managing GIL and sub-interpreter lifetimes. Doing so can
+  lead to confusion about lifetimes.  (For example, accidentally extending a
+  :class:`subinterpreter_scoped_activate` past the lifetime of it's :class:`subinterpreter`.)
 
-We'll just mention a couple of caveats the sub-interpreters support in pybind11:
+- While sub-interpreters each have their own GIL, there can now be multiple independent GILs in one
+  program so you need to consider the possibility of deadlocks caused by multiple GILs and/or the
+  interactions of the GIL(s) and your C++ code's own locking.
 
- 1. Sub-interpreters will not receive independent copies of embedded modules.
-    Instead, these are shared and modifications in one interpreter may be
-    reflected in another.
+- When using multiple threads to run independent sub-interpreters, the independent GILs allow
+  concurrent calls from different interpreters into the same C++ code from different threads.
+  So you must still consider the thread safety of your C++ code.  Remember, in Python 3.12
+  sub-interpreters must be destroyed on the same thread that they were created on.
 
- 2. Managing multiple threads, multiple interpreters and the GIL can be
-    challenging and there are several caveats here, even within the pure
-    CPython API (please refer to the Python docs for details). As for
-    pybind11, keep in mind that ``gil_scoped_release`` and ``gil_scoped_acquire``
-    do not take sub-interpreters into account.
+- Familiarize yourself with :ref:`misc_concurrency`.

docs/advanced/misc.rst

@@ -228,6 +228,8 @@ You can explicitly disable sub-interpreter support in your module by using the
 :func:`multiple_interpreters::not_supported()` tag. This is the default behavior if you do not
 specify a multiple_interpreters tag.
 
+.. _misc_concurrency:
+
 Concurrency and Parallelism in Python with pybind11
 ===================================================
 

include/pybind11/gil.h

@@ -130,7 +130,7 @@ public:
     }
 
     /// This method will disable the PyThreadState_DeleteCurrent call and the
-    /// GIL won't be acquired. This method should be used if the interpreter
+    /// GIL won't be released. This method should be used if the interpreter
     /// could be shutting down when this is called, as thread deletion is not
     /// allowed during shutdown. Check _Py_IsFinalizing() on Python 3.7+, and
     /// protect subsequent code.

include/pybind11/subinterpreter.h

@@ -0,0 +1,320 @@
+/*
+    pybind11/subinterpreter.h: Support for creating and using subinterpreters
+
+    Copyright (c) 2025 The Pybind Development Team.
+
+    All rights reserved. Use of this source code is governed by a
+    BSD-style license that can be found in the LICENSE file.
+*/
+
+#pragma once
+
+#include "detail/common.h"
+#include "detail/internals.h"
+#include "gil.h"
+
+#include <stdexcept>
+
+#if !defined(PYBIND11_SUBINTERPRETER_SUPPORT)
+#    error "This platform does not support subinterpreters, do not include this file."
+#endif
+
+PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE)
+PYBIND11_NAMESPACE_BEGIN(detail)
+PyInterpreterState *get_interpreter_state_unchecked() {
+    auto cur_tstate = get_thread_state_unchecked();
+    if (cur_tstate)
+        return cur_tstate->interp;
+    else
+        return nullptr;
+}
+PYBIND11_NAMESPACE_END(detail)
+
+class subinterpreter;
+
+/// Activate the subinterpreter and acquire its GIL, while also releasing any GIL and interpreter
+/// currently held. Upon exiting the scope, the previous subinterpreter (if any) and its
+/// associated GIL are restored to their state as they were before the scope was entered.
+class subinterpreter_scoped_activate {
+public:
+    explicit subinterpreter_scoped_activate(subinterpreter const &si);
+    ~subinterpreter_scoped_activate();
+
+    subinterpreter_scoped_activate(subinterpreter_scoped_activate &&) = delete;
+    subinterpreter_scoped_activate(subinterpreter_scoped_activate const &) = delete;
+    subinterpreter_scoped_activate &operator=(subinterpreter_scoped_activate &) = delete;
+    subinterpreter_scoped_activate &operator=(subinterpreter_scoped_activate const &) = delete;
+
+private:
+    PyThreadState *old_tstate_ = nullptr;
+    PyThreadState *tstate_ = nullptr;
+    PyGILState_STATE gil_state_;
+    bool simple_gil_ = false;
+};
+
+/// Holds a Python subinterpreter instance
+class subinterpreter {
+public:
+    /// empty/unusable, but move-assignable.  use create() to create a subinterpreter.
+    subinterpreter() = default;
+
+    subinterpreter(subinterpreter const &copy) = delete;
+    subinterpreter &operator=(subinterpreter const &copy) = delete;
+
+    subinterpreter(subinterpreter &&old) noexcept
+        : istate_(old.istate_), creation_tstate_(old.creation_tstate_) {
+        old.istate_ = nullptr;
+        old.creation_tstate_ = nullptr;
+    }
+
+    subinterpreter &operator=(subinterpreter &&old) noexcept {
+        std::swap(old.istate_, istate_);
+        std::swap(old.creation_tstate_, creation_tstate_);
+        return *this;
+    }
+
+    /// Create a new subinterpreter with the specified configuration
+    /// @note This function acquires (and then releases) the main interpreter GIL, but the main
+    /// interpreter and its GIL are not required to be held prior to calling this function.
+    static inline subinterpreter create(PyInterpreterConfig const &cfg) {
+        error_scope err_scope;
+        subinterpreter result;
+        {
+            // we must hold the main GIL in order to create a subinterpreter
+            subinterpreter_scoped_activate main_guard(main());
+
+            auto prev_tstate = PyThreadState_Get();
+
+            auto status = Py_NewInterpreterFromConfig(&result.creation_tstate_, &cfg);
+
+            // this doesn't raise a normal Python exception, it provides an exit() status code.
+            if (PyStatus_Exception(status)) {
+                pybind11_fail("failed to create new sub-interpreter");
+            }
+
+            // upon success, the new interpreter is activated in this thread
+            result.istate_ = result.creation_tstate_->interp;
+            detail::get_num_interpreters_seen() += 1; // there are now many interpreters
+            detail::get_internals(); // initialize internals.tstate, amongst other things...
+
+            // In 3.13+ this state should be deleted right away, and the memory will be reused for
+            // the next threadstate on this interpreter. However, on 3.12 we cannot do that, we
+            // must keep it around (but not use it) ... see destructor.
+#if PY_VERSION_HEX >= 0x030D0000
+            PyThreadState_Clear(result.creation_tstate_);
+            PyThreadState_DeleteCurrent();
+#endif
+
+            // we have to switch back to main, and then the scopes will handle cleanup
+            PyThreadState_Swap(prev_tstate);
+        }
+        return result;
+    }
+
+    /// Calls create() with a default configuration of an isolated interpreter that disallows fork,
+    /// exec, and Python threads.
+    static inline subinterpreter create() {
+        // same as the default config in the python docs
+        PyInterpreterConfig cfg;
+        std::memset(&cfg, 0, sizeof(cfg));
+        cfg.check_multi_interp_extensions = 1;
+        cfg.gil = PyInterpreterConfig_OWN_GIL;
+        return create(cfg);
+    }
+
+    ~subinterpreter() {
+        if (!creation_tstate_) {
+            // non-owning wrapper, do nothing.
+            return;
+        }
+
+        PyThreadState *destroy_tstate;
+        PyThreadState *old_tstate;
+
+        // Python 3.12 requires us to keep the original PyThreadState alive until we are ready to
+        // destroy the interpreter.  We prefer to use that to destroy the interpreter.
+#if PY_VERSION_HEX < 0x030D0000
+        // The tstate passed to Py_EndInterpreter MUST have been created on the current OS thread.
+        bool same_thread = false;
+#    ifdef PY_HAVE_THREAD_NATIVE_ID
+        same_thread = PyThread_get_thread_native_id() == creation_tstate_->native_thread_id;
+#    endif
+        if (same_thread) {
+            // OK it is safe to use the creation state here
+            destroy_tstate = creation_tstate_;
+            old_tstate = PyThreadState_Swap(destroy_tstate);
+        } else {
+            // We have to make a new tstate on this thread and use that.
+            destroy_tstate = PyThreadState_New(istate_);
+            old_tstate = PyThreadState_Swap(destroy_tstate);
+
+            // We can use the one we just created, so we must delete the creation state.
+            PyThreadState_Clear(creation_tstate_);
+            PyThreadState_Delete(creation_tstate_);
+        }
+#else
+        destroy_tstate = PyThreadState_New(istate_);
+        old_tstate = PyThreadState_Swap(destroy_tstate);
+#endif
+
+        bool switch_back = old_tstate && old_tstate->interp != istate_;
+
+        // Get the internals pointer (without creating it if it doesn't exist).  It's possible
+        // for the internals to be created during Py_EndInterpreter() (e.g. if a py::capsule
+        // calls `get_internals()` during destruction), so we get the pointer-pointer here and
+        // check it after.
+        auto *&internals_ptr_ptr = detail::get_internals_pp<detail::internals>();
+        auto *&local_internals_ptr_ptr = detail::get_internals_pp<detail::local_internals>();
+        {
+            dict sd = state_dict();
+            internals_ptr_ptr
+                = detail::get_internals_pp_from_capsule_in_state_dict<detail::internals>(
+                    sd, PYBIND11_INTERNALS_ID);
+            local_internals_ptr_ptr
+                = detail::get_internals_pp_from_capsule_in_state_dict<detail::local_internals>(
+                    sd, detail::get_local_internals_id());
+        }
+
+        // End it
+        Py_EndInterpreter(destroy_tstate);
+
+        // do NOT decrease detail::get_num_interpreters_seen, because it can never decrease
+        // while other threads are running...
+
+        if (internals_ptr_ptr) {
+            internals_ptr_ptr->reset();
+        }
+        if (local_internals_ptr_ptr) {
+            local_internals_ptr_ptr->reset();
+        }
+
+        // switch back to the old tstate and old GIL (if there was one)
+        if (switch_back)
+            PyThreadState_Swap(old_tstate);
+    }
+
+    /// Get a handle to the main interpreter that can be used with subinterpreter_scoped_activate
+    /// Note that destructing the handle is a noop, the main interpreter can only be ended by
+    /// py::finalize_interpreter()
+    static subinterpreter main() {
+        subinterpreter m;
+        m.istate_ = PyInterpreterState_Main();
+        m.disarm(); // make destruct a noop
+        return m;
+    }
+
+    /// Get a non-owning wrapper of the currently active interpreter (if any)
+    static subinterpreter current() {
+        subinterpreter c;
+        c.istate_ = detail::get_interpreter_state_unchecked();
+        c.disarm(); // make destruct a noop, we don't own this...
+        return c;
+    }
+
+    /// Get the numerical identifier for the sub-interpreter
+    int64_t id() const {
+        if (istate_ != nullptr)
+            return PyInterpreterState_GetID(istate_);
+        else
+            return -1; // CPython uses one-up numbers from 0, so negative should be safe to return
+                       // here.
+    }
+
+    /// Get the interpreter's state dict.  This interpreter's GIL must be held before calling!
+    dict state_dict() { return reinterpret_borrow<dict>(PyInterpreterState_GetDict(istate_)); }
+
+    /// abandon cleanup of this subinterpreter (leak it). this might be needed during
+    /// finalization...
+    void disarm() { creation_tstate_ = nullptr; }
+
+    /// An empty wrapper cannot be activated
+    bool empty() const { return istate_ == nullptr; }
+
+    /// Is this wrapper non-empty
+    explicit operator bool() const { return !empty(); }
+
+private:
+    friend class subinterpreter_scoped_activate;
+    PyInterpreterState *istate_ = nullptr;
+    PyThreadState *creation_tstate_ = nullptr;
+};
+
+class scoped_subinterpreter {
+public:
+    scoped_subinterpreter() : si_(subinterpreter::create()), scope_(si_) {}
+
+    explicit scoped_subinterpreter(PyInterpreterConfig const &cfg)
+        : si_(subinterpreter::create(cfg)), scope_(si_) {}
+
+private:
+    subinterpreter si_;
+    subinterpreter_scoped_activate scope_;
+};
+
+inline subinterpreter_scoped_activate::subinterpreter_scoped_activate(subinterpreter const &si) {
+    if (!si.istate_) {
+        pybind11_fail("null subinterpreter");
+    }
+
+    if (detail::get_interpreter_state_unchecked() == si.istate_) {
+        // we are already on this interpreter, make sure we hold the GIL
+        simple_gil_ = true;
+        gil_state_ = PyGILState_Ensure();
+        return;
+    }
+
+    // we can't really interact with the interpreter at all until we switch to it
+    // not even to, for example, look in its state dict or touch its internals
+    tstate_ = PyThreadState_New(si.istate_);
+
+    // make the interpreter active and acquire the GIL
+    old_tstate_ = PyThreadState_Swap(tstate_);
+
+    // save this in internals for scoped_gil calls
+    PYBIND11_TLS_REPLACE_VALUE(detail::get_internals().tstate, tstate_);
+}
+
+inline subinterpreter_scoped_activate::~subinterpreter_scoped_activate() {
+    if (simple_gil_) {
+        // We were on this interpreter already, so just make sure the GIL goes back as it was
+        PyGILState_Release(gil_state_);
+    } else {
+#if defined(PYBIND11_DETAILED_ERROR_MESSAGES)
+        bool has_active_exception;
+#    if defined(__cpp_lib_uncaught_exceptions)
+        has_active_exception = std::uncaught_exceptions() > 0;
+#    else
+        // removed in C++20, replaced with uncaught_exceptions
+        has_active_exception = std::uncaught_exception();
+#    endif
+        if (has_active_exception) {
+            try {
+                std::rethrow_exception(std::current_exception());
+            } catch (error_already_set &) {
+                // Because error_already_set holds python objects and what() acquires the GIL, it
+                // is basically never OK to let these exceptions propagate outside the current
+                // active interpreter.
+                pybind11_fail("~subinterpreter_scoped_activate: cannot propagate Python "
+                              "exceptions outside of their owning interpreter");
+            } catch (...) {
+            }
+        }
+#endif
+
+        if (tstate_) {
+#if defined(PYBIND11_DETAILED_ERROR_MESSAGES)
+            if (detail::get_thread_state_unchecked() != tstate_) {
+                pybind11_fail("~subinterpreter_scoped_activate: thread state must be current!");
+            }
+#endif
+            PYBIND11_TLS_DELETE_VALUE(detail::get_internals().tstate);
+            PyThreadState_Clear(tstate_);
+            PyThreadState_DeleteCurrent();
+        }
+
+        // Go back the previous interpreter (if any) and acquire THAT gil
+        PyThreadState_Swap(old_tstate_);
+    }
+}
+
+PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE)

tests/extra_python_package/test_files.py

@@ -58,6 +58,7 @@ main_headers = {
     "include/pybind11/options.h",
     "include/pybind11/pybind11.h",
     "include/pybind11/pytypes.h",
+    "include/pybind11/subinterpreter.h",
     "include/pybind11/stl.h",
     "include/pybind11/stl_bind.h",
     "include/pybind11/trampoline_self_life_support.h",

tests/test_embed/CMakeLists.txt

@@ -28,7 +28,7 @@ endif()
 
 find_package(Threads REQUIRED)
 
-add_executable(test_embed catch.cpp test_interpreter.cpp)
+add_executable(test_embed catch.cpp test_interpreter.cpp test_subinterpreter.cpp)
 pybind11_enable_warnings(test_embed)
 
 target_link_libraries(test_embed PRIVATE pybind11::embed Catch2::Catch2 Threads::Threads)

tests/test_embed/test_interpreter.cpp

@@ -19,6 +19,21 @@ size_t get_sys_path_size() {
     return py::len(sys_path);
 }
 
+bool has_state_dict_internals_obj() {
+    py::dict state = py::detail::get_python_state_dict();
+    return state.contains(PYBIND11_INTERNALS_ID);
+}
+
+bool has_pybind11_internals_static() {
+    auto *&ipp = py::detail::get_internals_pp<py::detail::internals>();
+    return (ipp != nullptr) && *ipp;
+}
+
+uintptr_t get_details_as_uintptr() {
+    return reinterpret_cast<uintptr_t>(
+        py::detail::get_internals_pp<py::detail::internals>()->get());
+}
+
 class Widget {
 public:
     explicit Widget(std::string message) : message(std::move(message)) {}
@@ -258,21 +273,6 @@ TEST_CASE("Add program dir to path using PyConfig") {
 }
 #endif
 
-bool has_state_dict_internals_obj() {
-    py::dict state = py::detail::get_python_state_dict();
-    return state.contains(PYBIND11_INTERNALS_ID);
-}
-
-bool has_pybind11_internals_static() {
-    auto *&ipp = py::detail::get_internals_pp<py::detail::internals>();
-    return (ipp != nullptr) && *ipp;
-}
-
-uintptr_t get_details_as_uintptr() {
-    return reinterpret_cast<uintptr_t>(
-        py::detail::get_internals_pp<py::detail::internals>()->get());
-}
-
 TEST_CASE("Restart the interpreter") {
     // Verify pre-restart state.
     REQUIRE(py::module_::import("widget_module").attr("add")(1, 2).cast<int>() == 3);
@@ -336,279 +336,6 @@ TEST_CASE("Restart the interpreter") {
     REQUIRE(py_widget.attr("the_message").cast<std::string>() == "Hello after restart");
 }
 
-#if defined(PYBIND11_SUBINTERPRETER_SUPPORT)
-TEST_CASE("Subinterpreter") {
-    py::module_::import("external_module"); // in the main interpreter
-
-    // Add tags to the modules in the main interpreter and test the basics.
-    py::module_::import("__main__").attr("main_tag") = "main interpreter";
-    {
-        auto m = py::module_::import("widget_module");
-        m.attr("extension_module_tag") = "added to module in main interpreter";
-
-        REQUIRE(m.attr("add")(1, 2).cast<int>() == 3);
-    }
-
-    auto main_int
-        = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
-
-    REQUIRE(has_state_dict_internals_obj());
-    REQUIRE(has_pybind11_internals_static());
-
-    /// Create and switch to a subinterpreter.
-    auto *main_tstate = PyThreadState_Get();
-    auto *sub_tstate = Py_NewInterpreter();
-
-    py::detail::get_num_interpreters_seen()++;
-
-    // Subinterpreters get their own copy of builtins.
-    REQUIRE_FALSE(has_state_dict_internals_obj());
-
-    // internals hasn't been populated yet, but will be different for the subinterpreter
-    REQUIRE_FALSE(has_pybind11_internals_static());
-
-    py::list(py::module_::import("sys").attr("path")).append(py::str("."));
-
-    auto ext_int = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
-    py::detail::get_internals();
-    REQUIRE(has_pybind11_internals_static());
-    REQUIRE(get_details_as_uintptr() == ext_int);
-    REQUIRE(main_int != ext_int);
-
-    // Modules tags should be gone.
-    REQUIRE_FALSE(py::hasattr(py::module_::import("__main__"), "tag"));
-    {
-        REQUIRE_NOTHROW(py::module_::import("widget_module"));
-        auto m = py::module_::import("widget_module");
-        REQUIRE_FALSE(py::hasattr(m, "extension_module_tag"));
-
-        // Function bindings should still work.
-        REQUIRE(m.attr("add")(1, 2).cast<int>() == 3);
-    }
-
-    // The subinterpreter now has internals populated since we imported a pybind11 module
-    REQUIRE(has_pybind11_internals_static());
-
-    // Restore main interpreter.
-    Py_EndInterpreter(sub_tstate);
-    py::detail::get_num_interpreters_seen() = 1;
-    PyThreadState_Swap(main_tstate);
-
-    REQUIRE(py::hasattr(py::module_::import("__main__"), "main_tag"));
-    REQUIRE(py::hasattr(py::module_::import("widget_module"), "extension_module_tag"));
-    REQUIRE(has_state_dict_internals_obj());
-}
-
-TEST_CASE("Multiple Subinterpreters") {
-    // Make sure the module is in the main interpreter and save its pointer
-    auto *main_ext = py::module_::import("external_module").ptr();
-    auto main_int
-        = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
-    py::module_::import("external_module").attr("multi_interp") = "1";
-
-    auto *main_tstate = PyThreadState_Get();
-
-    /// Create and switch to a subinterpreter.
-    auto *sub1_tstate = Py_NewInterpreter();
-    py::detail::get_num_interpreters_seen()++;
-
-    py::list(py::module_::import("sys").attr("path")).append(py::str("."));
-
-    // The subinterpreter has its own copy of this module which is completely separate from main
-    auto *sub1_ext = py::module_::import("external_module").ptr();
-    REQUIRE(sub1_ext != main_ext);
-    REQUIRE_FALSE(py::hasattr(py::module_::import("external_module"), "multi_interp"));
-    py::module_::import("external_module").attr("multi_interp") = "2";
-    // The subinterpreter also has its own internals
-    auto sub1_int
-        = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
-    REQUIRE(sub1_int != main_int);
-
-    // Create another interpreter
-    auto *sub2_tstate = Py_NewInterpreter();
-    py::detail::get_num_interpreters_seen()++;
-
-    py::list(py::module_::import("sys").attr("path")).append(py::str("."));
-
-    // The second subinterpreter is separate from both main and the other subinterpreter
-    auto *sub2_ext = py::module_::import("external_module").ptr();
-    REQUIRE(sub2_ext != main_ext);
-    REQUIRE(sub2_ext != sub1_ext);
-    REQUIRE_FALSE(py::hasattr(py::module_::import("external_module"), "multi_interp"));
-    py::module_::import("external_module").attr("multi_interp") = "3";
-    // The subinterpreter also has its own internals
-    auto sub2_int
-        = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
-    REQUIRE(sub2_int != main_int);
-    REQUIRE(sub2_int != sub1_int);
-
-    PyThreadState_Swap(sub1_tstate); // go back to sub1
-
-    REQUIRE(py::cast<std::string>(py::module_::import("external_module").attr("multi_interp"))
-            == "2");
-
-    PyThreadState_Swap(main_tstate); // go back to main
-
-    auto post_int
-        = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
-    // Make sure internals went back the way it was before
-    REQUIRE(main_int == post_int);
-
-    REQUIRE(py::cast<std::string>(py::module_::import("external_module").attr("multi_interp"))
-            == "1");
-
-    PyThreadState_Swap(sub1_tstate);
-    Py_EndInterpreter(sub1_tstate);
-    PyThreadState_Swap(sub2_tstate);
-    Py_EndInterpreter(sub2_tstate);
-
-    py::detail::get_num_interpreters_seen() = 1;
-    PyThreadState_Swap(main_tstate);
-}
-#endif
-
-#if defined(Py_MOD_PER_INTERPRETER_GIL_SUPPORTED) && defined(PYBIND11_SUBINTERPRETER_SUPPORT)
-TEST_CASE("Per-Subinterpreter GIL") {
-    auto main_int
-        = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
-
-    std::atomic<int> started, sync, failure;
-    started = 0;
-    sync = 0;
-    failure = 0;
-
-// REQUIRE throws on failure, so we can't use it within the thread
-#    define T_REQUIRE(status)                                                                     \
-        do {                                                                                      \
-            assert(status);                                                                       \
-            if (!(status))                                                                        \
-                ++failure;                                                                        \
-        } while (0)
-
-    auto &&thread_main = [&](int num) {
-        while (started == 0)
-            std::this_thread::sleep_for(std::chrono::microseconds(1));
-        ++started;
-
-        py::gil_scoped_acquire gil;
-        auto main_tstate = PyThreadState_Get();
-
-        // we have the GIL, we can access the main interpreter
-        auto t_int
-            = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
-        T_REQUIRE(t_int == main_int);
-        py::module_::import("external_module").attr("multi_interp") = "1";
-
-        PyThreadState *sub = nullptr;
-        PyInterpreterConfig cfg;
-        memset(&cfg, 0, sizeof(cfg));
-        cfg.check_multi_interp_extensions = 1;
-        cfg.gil = PyInterpreterConfig_OWN_GIL;
-        auto status = Py_NewInterpreterFromConfig(&sub, &cfg);
-        T_REQUIRE(!PyStatus_IsError(status));
-
-        py::detail::get_num_interpreters_seen()++;
-
-        py::list(py::module_::import("sys").attr("path")).append(py::str("."));
-
-        // we have switched to the new interpreter and released the main gil
-
-        // trampoline_module did not provide the per_interpreter_gil tag, so it cannot be
-        // imported
-        bool caught = false;
-        try {
-            py::module_::import("trampoline_module");
-        } catch (pybind11::error_already_set &pe) {
-            T_REQUIRE(pe.matches(PyExc_ImportError));
-            std::string msg(pe.what());
-            T_REQUIRE(msg.find("does not support loading in subinterpreters")
-                      != std::string::npos);
-            caught = true;
-        }
-        T_REQUIRE(caught);
-
-        // widget_module did provide the per_interpreter_gil tag, so it this does not throw
-        py::module_::import("widget_module");
-
-        T_REQUIRE(!py::hasattr(py::module_::import("external_module"), "multi_interp"));
-        py::module_::import("external_module").attr("multi_interp") = std::to_string(num);
-
-        // wait for something to set sync to our thread number
-        // we are holding our subinterpreter's GIL
-        while (sync != num)
-            std::this_thread::sleep_for(std::chrono::microseconds(1));
-
-        // now change it so the next thread can mvoe on
-        ++sync;
-
-        // but keep holding the GIL until after the next thread moves on as well
-        while (sync == num + 1)
-            std::this_thread::sleep_for(std::chrono::microseconds(1));
-
-        // one last check before quitting the thread, the internals should be different
-        auto sub_int
-            = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
-        T_REQUIRE(sub_int != main_int);
-
-        Py_EndInterpreter(sub);
-
-        // switch back so the scoped_acquire can release the GIL properly
-        PyThreadState_Swap(main_tstate);
-    };
-
-    std::thread t1(thread_main, 1);
-    std::thread t2(thread_main, 2);
-
-    // we spawned two threads, at this point they are both waiting for started to increase
-    ++started;
-
-    // ok now wait for the threads to start
-    while (started != 3)
-        std::this_thread::sleep_for(std::chrono::microseconds(1));
-
-    // we still hold the main GIL, at this point both threads are waiting on the main GIL
-    // IN THE CASE of free threading, the threads are waiting on sync (because there is no GIL)
-
-    // IF the below code hangs in one of the wait loops, then the child thread GIL behavior did not
-    // function as expected.
-    {
-        // release the GIL and allow the threads to run
-        py::gil_scoped_release nogil;
-
-        // the threads are now waiting on the sync
-        REQUIRE(sync == 0);
-
-        // this will trigger thread 1 and then advance and trigger 2 and then advance
-        sync = 1;
-
-        // wait for thread 2 to advance
-        while (sync != 3)
-            std::this_thread::sleep_for(std::chrono::microseconds(1));
-
-        // we know now that thread 1 has run and may be finishing
-        // and thread 2 is waiting for permission to advance
-
-        // so we move sync so that thread 2 can finish executing
-        ++sync;
-
-        // now wait for both threads to complete
-        t1.join();
-        t2.join();
-    }
-
-    // now we have the gil again, sanity check
-    REQUIRE(py::cast<std::string>(py::module_::import("external_module").attr("multi_interp"))
-            == "1");
-
-    // the threads are stopped. we can now lower this for the rest of the test
-    py::detail::get_num_interpreters_seen() = 1;
-
-    // make sure nothing unexpected happened inside the threads, now that they are completed
-    REQUIRE(failure == 0);
-#    undef T_REQUIRE
-}
-#endif
-
 TEST_CASE("Execution frame") {
     // When the interpreter is embedded, there is no execution frame, but `py::exec`
     // should still function by using reasonable globals: `__main__.__dict__`.

tests/test_embed/test_subinterpreter.cpp

@@ -0,0 +1,431 @@
+#include <pybind11/embed.h>
+#ifdef PYBIND11_SUBINTERPRETER_SUPPORT
+#    include <pybind11/subinterpreter.h>
+
+// Silence MSVC C++17 deprecation warning from Catch regarding std::uncaught_exceptions (up to
+// catch 2.0.1; this should be fixed in the next catch release after 2.0.1).
+PYBIND11_WARNING_DISABLE_MSVC(4996)
+
+#    include <catch.hpp>
+#    include <cstdlib>
+#    include <fstream>
+#    include <functional>
+#    include <thread>
+#    include <utility>
+
+namespace py = pybind11;
+using namespace py::literals;
+
+bool has_state_dict_internals_obj();
+bool has_pybind11_internals_static();
+uintptr_t get_details_as_uintptr();
+
+void unsafe_reset_internals_for_single_interpreter() {
+    // unsafe normally, but for subsequent tests, put this back.. we know there are no threads
+    // running and only 1 interpreter
+    py::detail::get_num_interpreters_seen() = 1;
+    py::detail::get_internals_pp<py::detail::internals>() = nullptr;
+    py::detail::get_internals();
+    py::detail::get_internals_pp<py::detail::local_internals>() = nullptr;
+    py::detail::get_local_internals();
+}
+
+TEST_CASE("Single Subinterpreter") {
+    py::module_::import("external_module"); // in the main interpreter
+
+    // Add tags to the modules in the main interpreter and test the basics.
+    py::module_::import("__main__").attr("main_tag") = "main interpreter";
+    {
+        auto m = py::module_::import("widget_module");
+        m.attr("extension_module_tag") = "added to module in main interpreter";
+
+        REQUIRE(m.attr("add")(1, 2).cast<int>() == 3);
+    }
+    REQUIRE(has_state_dict_internals_obj());
+    REQUIRE(has_pybind11_internals_static());
+
+    auto main_int
+        = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
+
+    /// Create and switch to a subinterpreter.
+    {
+        py::scoped_subinterpreter ssi;
+
+        // The subinterpreter has internals populated
+        REQUIRE(has_pybind11_internals_static());
+
+        py::list(py::module_::import("sys").attr("path")).append(py::str("."));
+
+        auto ext_int
+            = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
+        py::detail::get_internals();
+        REQUIRE(has_pybind11_internals_static());
+        REQUIRE(get_details_as_uintptr() == ext_int);
+        REQUIRE(ext_int != main_int);
+
+        // Modules tags should be gone.
+        REQUIRE_FALSE(py::hasattr(py::module_::import("__main__"), "tag"));
+        {
+            auto m = py::module_::import("widget_module");
+            REQUIRE_FALSE(py::hasattr(m, "extension_module_tag"));
+
+            // Function bindings should still work.
+            REQUIRE(m.attr("add")(1, 2).cast<int>() == 3);
+        }
+    }
+
+    REQUIRE(py::hasattr(py::module_::import("__main__"), "main_tag"));
+    REQUIRE(py::hasattr(py::module_::import("widget_module"), "extension_module_tag"));
+    REQUIRE(has_state_dict_internals_obj());
+
+    unsafe_reset_internals_for_single_interpreter();
+}
+
+#    if PY_VERSION_HEX >= 0x030D0000
+TEST_CASE("Move Subinterpreter") {
+    std::unique_ptr<py::subinterpreter> sub(new py::subinterpreter(py::subinterpreter::create()));
+
+    // on this thread, use the subinterpreter and import some non-trivial junk
+    {
+        py::subinterpreter_scoped_activate activate(*sub);
+
+        py::list(py::module_::import("sys").attr("path")).append(py::str("."));
+        py::module_::import("datetime");
+        py::module_::import("threading");
+        py::module_::import("external_module");
+    }
+
+    std::thread([&]() {
+        // Use it again
+        {
+            py::subinterpreter_scoped_activate activate(*sub);
+            py::module_::import("external_module");
+        }
+        sub.reset();
+    }).join();
+
+    REQUIRE(!sub);
+
+    unsafe_reset_internals_for_single_interpreter();
+}
+#    endif
+
+TEST_CASE("GIL Subinterpreter") {
+
+    PyInterpreterState *main_interp = PyInterpreterState_Get();
+
+    {
+        auto sub = py::subinterpreter::create();
+
+        REQUIRE(main_interp == PyInterpreterState_Get());
+
+        PyInterpreterState *sub_interp = nullptr;
+
+        {
+            py::subinterpreter_scoped_activate activate(sub);
+
+            sub_interp = PyInterpreterState_Get();
+            REQUIRE(sub_interp != main_interp);
+
+            py::list(py::module_::import("sys").attr("path")).append(py::str("."));
+            py::module_::import("datetime");
+            py::module_::import("threading");
+            py::module_::import("external_module");
+
+            {
+                py::subinterpreter_scoped_activate main(py::subinterpreter::main());
+                REQUIRE(PyInterpreterState_Get() == main_interp);
+
+                {
+                    py::gil_scoped_release nogil{};
+                    {
+                        py::gil_scoped_acquire yesgil{};
+                        REQUIRE(PyInterpreterState_Get() == main_interp);
+                    }
+                }
+
+                REQUIRE(PyInterpreterState_Get() == main_interp);
+            }
+
+            REQUIRE(PyInterpreterState_Get() == sub_interp);
+
+            {
+                py::gil_scoped_release nogil{};
+                {
+                    py::gil_scoped_acquire yesgil{};
+                    REQUIRE(PyInterpreterState_Get() == sub_interp);
+                }
+            }
+
+            REQUIRE(PyInterpreterState_Get() == sub_interp);
+        }
+
+        REQUIRE(PyInterpreterState_Get() == main_interp);
+
+        {
+            py::gil_scoped_release nogil{};
+            {
+                py::gil_scoped_acquire yesgil{};
+                REQUIRE(PyInterpreterState_Get() == main_interp);
+            }
+        }
+
+        REQUIRE(PyInterpreterState_Get() == main_interp);
+
+        bool thread_result;
+
+        {
+            thread_result = false;
+            py::gil_scoped_release nogil{};
+            std::thread([&]() {
+                {
+                    py::subinterpreter_scoped_activate ssa{sub};
+                }
+                {
+                    py::gil_scoped_acquire gil{};
+                    thread_result = (PyInterpreterState_Get() == main_interp);
+                }
+            }).join();
+        }
+        REQUIRE(thread_result);
+
+        {
+            thread_result = false;
+            py::gil_scoped_release nogil{};
+            std::thread([&]() {
+                py::gil_scoped_acquire gil{};
+                thread_result = (PyInterpreterState_Get() == main_interp);
+            }).join();
+        }
+        REQUIRE(thread_result);
+    }
+
+    REQUIRE(PyInterpreterState_Get() == main_interp);
+    unsafe_reset_internals_for_single_interpreter();
+}
+
+TEST_CASE("Multiple Subinterpreters") {
+    // Make sure the module is in the main interpreter and save its pointer
+    auto *main_ext = py::module_::import("external_module").ptr();
+    auto main_int
+        = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
+    py::module_::import("external_module").attr("multi_interp") = "1";
+
+    {
+        py::subinterpreter si1 = py::subinterpreter::create();
+        std::unique_ptr<py::subinterpreter> psi2;
+
+        PyObject *sub1_ext = nullptr;
+        PyObject *sub2_ext = nullptr;
+        uintptr_t sub1_int = 0;
+        uintptr_t sub2_int = 0;
+
+        {
+            py::subinterpreter_scoped_activate scoped(si1);
+            py::list(py::module_::import("sys").attr("path")).append(py::str("."));
+
+            // The subinterpreter has its own copy of this module which is completely separate from
+            // main
+            sub1_ext = py::module_::import("external_module").ptr();
+            REQUIRE(sub1_ext != main_ext);
+            REQUIRE_FALSE(py::hasattr(py::module_::import("external_module"), "multi_interp"));
+            py::module_::import("external_module").attr("multi_interp") = "2";
+            // The subinterpreter also has its own internals
+            sub1_int
+                = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
+            REQUIRE(sub1_int != main_int);
+
+            // while the old one is active, create a new one
+            psi2.reset(new py::subinterpreter(py::subinterpreter::create()));
+        }
+
+        {
+            py::subinterpreter_scoped_activate scoped(*psi2);
+            py::list(py::module_::import("sys").attr("path")).append(py::str("."));
+
+            // The second subinterpreter is separate from both main and the other subinterpreter
+            sub2_ext = py::module_::import("external_module").ptr();
+            REQUIRE(sub2_ext != main_ext);
+            REQUIRE(sub2_ext != sub1_ext);
+            REQUIRE_FALSE(py::hasattr(py::module_::import("external_module"), "multi_interp"));
+            py::module_::import("external_module").attr("multi_interp") = "3";
+            // The subinterpreter also has its own internals
+            sub2_int
+                = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
+            REQUIRE(sub2_int != main_int);
+            REQUIRE(sub2_int != sub1_int);
+        }
+
+        {
+            py::subinterpreter_scoped_activate scoped(si1);
+            REQUIRE(
+                py::cast<std::string>(py::module_::import("external_module").attr("multi_interp"))
+                == "2");
+        }
+
+        // out here we should be in the main interpreter, with the GIL, with the other 2 still
+        // alive
+
+        auto post_int
+            = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
+        // Make sure internals went back the way it was before
+        REQUIRE(main_int == post_int);
+
+        REQUIRE(py::cast<std::string>(py::module_::import("external_module").attr("multi_interp"))
+                == "1");
+    }
+
+    // now back to just main
+
+    auto post_int
+        = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
+    // Make sure internals went back the way it was before
+    REQUIRE(main_int == post_int);
+
+    REQUIRE(py::cast<std::string>(py::module_::import("external_module").attr("multi_interp"))
+            == "1");
+
+    unsafe_reset_internals_for_single_interpreter();
+}
+
+#    ifdef Py_MOD_PER_INTERPRETER_GIL_SUPPORTED
+TEST_CASE("Per-Subinterpreter GIL") {
+    auto main_int
+        = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
+
+    std::atomic<int> started, sync, failure;
+    started = 0;
+    sync = 0;
+    failure = 0;
+
+// REQUIRE throws on failure, so we can't use it within the thread
+#        define T_REQUIRE(status)                                                                 \
+            do {                                                                                  \
+                assert(status);                                                                   \
+                if (!(status))                                                                    \
+                    ++failure;                                                                    \
+            } while (0)
+
+    auto &&thread_main = [&](int num) {
+        while (started == 0)
+            std::this_thread::sleep_for(std::chrono::microseconds(1));
+        ++started;
+
+        py::gil_scoped_acquire gil;
+
+        // we have the GIL, we can access the main interpreter
+        auto t_int
+            = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
+        T_REQUIRE(t_int == main_int);
+        py::module_::import("external_module").attr("multi_interp") = "1";
+
+        auto sub = py::subinterpreter::create();
+
+        {
+            py::subinterpreter_scoped_activate sguard{sub};
+
+            py::list(py::module_::import("sys").attr("path")).append(py::str("."));
+
+            // we have switched to the new interpreter and released the main gil
+
+            // trampoline_module did not provide the per_interpreter_gil tag, so it cannot be
+            // imported
+            bool caught = false;
+            try {
+                py::module_::import("trampoline_module");
+            } catch (pybind11::error_already_set &pe) {
+                T_REQUIRE(pe.matches(PyExc_ImportError));
+                std::string msg(pe.what());
+                T_REQUIRE(msg.find("does not support loading in subinterpreters")
+                          != std::string::npos);
+                caught = true;
+            }
+            T_REQUIRE(caught);
+
+            // widget_module did provide the per_interpreter_gil tag, so it this does not throw
+            try {
+                py::module_::import("widget_module");
+                caught = false;
+            } catch (pybind11::error_already_set &) {
+                caught = true;
+            }
+            T_REQUIRE(!caught);
+
+            // widget_module did provide the per_interpreter_gil tag, so it this does not throw
+            py::module_::import("widget_module");
+
+            T_REQUIRE(!py::hasattr(py::module_::import("external_module"), "multi_interp"));
+            py::module_::import("external_module").attr("multi_interp") = std::to_string(num);
+
+            // wait for something to set sync to our thread number
+            // we are holding our subinterpreter's GIL
+            while (sync != num)
+                std::this_thread::sleep_for(std::chrono::microseconds(1));
+
+            // now change it so the next thread can move on
+            ++sync;
+
+            // but keep holding the GIL until after the next thread moves on as well
+            while (sync == num + 1)
+                std::this_thread::sleep_for(std::chrono::microseconds(1));
+
+            // one last check before quitting the thread, the internals should be different
+            auto sub_int
+                = py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>();
+            T_REQUIRE(sub_int != main_int);
+        }
+    };
+#        undef T_REQUIRE
+
+    std::thread t1(thread_main, 1);
+    std::thread t2(thread_main, 2);
+
+    // we spawned two threads, at this point they are both waiting for started to increase
+    ++started;
+
+    // ok now wait for the threads to start
+    while (started != 3)
+        std::this_thread::sleep_for(std::chrono::microseconds(1));
+
+    // we still hold the main GIL, at this point both threads are waiting on the main GIL
+    // IN THE CASE of free threading, the threads are waiting on sync (because there is no GIL)
+
+    // IF the below code hangs in one of the wait loops, then the child thread GIL behavior did not
+    // function as expected.
+    {
+        // release the GIL and allow the threads to run
+        py::gil_scoped_release nogil;
+
+        // the threads are now waiting on the sync
+        REQUIRE(sync == 0);
+
+        // this will trigger thread 1 and then advance and trigger 2 and then advance
+        sync = 1;
+
+        // wait for thread 2 to advance
+        while (sync != 3)
+            std::this_thread::sleep_for(std::chrono::microseconds(1));
+
+        // we know now that thread 1 has run and may be finishing
+        // and thread 2 is waiting for permission to advance
+
+        // so we move sync so that thread 2 can finish executing
+        ++sync;
+
+        // now wait for both threads to complete
+        t1.join();
+        t2.join();
+    }
+
+    // now we have the gil again, sanity check
+    REQUIRE(py::cast<std::string>(py::module_::import("external_module").attr("multi_interp"))
+            == "1");
+
+    unsafe_reset_internals_for_single_interpreter();
+
+    // make sure nothing unexpected happened inside the threads, now that they are completed
+    REQUIRE(failure == 0);
+}
+#    endif // Py_MOD_PER_INTERPRETER_GIL_SUPPORTED
+
+#endif // PYBIND11_SUBINTERPRETER_SUPPORT