[NOGIL] Guard AdminClient against concurrent close() vs method-call races - #2317
Conversation
|
🎉 All Contributor License Agreements have been signed. Ready to merge. |
|
b0425e3 to
7d63d3b
Compare
015da26 to
4c750a2
Compare
06cc23f to
91b4550
Compare
491e309 to
3f16d63
Compare
2699b0e to
aabf667
Compare
Kaushik Raina (k-raina)
left a comment
There was a problem hiding this comment.
Couple of comments on tests
| def run_poll(): | ||
| nonlocal poll_finished_at | ||
| poll_started.set() | ||
| admin.poll(poll_duration) |
There was a problem hiding this comment.
If broker is unreachable remaining_poll_duration is poll_finished_at − exit_start, and since poll() already finished ~2s before exit_start, it's negative. So the assertion is (≥0) >= (negative) is always true.
There was a problem hiding this comment.
Nice catch. I have replaced poll() with list_topics() which actually blocks for the given timeout. Also added assertion that ensures list call finished after exit started.
|
|
||
|
|
||
| @subprocess_isolated | ||
| def test_exit_races_list_topics(): |
There was a problem hiding this comment.
Do we need similiar test for set_sasl_credentials ?
There was a problem hiding this comment.
Added a test case
| except Exception: # noqa: BLE001 - broker-less errors expected, not the bug we're after | ||
| pass | ||
| except RuntimeError: | ||
| break |
There was a problem hiding this comment.
I am not sure if we should treat runtime error as sucess? Please check for rest of PR
There was a problem hiding this comment.
Runtime errors are expected when the call starts but close has already been initiated on another thread. To harden it, I can add an assertion inside the except to check if it is the "closing" error and nothing else.
| while not stop_event.is_set(): | ||
| try: | ||
| admin.list_topics(timeout=0.05) | ||
| except RuntimeError: |
There was a problem hiding this comment.
Replied above
bb8a620 to
a2f9446
Compare
2dca38c to
74860f0
Compare
a2f9446 to
42b40cc
Compare
74860f0 to
143a219
Compare
143a219 to
01413bc
Compare
Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer.
d229931 to
2dd1397
Compare
|
d858468
into
dev_thread_free_support_preview
…aces (#2317) * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Trigger CLA check * Add Integration test coverage for AIO Producer * Add Tx related test cases + Add abort_tx call in Producer close * Fix styling * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Trigger CLA check * Trigger CLA check * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Style fixes * Addressed comments * Trigger CLA check * Address comments * Add Tx related test cases + Add abort_tx call in Producer close * Fix for serializing concurrent calls from a Async callback * Fix one test case * Fix styling * [NOGIL] Guard AdminClient against concurrent close()-vs-method-call races * Add handle check in common APIs and refactor overall handle logic * Remove conflict marker * Remove another conflict marker * Rename handle functions * Add more tests * Fix flaky test * Address comments * Fix corrupted _common.py * Fix other corrupted files * Fix flaky test case
…aces (#2317) * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Trigger CLA check * Add Integration test coverage for AIO Producer * Add Tx related test cases + Add abort_tx call in Producer close * Fix styling * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Trigger CLA check * Trigger CLA check * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Style fixes * Addressed comments * Trigger CLA check * Address comments * Add Tx related test cases + Add abort_tx call in Producer close * Fix for serializing concurrent calls from a Async callback * Fix one test case * Fix styling * [NOGIL] Guard AdminClient against concurrent close()-vs-method-call races * Add handle check in common APIs and refactor overall handle logic * Remove conflict marker * Remove another conflict marker * Rename handle functions * Add more tests * Fix flaky test * Address comments * Fix corrupted _common.py * Fix other corrupted files * Fix flaky test case
…aces (#2317) * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Trigger CLA check * Add Integration test coverage for AIO Producer * Add Tx related test cases + Add abort_tx call in Producer close * Fix styling * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Trigger CLA check * Trigger CLA check * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Style fixes * Addressed comments * Trigger CLA check * Address comments * Add Tx related test cases + Add abort_tx call in Producer close * Fix for serializing concurrent calls from a Async callback * Fix one test case * Fix styling * [NOGIL] Guard AdminClient against concurrent close()-vs-method-call races * Add handle check in common APIs and refactor overall handle logic * Remove conflict marker * Remove another conflict marker * Rename handle functions * Add more tests * Fix flaky test * Address comments * Fix corrupted _common.py * Fix other corrupted files * Fix flaky test case
…aces (#2317) * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Trigger CLA check * Add Integration test coverage for AIO Producer * Add Tx related test cases + Add abort_tx call in Producer close * Fix styling * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Trigger CLA check * Trigger CLA check * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Style fixes * Addressed comments * Trigger CLA check * Address comments * Add Tx related test cases + Add abort_tx call in Producer close * Fix for serializing concurrent calls from a Async callback * Fix one test case * Fix styling * [NOGIL] Guard AdminClient against concurrent close()-vs-method-call races * Add handle check in common APIs and refactor overall handle logic * Remove conflict marker * Remove another conflict marker * Rename handle functions * Add more tests * Fix flaky test * Address comments * Fix corrupted _common.py * Fix other corrupted files * Fix flaky test case
…aces (#2317) * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Trigger CLA check * Add Integration test coverage for AIO Producer * Add Tx related test cases + Add abort_tx call in Producer close * Fix styling * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Trigger CLA check * Trigger CLA check * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Style fixes * Addressed comments * Trigger CLA check * Address comments * Add Tx related test cases + Add abort_tx call in Producer close * Fix for serializing concurrent calls from a Async callback * Fix one test case * Fix styling * [NOGIL] Guard AdminClient against concurrent close()-vs-method-call races * Add handle check in common APIs and refactor overall handle logic * Remove conflict marker * Remove another conflict marker * Rename handle functions * Add more tests * Fix flaky test * Address comments * Fix corrupted _common.py * Fix other corrupted files * Fix flaky test case
…aces (#2317) (#2346) * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Trigger CLA check * Add Integration test coverage for AIO Producer * Add Tx related test cases + Add abort_tx call in Producer close * Fix styling * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Trigger CLA check * Trigger CLA check * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Style fixes * Addressed comments * Trigger CLA check * Address comments * Add Tx related test cases + Add abort_tx call in Producer close * Fix for serializing concurrent calls from a Async callback * Fix one test case * Fix styling * [NOGIL] Guard AdminClient against concurrent close()-vs-method-call races * Add handle check in common APIs and refactor overall handle logic * Remove conflict marker * Remove another conflict marker * Rename handle functions * Add more tests * Fix flaky test * Address comments * Fix corrupted _common.py * Fix other corrupted files * Fix flaky test case
…aces (#2317) (#2346) * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Trigger CLA check * Add Integration test coverage for AIO Producer * Add Tx related test cases + Add abort_tx call in Producer close * Fix styling * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Trigger CLA check * Trigger CLA check * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Style fixes * Addressed comments * Trigger CLA check * Address comments * Add Tx related test cases + Add abort_tx call in Producer close * Fix for serializing concurrent calls from a Async callback * Fix one test case * Fix styling * [NOGIL] Guard AdminClient against concurrent close()-vs-method-call races * Add handle check in common APIs and refactor overall handle logic * Remove conflict marker * Remove another conflict marker * Rename handle functions * Add more tests * Fix flaky test * Address comments * Fix corrupted _common.py * Fix other corrupted files * Fix flaky test case
…aces (#2317) (#2346) * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Trigger CLA check * Add Integration test coverage for AIO Producer * Add Tx related test cases + Add abort_tx call in Producer close * Fix styling * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Trigger CLA check * Trigger CLA check * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Style fixes * Addressed comments * Trigger CLA check * Address comments * Add Tx related test cases + Add abort_tx call in Producer close * Fix for serializing concurrent calls from a Async callback * Fix one test case * Fix styling * [NOGIL] Guard AdminClient against concurrent close()-vs-method-call races * Add handle check in common APIs and refactor overall handle logic * Remove conflict marker * Remove another conflict marker * Rename handle functions * Add more tests * Fix flaky test * Address comments * Fix corrupted _common.py * Fix other corrupted files * Fix flaky test case
…aces (#2317) (#2346) * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Trigger CLA check * Add Integration test coverage for AIO Producer * Add Tx related test cases + Add abort_tx call in Producer close * Fix styling * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Trigger CLA check * Trigger CLA check * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Style fixes * Addressed comments * Trigger CLA check * Address comments * Add Tx related test cases + Add abort_tx call in Producer close * Fix for serializing concurrent calls from a Async callback * Fix one test case * Fix styling * [NOGIL] Guard AdminClient against concurrent close()-vs-method-call races * Add handle check in common APIs and refactor overall handle logic * Remove conflict marker * Remove another conflict marker * Rename handle functions * Add more tests * Fix flaky test * Address comments * Fix corrupted _common.py * Fix other corrupted files * Fix flaky test case
* [NOGIL] Add CI verification jobs for free-threaded Python 3.14t (#2304) * Add CI verification jobs for free-threaded Python 3.14t - Add a Semaphore block running source package verification and integration tests on CPython 3.14t (classic and consumer group protocols). cimpl does not declare free-threading support yet, so importing it re-enables the GIL: these jobs validate the 3.14t toolchain and packaging until that declaration ships. - Skip the CI-only orjson install on free-threaded interpreters: no free-threaded orjson wheels exist and the source build would fail; the stdlib JSON fallback path stays covered. - Add a module-scoped autouse fixture (defined on free-threaded builds only) that warns when the GIL is re-enabled around a test module. Hard asserts are staged behind TODO FTS markers, to be enabled in the same PR that declares Py_MOD_GIL_NOT_USED. Interpreter detection follows the free-threading HOWTO: https://docs.python.org/3/howto/free-threading-python.html * Handle deps without free-threaded wheels in test setup On free-threaded (no-GIL) builds, the rules and json-fast extras' compiled deps (tink, google-re2, grpcio; orjson) ship no free-threaded wheels and fail to build from source, so: - Add requirements-tests-install-nogil.txt, a variant of requirements-tests-install.txt without those extras, and install it from source-package-verification.sh when the interpreter is free-threaded (detected via Py_GIL_DISABLED). - Exclude the schema_registry test modules that import tink/celpy/orjson at the top of the file from collection on free-threaded builds only; on regular builds a missing dep stays a loud collection error rather than a silent skip. Plain serdes coverage recovery is marked as a TODO NOGIL follow-up. * Style fixes * [NOGIL] Split schema_registry tests to narrow free-threaded test exclusions (#2309) * [NOGIL] Split schema_registry tests to narrow free-threaded test exclusions Each of test_avro_serdes.py, test_config.py, test_json_serdes.py, and test_proto_serdes.py is split into a plain file (no rules/encryption dependency) and a _rules file (CEL/encryption/JSONata-dependent tests). This lets the plain tests run on free-threaded (3.14t) builds, where tink/celpy/orjson have no free-threaded wheels, while only the _rules files stay excluded via conftest.py's collect_ignore. * Fix isort/black formatting in split schema_registry tests * [NOGIL] Fix post-rebase gap: exclude azure/tink-dependent schema_registry tests from free-threaded collection test_azure_aead.py, test_azure_client.py, test_azure_driver.py, and test_encrypt_executor.py import azure/tink unconditionally at module level but were missing from conftest.py's free-threaded collect_ignore list, causing collection errors on 3.14t CI. These tests were added to master after the original NOGIL schema_registry exclusion list (#2309) landed, so they weren't accounted for; rebasing onto master surfaced the gap. * [NOGIL] Fix post-rebase gap: update schema_registry free-threaded exclusions test_json.py's orjson import became pytest.importorskip upstream, so it no longer needs excluding. test_dlq_serdes.py is new post-rebase and hard- imports celpy, so it needs excluding. * [NOGIL] Exclude test_dlq.py from free-threaded schema_registry integration test collection * [NOGIL] Fix Producer.close() races with concurrent calls and with itself (#2313) * Fix Producer.close() races with concurrent calls and with itself Producer.close() previously raced with concurrent produce()/poll()/ flush()/produce_batch()/transaction calls and with itself when called from multiple threads, both leading to use-after-free/double-free on the underlying rd_kafka_t handle. Adds an active_calls/closing guard (Handle_enter_rk_use/Handle_exit_rk_use) so every method that touches self->rk registers itself before use, and close() drains in-flight calls before tearing down; a CAS on `closing` ensures only one concurrent close() call performs the actual teardown, with losing callers waiting for it to finish rather than racing it. Adds tests/parallel/test_producer_close_race.py covering each affected method racing close(), close() racing itself, and close()'s blocking behavior. Uses pytest-forked (POSIX only) so a regression segfault fails only that test. Integration tests against a real broker are still pending. * Replace pytest-forked with subprocess-based test isolation * Fix subprocess_isolated import and flaky close() timing assertion * Rename tests/parallel to tests/concurrency and add integration tests for Producer close()/transaction races * Clarified comment * Make Producer.close() non-blocking for concurrent callers, add reentrancy tests Concurrent close() calls now return False immediately with a warning instead of waiting for the CAS winner, since waiting could deadlock a caller that already holds an active_calls slot (e.g. a callback invoked from its own poll()/flush()). poll()/flush() now also exit early once closing is set instead of blocking the drain-wait. Fixes an ordering bug in produce_batch() where the topic handle was destroyed after releasing the active_calls slot. Adds integration tests for reentrant callbacks and close()'s internal flush delivering all messages, and documents close()-from-callback as unsupported. * Fix close() losers to wait for winner instead of returning False early * Fix CI flakiness in test_close_races_close_losers_wait_for_slow_winner * Move signal/slow-winner close() race tests to integration suite * Allow reentrant Producer calls from close()'s own delivery callback * Update docstring of test case * Harden race checks * [NOGIL] Restrict sharing of Consumer instances across threads (#2322) * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Style fixes * Fix flaky tests * Add TODO for handling concurrent calls inside callback * Addressed comments * Trigger CLA check * Address comments * Return consistent error when consumer is closed * Fix one test case * Fix styling * [NOGIL] Exclude Avro tests from running on free threaded python builds (#2335) * Exclude Avro tests from running on free threaded python builds * Exclude only fastavro * temoporarily install fastavro for docs * [NOGIL] Declare thread free support * [NOGIL] Improve Test Coverage for Producer + Some Edge Case Fixes (#2329) * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Trigger CLA check * Add Integration test coverage for AIO Producer * Add Tx related test cases + Add abort_tx call in Producer close * Fix flaky AIO Producer test case * Fix for serializing concurrent calls from a Async callback * Minor improvements to tests * Fix styling * Fix flaky test case * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Revert "Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant" This reverts commit 42b40cc. * Revert "Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant" This reverts commit f75dba3. * Add TODO * [NOGIL] Fix post-rebase gap: move rules test from test_proto_serdes.py to test_proto_serdes_rules.py * [NOGIL] Fix post-rebase gap: Fix misplaced CEL test and missing free-threaded collect_ignore entries * [NOGIL] Fix post-rebase gap: Initialize queue and add exit call in Consumer_consume * [NOGIL] Guard AdminClient against concurrent close() vs method-call races (#2317) (#2346) * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Trigger CLA check * Add Integration test coverage for AIO Producer * Add Tx related test cases + Add abort_tx call in Producer close * Fix styling * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Trigger CLA check * Trigger CLA check * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Style fixes * Addressed comments * Trigger CLA check * Address comments * Add Tx related test cases + Add abort_tx call in Producer close * Fix for serializing concurrent calls from a Async callback * Fix one test case * Fix styling * [NOGIL] Guard AdminClient against concurrent close()-vs-method-call races * Add handle check in common APIs and refactor overall handle logic * Remove conflict marker * Remove another conflict marker * Rename handle functions * Add more tests * Fix flaky test * Address comments * Fix corrupted _common.py * Fix other corrupted files * Fix flaky test case * [NOGIL] Fix borrowed-reference/stale-count races (#2334) (#2348) * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Trigger CLA check * Add Integration test coverage for AIO Producer * Add Tx related test cases + Add abort_tx call in Producer close * Fix styling * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Trigger CLA check * Trigger CLA check * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Style fixes * Addressed comments * Trigger CLA check * Address comments * Add Tx related test cases + Add abort_tx call in Producer close * Fix for serializing concurrent calls from a Async callback * Fix one test case * Fix styling * [NOGIL] Guard AdminClient against concurrent close()-vs-method-call races * Add handle check in common APIs and refactor overall handle logic * Remove conflict marker * Remove another conflict marker * Rename handle functions * Add more tests * Fix flaky test * Address comments * Fix corrupted _common.py * Fix other corrupted files * Fix flaky test case * [NOGIL] Fix borrowed-reference/stale-count races in Admin/Producer/Consumer parse loop * Fix list mutation in Admin API and a flaky test case * Fix data race on Handle.rk in close()/__exit__()/__len__() by making liveness checks atomic * Fix flaky test * Address comments * [NOGIL] Make Message class methods thread safe (#2350) * Make Message class methods thread safe * Add thread safety for eq and len functions, ref counting test cases * Use getters instead of direct access in Consumer methods * Fix mem leak * Use a single object-level lock for Message, snapshot via Message_copy, and fix __eq__ field comparison (treat unset/None alike, no crash)


Problem
An AdminClient instance can be shared across multiple threads — this is a common, previously-supported pattern. One long-standing race in this pattern: one thread destroys the Admin client (setting rk to NULL) while another thread is concurrently calling a librdkafka API on that same rk. This race has always existed, but on free-threaded builds the window widens significantly since there's no GIL to serialize the two threads.
Change
To close this race, every Admin API is now gated using active_calls/closing atomic variables. A CAS (compare-and-swap) construct has been added to exit() to correctly handle concurrent exit() calls. This mirrors the gating already implemented for Producer.
Other changes
What
Checklist
References
JIRA:
Test & Review
Open questions / Follow-ups