Skip to content

Wait for split producer batches during flush - #3199

Open
AryanRG wants to merge 1 commit into
dpkp:masterfrom
AryanRG:fix/flush-split-batches
Open

AryanRG wants to merge 1 commit into
dpkp:masterfrom
AryanRG:fix/flush-split-batches

Conversation

@AryanRG

@AryanRG AryanRG commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Wait for split producer batches during flush

KafkaProducer.flush() can raise UnknownError("Future not done") after an oversized batch is split for retry. Rebinding wakes the original batch's latch, but that batch stays unfinished.

Follow each record's replacement batch and wait for its completion latch. Further splits rebind the next wait. The remaining original-batch timeout covers all replacement waits, and the latch preserves the existing callback-completion barrier. Failed records still count as completed for flush.

Extends one broker-free regression through send() and flush() to check both delivery and the final send callback.

Validation on Python 3.13.12 / macOS arm64

  • The regression exposes the original error on master and the callback-order failure on the first patch, then passes with this version.
  • Default broker-free suite rerun passes 2,760 tests with one generated-stub skip. An initial consumer rejoin assertion is recorded. The full unchanged-base comparison passes.
  • 44 separate public-producer probes pass with explicit net='selector' and net='asyncio', asserting actual backend identity.
  • Changed-module Pylint, unicode lint and diff checks pass.

Real-broker integration and other Python versions remain untested locally.

@AryanRG

AryanRG commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

The Python 3.8 and 3.9 jobs stop in setup-python because those versions are unavailable on Ubuntu 26.04. The failure occurs before tests start.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant