[Python 3.12.0] ModuleNotFoundError: No module named 'kafka.vendor.six.moves'

[pawel-swiecki-saucelabs]

Given

$ python --version
Python 3.12.0

$ pip freeze
kafka-python==2.0.2

my system: Darwin Kernel Version 22.5.0: Thu Jun 8 22:22:23 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T6020 arm64

Current behaviour

Python 3.12.0 (main, Oct  9 2023, 10:31:21) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from kafka.consumer.group import KafkaConsumer
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/[...]/lib/python3.12/site-packages/kafka/__init__.py", line 23, in <module>
    from kafka.consumer import KafkaConsumer
  File "/Users/[...]/lib/python3.12/site-packages/kafka/consumer/__init__.py", line 3, in <module>
    from kafka.consumer.group import KafkaConsumer
  File "/Users/[...]/lib/python3.12/site-packages/kafka/consumer/group.py", line 13, in <module>
    from kafka.consumer.fetcher import Fetcher
  File "/Users/[...]/lib/python3.12/site-packages/kafka/consumer/fetcher.py", line 19, in <module>
    from kafka.record import MemoryRecords
  File "/Users/[...]/lib/python3.12/site-packages/kafka/record/__init__.py", line 1, in <module>
    from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder
  File "/Users/[...]/lib/python3.12/site-packages/kafka/record/memory_records.py", line 27, in <module>
    from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder
  File "/Users/[...]/lib/python3.12/site-packages/kafka/record/legacy_records.py", line 50, in <module>
    from kafka.codec import (
  File "/Users/[...]/lib/python3.12/site-packages/kafka/codec.py", line 9, in <module>
    from kafka.vendor.six.moves import range
ModuleNotFoundError: No module named 'kafka.vendor.six.moves'
>>> 

Expected behaviour

Python 3.12.0 (main, Oct  9 2023, 10:31:21) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from kafka.consumer.group import KafkaConsumer
>>>

(no error)

[pawel-swiecki-saucelabs]

On the latest Python 3.11 it works:

$ python --version
Python 3.11.6

$ pip freeze
kafka-python==2.0.2

$ python
Python 3.11.6 (main, Oct 10 2023, 17:17:53) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from kafka.consumer.group import KafkaConsumer
>>> KafkaConsumer
<class 'kafka.consumer.group.KafkaConsumer'>
>>>

[pawel-swiecki-saucelabs]

It seems this might be resolved by bumping /vendor/six.py to 1.16.0. I think currently it's 1.11.0.

$ pip freeze
six==1.11.0

$ python
Python 3.12.0 (main, Oct  9 2023, 10:31:21) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from six.moves import range
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'six.moves'

however:

$ pip freeze
six==1.16.0

$ python
Python 3.12.0 (main, Oct  9 2023, 10:31:21) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from six.moves import range
>>>

(On Python 3.11.6 from six.moves import range works with both six versions.)

However, this might not be enough, see benjaminp/six#376 (this one might not be relevant, but there might be other random issues like this).

[pawel-swiecki-saucelabs]

It turns out there already is a relevant PR waiting for review.

[mnito]

Until the above PR is merged and a new version of this library is released, running the snippet below before importing kafka-python works around the issue (assuming six>=1.16.0 is installed):

import six

if sys.version_info >= (3, 12, 0):
    sys.modules['kafka.vendor.six.moves'] = six.moves