Skip to content

Add databricks-sqlalchemy to the Databricks handler requirements - #9

Open
maharanay22 wants to merge 1 commit into
mindsdb:mainfrom
maharanay22:fix-databricks-sqlalchemy-dependency
Open

maharanay22 wants to merge 1 commit into
mindsdb:mainfrom
maharanay22:fix-databricks-sqlalchemy-dependency

Conversation

@maharanay22

Copy link
Copy Markdown

Description

The Databricks integration can't be used on a fresh install. After pip install "mindsdb[databricks]", the handler fails to load with:

No module named 'databricks.sqlalchemy'

so CREATE DATABASE ... WITH ENGINE = 'databricks' doesn't work and information_schema.handlers shows the import error

Repro: in a clean virtualenv, run pip install "mindsdb[databricks]" from main, then from mindsdb.integrations.handlers.databricks_handler import import_error; print(import_error)

Root cause: the handler requirements were bumped to databricks-sql-connector==4.2.3 (#12334 in the old repo). Starting with 4.0, the connector no longer ships the SQLAlchemy dialect, which moved to the separate databricks-sqlalchemy package. databricks_handler.py still does from databricks.sqlalchemy import DatabricksDialect. CI didn't catch it because test_databricks.py skips the whole module when the import fails, so its 39 tests have been reported as skipped instead of failing

The fix pins databricks-sqlalchemy==2.0.10 in the handler requirements. That's the current release, and it supports databricks-sql-connector>=4.0.1 and sqlalchemy>=2.0.21, which matches this repo. The databricks extra and the Docker images pick it up from the same file. I also mapped the package to the databricks module in tests/scripts/check_requirements.py, the same way snowflake-sqlalchemy is mapped, since deptry otherwise reports it as unused (DEP002)

I left the skip-on-ImportError pattern in the test as it is, because most handler tests use it on purpose

I used an AI assistant to help investigate this and draft the change and tests, and I reviewed and tested it myself

Fixes: no issue to link, because issues are disabled on this repo. The bug is described above

Type of change

  • 🐛 Bug fix (non-breaking change which fixes an issue)

Verification Process

To ensure the changes are working as expected:

  • Test Location: mindsdb/integrations/handlers/databricks_handler/requirements.txt, tests/unit/handlers/community_handlers/test_databricks.py
  • Verification Steps: create two clean Python 3.11 virtualenvs, install .[databricks] from main in one and from this branch in the other, then import the handler and run the Databricks unit tests

Before (main):

$ uv pip list | grep -i -E "^databricks|^sqlalchemy"
databricks-sql-connector                 4.2.3
sqlalchemy                               2.1.4
$ python -c "from mindsdb.integrations.handlers.databricks_handler import import_error, Handler; print(repr(import_error)); print(Handler)"
ModuleNotFoundError("No module named 'databricks.sqlalchemy'")
None
$ pytest tests/unit/handlers/community_handlers/test_databricks.py -q -rs
SKIPPED [39] tests\unit\handlers\community_handlers\test_databricks.py: Databricks handler not installed
39 skipped in 2.33s

After (this branch):

$ uv pip list | grep -i -E "^databricks|^sqlalchemy"
databricks-sql-connector                 4.2.3
databricks-sqlalchemy                    2.0.10
sqlalchemy                               2.1.4
$ python -c "from mindsdb.integrations.handlers.databricks_handler import import_error, Handler; print(repr(import_error)); print(Handler)"
None
<class 'mindsdb.integrations.handlers.databricks_handler.databricks_handler.DatabricksHandler'>
$ pytest tests/unit/handlers/community_handlers/test_databricks.py -q -rs
39 passed in 4.98s

deptry with the same settings as tests/scripts/check_requirements.py reports no errors for the Databricks handler after the change

Additional Media:

  • I have attached a brief loom video or screenshots showcasing the new functionality or change. (Terminal before and after output is included above in place of a video)

Checklist:

  • My code follows the style guidelines(PEP 8) of MindsDB.
  • I have appropriately commented on my code, especially in complex areas.
  • Necessary documentation updates are either made or tracked in issues.
  • Relevant unit and integration tests are updated or added. (No new tests here, the existing 39 Databricks unit tests now run instead of being skipped)

@maharanay22
maharanay22 requested a review from a team as a code owner October 13, 2026 15:48
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@maharanay22

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

@maharanay22

Copy link
Copy Markdown
Author

recheck

databricks-sql-connector 4.x no longer ships the SQLAlchemy dialect,
it moved to the separate databricks-sqlalchemy package. The handler
still imports databricks.sqlalchemy, so after the bump to
databricks-sql-connector 4.2.3 it failed to load with
"No module named 'databricks.sqlalchemy'", and the unit tests for it
were skipped as "Databricks handler not installed".

Pin databricks-sqlalchemy 2.0.10, which supports connector 4.x and
SQLAlchemy 2, and map it to the databricks module for deptry.
@maharanay22
maharanay22 force-pushed the fix-databricks-sqlalchemy-dependency branch from 7d0c1b2 to 2d420d5 Compare October 13, 2026 15:57

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant