Skip to content

feat(appkit): migrate analytics to the modular @databricks/sdk-* - #562

Draft
MarioCadenas wants to merge 1 commit into
mainfrom
analytics-migration-sdk
Draft

feat(appkit): migrate analytics to the modular @databricks/sdk-*#562
MarioCadenas wants to merge 1 commit into
mainfrom
analytics-migration-sdk

Conversation

@MarioCadenas

Copy link
Copy Markdown
Collaborator

What

Migrates the analytics stack (SQLWarehouseConnector + type-generator) off the legacy monolithic @databricks/sdk-experimental onto the new modular per-service @databricks/sdk-* SDK (v0.46.0, ESM-only), behind the existing packages/shared/src/workspace-client/ facade seam.

Analytics doesn't touch the SDK directly — it delegates to SQLWarehouseConnector, which uses two services: warehouses and statementExecution. Both move together in this increment; every other service (files/genie/jobs/serving/currentUser/apiClient) still routes through the legacy client behind the same facade (the seam's designed incremental state).

How

  • New boundary module packages/shared/src/workspace-client/modular.ts — the sole importer of @databricks/sdk-* (oxlint no-restricted-imports boundary), mirroring legacy.ts. Builds per-service WarehousesClient / StatementExecutionClient; maps wrapper options → ClientOptions (host scheme-normalization, PAT empty-token privilege-escalation guard, profile); stamps process-global client-info (sanitized, best-effort).
  • Connector + type-generator rewritten to the modular API: method renames (getStatementgetStatementResult, getStatementResultChunkNgetResultData), camelCase response model, CallOptions { signal } for cancellation.
  • pnpm patch (patches/@databricks__sdk-statementexecution@0.46.0.patch) restores the undocumented Reyden attachment response field that the SDK's generated unmarshal transform would otherwise strip. Version-pinned — re-apply on any SDK bump; an upstream fix is preferred long-term.

Runtime bugs found + fixed (caught against live warehouses, not unit tests)

  1. Host scheme — the modular SDK doesn't prepend https://; a bare DATABRICKS_HOSTInvalid URL. Fixed via normalizeHost.
  2. Client-infosetProduct("@databricks/appkit") throws on @//; sanitized + wrapped best-effort.
  3. ApiError.code — the modular ApiError carries the Databricks error code on .code, not .errorCode; the connector now reads either, so the arrow disposition/format capability-rejection fallback (INLINE↔EXTERNAL_LINKS) fires again.
  4. BigInt serialization — the modular SDK types row/byte counts as bigint; Reyden's cached INLINE+ARROW_STREAM result then broke JSON.stringify. Counts are coerced back to number at the connector boundary.

Testing

  • Full appkit + shared suites green (3877 passed / 1 skipped), including new regression tests for the .code read and bigint normalization.
  • pnpm -r typecheck, lint, format clean.
  • Verified live against standard + Reyden serverless warehouses: JSON + arrow (INLINE attachment and EXTERNAL_LINKS), OBO (.obo.sql), warehouse auto-start, metric views.

Notes

  • Metric views run entirely on the migrated services (they compile to SQL); no separate migration needed.
  • Regenerated the workspace-client API docs to match the new types (included).

Migrate the analytics stack (SQLWarehouseConnector + type-generator) off
the legacy monolithic @databricks/sdk-experimental onto the new modular
per-service @databricks/sdk-* SDK (v0.46.0, ESM-only), behind the existing
workspace-client facade seam. The two services analytics depends on —
warehouses and statementExecution — move together; every other service
still routes through the legacy client (mixed state by design).

- New packages/shared/src/workspace-client/modular.ts is the sole importer
  of @databricks/sdk-* (oxlint no-restricted-imports boundary), mirroring
  legacy.ts. Builds per-service WarehousesClient / StatementExecutionClient;
  maps wrapper options -> ClientOptions (host scheme-normalization, PAT
  empty-token guard, profile); stamps process-global client-info
  (sanitized, best-effort).
- Connector + type-generator rewritten to the modular API: method renames
  (getStatement -> getStatementResult, getStatementResultChunkN ->
  getResultData), camelCase response model, CallOptions { signal }.
- statementExecution relies on a pinned pnpm patch that restores the
  undocumented Reyden `attachment` field the SDK's unmarshal transform
  would otherwise strip.
- Coerce the SDK's bigint row/byte counts back to number at the connector
  boundary so INLINE + ARROW_STREAM results stay JSON-serializable
  (cache / SSE frames).
- Read the modular ApiError's `.code` (not only the legacy `.errorCode`) so
  the arrow disposition/format capability-rejection fallback still fires.

Verified against live warehouses (standard + Reyden serverless): JSON and
arrow (INLINE attachment + EXTERNAL_LINKS), OBO, warehouse auto-start, and
metric views. Full appkit + shared suite green (3877 tests).

Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
@MarioCadenas
MarioCadenas force-pushed the analytics-migration-sdk branch from 9e1a7d5 to 2931182 Compare September 10, 2026 14:43
@github-actions

Copy link
Copy Markdown
Contributor

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

@databricks/appkit

npm tarball (packed): 1.1 MB (+7.5 KB) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 1.1 MB (+7.6 KB) 414 KB (+3.2 KB)
Type declarations 425 KB (+1.7 KB) 153 KB (+580 B)
Source maps 2.3 MB (+14 KB) 774 KB (+5.1 KB)
Other 11 KB 3.7 KB
Total 3.9 MB (+23 KB) 1.3 MB (+8.8 KB)
Per-entry composition (own code — deps external (as shipped))
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
. 96 KB (+727 B) 2.5 KB 99 KB (+727 B) external 314 KB (+1.6 KB)
./beta 92 KB (+560 B) 456 B (-1 B) 93 KB (+559 B) external 279 KB (+1.2 KB)
./testing 17 KB (+381 B) 0 B 17 KB (+381 B) external 53 KB (+1.0 KB)
./tsdown 520 B 0 B 520 B external 813 B
./type-generator 23 KB (+609 B) 0 B 23 KB (+609 B) external 67 KB (+1.4 KB)

Chunks:

Entry Chunk Load Size (gz)
. index.js initial 92 KB
. utils.js initial 4.0 KB
. remote-tunnel-manager.js lazy 2.5 KB
./beta beta.js initial 76 KB
./beta stream-manager.js initial 5.8 KB
./beta wide-event-emitter.js initial 3.2 KB
./beta databricks.js initial 3.2 KB
./beta configuration.js initial 2.3 KB
./beta service-context.js initial 1.3 KB
./beta modular.js initial 867 B
./beta client-options.js initial 220 B
./beta supervisor-api.js lazy 192 B
./beta databricks.js lazy 141 B
./beta index.js lazy 123 B
./testing index.js initial 17 KB
./tsdown index.js initial 520 B
./type-generator index.js initial 23 KB

@databricks/appkit-ui

npm tarball (packed): 350 KB (+81 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 395 KB 132 KB
Type declarations 229 KB (+155 B) 84 KB (+113 B)
Source maps 766 KB 253 KB
CSS 16 KB 3.2 KB
Total 1.4 MB (+155 B) 473 KB (+113 B)
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
./js 5.3 KB 49 KB 55 KB 208 KB 14 KB
./js/beta 20 B 0 B 20 B 0 B 0 B
./react 432 KB 49 KB 481 KB 1.3 MB 177 KB
./react/beta 1.0 KB 0 B 1.0 KB 0 B 1.9 KB

Chunks:

Entry Chunk Load Size (gz)
./js index.js initial 5.2 KB
./js chunk initial 120 B
./js apache-arrow lazy 49 KB
./js/beta beta.js initial 20 B
./react index.js initial 430 KB
./react tslib initial 2.1 KB
./react apache-arrow lazy 49 KB
./react/beta beta.js initial 1.0 KB

@github-actions

Copy link
Copy Markdown
Contributor

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 34490915012 -R databricks/appkit -n appkit-template-0.74.0-pr.11cbd3f-analytics-migration-sdk-562 -D appkit-pr-562 \
  && unzip -o "appkit-pr-562/appkit-template-0.74.0-pr.11cbd3f-analytics-migration-sdk-562.zip" -d "appkit-pr-562" \
  && databricks apps init --template "appkit-pr-562"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant