Generate PyDABs reference doc pages from generated modules - #6727
Open
Sankalp-Mittal wants to merge 3 commits into
Open
Sankalp-Mittal wants to merge 3 commits into
Sankalp-Mittal wants to merge 3 commits into
Conversation
The Sphinx .rst pages under python/docs/ were hand-written and listed only 5 resources, while codegen now emits 31 resource modules — so the published Python reference docs (databricks.github.io/cli/python) went stale. Generate the per-resource .rst pages and index.rst toctree in pydabs-codegen, driven by scanning the generated databricks/bundles/*/ package dirs (decoupled from RESOURCE_NAMESPACE). The existing hand-written pages regenerate byte-identical; core.rst stays hand-written. CI's generate-check + git diff now enforces the doc list never drifts from the generated resources. Co-authored-by: Isaac <no-reply@databricks.com>
Extract the RST page and index layout from generated_docs.py into doc_page.rst.tmpl and doc_index.rst.tmpl, loaded via string.Template like the existing *.py.tmpl codegen templates. This lets the doc structure be reviewed and tracked separately from the generation code. Output is byte-identical. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
added this pull request to stack #6730
September 17, 2026 10:08
janniklasrose
left a comment
janniklasrose
left a comment
Member
There was a problem hiding this comment.
tested task pydabs-docs:
/Users/jan.rose/pub/cli.worktrees/pr-6727/python/databricks/bundles/sql_warehouses/_models/spot_instance_policy.py:docstring of databricks.bundles.sql_warehouses._models.spot_instance_policy.SpotInstancePolicy:5: ERROR: Malformed table.
+-------+--------------------------------------+--------------------------------+
| Cloud | COST_OPTIMIZED | RELIABILITY_OPTIMIZED |
+-------+--------------------------------------+--------------------------------+ [docutils]
/Users/jan.rose/pub/cli.worktrees/pr-6727/python/databricks/bundles/sql_warehouses/_models/spot_instance_policy.py:docstring of databricks.bundles.sql_warehouses._models.spot_instance_policy.SpotInstancePolicy:8: WARNING: Blank line required after table. [docutils]
/Users/jan.rose/pub/cli.worktrees/pr-6727/python/databricks/bundles/sql_warehouses/_models/spot_instance_policy.py:docstring of databricks.bundles.sql_warehouses._models.spot_instance_policy.SpotInstancePolicy:9: WARNING: Line block ends without a blank line. [docutils]
/Users/jan.rose/pub/cli.worktrees/pr-6727/python/databricks/bundles/sql_warehouses/_models/spot_instance_policy.py:docstring of databricks.bundles.sql_warehouses._models.spot_instance_policy.SpotInstancePolicy:9: ERROR: Malformed table.
+-------+--------------------------------------+--------------------------------+
| AWS | On Demand Driver with Spot Executors | On Demand Driver and [docutils]
/Users/jan.rose/pub/cli.worktrees/pr-6727/python/databricks/bundles/sql_warehouses/_models/spot_instance_policy.py:docstring of databricks.bundles.sql_warehouses._models.spot_instance_policy.SpotInstancePolicy:11: WARNING: Blank line required after table. [docutils]
/Users/jan.rose/pub/cli.worktrees/pr-6727/python/databricks/bundles/core/_resources.py:docstring of databricks.bundles.core._resources.Resources:23: WARNING: py:meth reference target not found: add_job [ref.meth]
Collaborator
Integration test reportCommit: eb02251
Top 3 slowest tests (at least 2 minutes):
|
Contributor
Author
This is a issue with the comment in the protos, we would have to change the upstream descriptions, or special case this, I would prefer doing the former |
Member
please fix this upstream. but with the auto-gen + pipeline you're building in this stack, we need a way to surface these. erroring is probably not ok, ideally this should fail at SDK bump stage with an override that we can burn down again upstream |
Contributor
Author
Just a clarification this is not an error but a warning so the docs would be generated but they would just be broken |
The sql.SpotInstancePolicy description is a hard-wrapped ASCII grid table that docutils rejects as malformed, so the sql_warehouses page rendered broken and the Sphinx build logged errors. Add a burn-down override list in jsonschema_patch that rewrites such upstream descriptions into valid reStructuredText until the proto comment is fixed upstream (see sqlgateway/scheduler/api/proto/endpoint_common.proto). The override raises if it becomes a no-op, flagging entries to remove once upstream lands. Pairs with the strict (-W) docs build added later in the stack. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
Added an override here, and the hard error is added to the other PR on this stack |
janniklasrose
approved these changes
Sep 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
.rstdoc pages from the generated modules (base) ← this PRgh-pages+ make the docs build strictWhy
The PyDABs Python reference docs (https://databricks.github.io/cli/python/) went stale. Codegen now emits 31 resource modules under
python/databricks/bundles/*/, but the Sphinx.rstpages were hand-written and listed only 5 (core,jobs,pipelines,schemas,volumes).What
Generate the per-resource
.rstpages and theindex.rsttoctree insidepydabs-codegen:python/codegen/codegen/generated_docs.py—write_docs()scans the generateddatabricks/bundles/*/package dirs (decoupled fromRESOURCE_NAMESPACE, robust to future renames), writes one page per resource, and regenerates theindex.rsttoctree. Called frommain.pyafter the existing generation.core.rststays hand-written. The 4 previously hand-written resource pages (jobs,pipelines,schemas,volumes) regenerate byte-identical — no diff.index.rst's prose header is preserved verbatim; only its toctree grows.SQL Warehouses,MCP Services) handled via a small override map.doc_page.rst.tmplanddoc_index.rst.tmpl(loaded viastring.Template, like the existing*.py.tmplcodegen templates) so it can be reviewed separately from the generation code.python/Taskfile.yml: added the generated doc outputs to the task'sgenerates:list.Also fold in a related fix surfaced in review: some upstream API descriptions aren't valid reStructuredText and break the Sphinx build (e.g.
sql.SpotInstancePolicy, whose comment is a hard-wrapped ASCII grid table docutils rejects as malformed).jsonschema_patch.override_descriptionsrewrites such descriptions into valid RST until the proto comment is fixed upstream (seesqlgateway/scheduler/api/proto/endpoint_common.proto). The override raises if it becomes a no-op, flagging entries to remove once upstream lands.-Wdocs gate that surfaces any new such breakage (at PR/SDK-bump time) lands in Auto-publish PyDABs reference docs to gh-pages on release #6729.CI's
generate-check+git diff --exit-codenow enforces the documented resource list can never drift from the generated modules.Testing
./task pydabs-codegen— regenerates code + 31.rstpages +index.rst; the 4 existing pages show no diff; idempotent on re-run../task pydabs-docs— Sphinx build succeeds and now emits all 32databricks.bundles.*.htmlpages; thesql_warehousespage renders a proper table instead of the malformed one.This pull request and its description were written by Isaac.