initial autocdc type1 docs - #57149
Open
suhuruli wants to merge 1 commit into
Open
Conversation
suhuruli
marked this pull request as ready for review
July 21, 2026 05:21
anew
reviewed
Jul 22, 2026
anew
left a comment
anew
left a comment
Contributor
There was a problem hiding this comment.
Overall looks good to me. See my comments.
|
|
||
| Given an ordered stream of change events, Auto CDC keeps the target table in sync with the source: | ||
|
|
||
| - **Inserts and updates** - For each key, the event with the highest sequence value wins. If no row exists for the key, it's inserted; if one exists, it's overwritten with the latest values. |
Contributor
There was a problem hiding this comment.
what's a sequence value? Do you want to introduce that first? (a set of columns defining the event order)
| - **Deletes** - Events that match a delete condition you supply remove the corresponding row from the target. | ||
| - **Out-of-order events** - Events don't have to arrive in order. Auto CDC uses the sequencing expression to determine the latest state per key, so a late-arriving event with a lower sequence value doesn't overwrite newer data. | ||
|
|
||
| This behavior implements **Slowly Changing Dimensions (SCD) Type 1**: the target keeps only the current version of each row, with no history of prior values. SCD Type 1 is the only mode currently supported. |
| | 2 | bob | 1 | UPSERT | | ||
| | 1 | alice_v2 | 2 | UPSERT | | ||
| | 2 | bob | 2 | DELETE | | ||
| | 3 | carol | 1 | UPSERT | |
Contributor
There was a problem hiding this comment.
this has no out-of order events for any of the keys, would it make sense?
| |----|----------|---------|--------| | ||
| | 1 | alice | 1 | UPSERT | | ||
| | 2 | bob | 1 | UPSERT | | ||
| | 1 | alice_v2 | 2 | UPSERT | |
Contributor
There was a problem hiding this comment.
_v2 is a bit weird for a name. Would "alicia" be a more natural update?
|
|
||
| - The **target must be a streaming table** that already exists in the pipeline. Create it with `create_streaming_table` (Python) or `CREATE STREAMING TABLE` (SQL) before defining the Auto CDC flow, or use the combined SQL form shown below that does both at once. | ||
| - The **source must be a streaming source** (read with `spark.readStream` in Python or `STREAM(...)` in SQL). CDC is an incremental operation over newly arriving change events. | ||
| - You must provide a **key** (one or more columns that identify a row) and a **sequencing expression** (used to order events per key). |
Contributor
There was a problem hiding this comment.
can the sequence really be an arbitrary expression?
| |-----------|----------|-------------| | ||
| | `target` | Yes | Name of the target streaming table that receives the changes. It must already be defined in the pipeline. | | ||
| | `source` | Yes | Name of the CDC source dataset to stream change events from. | | ||
| | `keys` | Yes | The column or columns that uniquely identify a row. A list of column names (strings) or `Column` objects, given as unqualified identifiers (for example `["id"]`, not `col("source.id")`). | |
Contributor
There was a problem hiding this comment.
for example, "id" or col("id"), but not "source.id"
| | `stored_as_scd_type` | No | The SCD type of the target. Only `1` (or `"1"`) is supported. | | ||
| | `name` | No | The name of the flow. Defaults to the target table name. | | ||
|
|
||
| If you specify neither `column_list` nor `except_column_list`, all columns from the source are written to the target. |
Contributor
There was a problem hiding this comment.
Maybe add: This is not recommended, because it would include the operation column in the target.
|
|
||
| Suppose the source receives two batches of change events. The first batch inserts two customers: | ||
|
|
||
| | id | name | version | op | |
Contributor
There was a problem hiding this comment.
again, no out-of-order events. What about an example that has an out-of order late update (or delete) in batch 2? And you run the pipeline after each batch? Then it wold demonstrate that the late event had no effect.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR documents Auto CDC (Change Data Capture) support for Spark Declarative Pipelines (SDP) in the programming guide.
It adds a new section, "Change Data Capture (CDC) with Auto CDC," to docs/declarative-pipelines-programming-guide.md, placed after the SQL programming section and before "Writing Data to External Targets with Sinks." The section covers:
No functional code is changed; this is documentation only.
Why are the changes needed?
AutoCDC Type 1 recently merged into 4.2, this adds prog guide.
Does this PR introduce any user-facing change?
Yes, but documentation only. It adds a new section to the Spark Declarative Pipelines programming guide. There is no change to behavior, APIs, or output.
How was this patch tested?
No tests were added; this is a documentation-only change. The Markdown was verified locally to ensure it is ASCII-only (per the repository convention) and that the internal anchor link (#spark-pipelines-run) resolves to an existing heading. All code snippets and API parameters were cross-checked against the current implementation (create_auto_cdc_flow, the autoCdcCommand SQL grammar, and the pipelines.proto definitions).
Was this patch authored or co-authored using generative AI tooling?
Yes