diff --git a/client-sdks/advanced/checkpoint-requests.mdx b/client-sdks/advanced/checkpoint-requests.mdx index d630038a..bbd613dd 100644 --- a/client-sdks/advanced/checkpoint-requests.mdx +++ b/client-sdks/advanced/checkpoint-requests.mdx @@ -42,10 +42,13 @@ Before creating a checkpoint request: ```typescript JavaScript/TypeScript +// Requires @powersync/web 2.3.0, @powersync/react-native 2.2.0, +// @powersync/node 1.0.0, or @powersync/capacitor 0.9.0 or later await db.connect(connector, { checkpointMode: 'requests' }); ``` ```dart Dart +// Requires powersync 2.4.0 or later await db.connect( connector: connector, options: SyncOptions(checkpointMode: .requests()), @@ -53,6 +56,7 @@ await db.connect( ``` ```kotlin Kotlin +// Requires PowerSync Kotlin SDK 1.15.0 or later database.connect( connector, options = SyncOptions( @@ -62,6 +66,7 @@ database.connect( ``` ```swift Swift +// Requires PowerSync Swift SDK 1.16.0 or later try await database.connect( connector: connector, options: ConnectOptions(checkpointMode: .requests()) @@ -69,9 +74,10 @@ try await database.connect( ``` ```cs .NET +// Requires PowerSync.Common 0.1.5 or later await database.Connect( connector: connector, - options: new PowerSyncConnectOptions(checkpointMode: new CheckpointMode.Requests()) + options: new PowerSyncConnectionOptions(checkpointMode: new CheckpointMode.Requests()) ); ``` @@ -284,9 +290,13 @@ The upload response remains the authority on whether your backend accepted, chan ## Asynchronous Upload Backends -The managed flow assumes that `uploadData()` returns only after your backend commits the uploaded changes to the source database. If your backend queues uploads for later processing, use custom checkpoint requests. This feature is available for customers on [Team and Enterprise](https://www.powersync.com/pricing) plans. + + This feature is available for customers on [Team and Enterprise](https://www.powersync.com/pricing) plans. + + +The managed flow assumes that `uploadData()` returns only after your backend commits the uploaded changes to the source database. If your backend queues uploads for later processing, use custom checkpoint requests. -Follow the [Custom Write Checkpoints source-side setup](/handling-writes/custom-write-checkpoints#sync-rules-requirements), including its `checkpoint_requests` event definition. The `checkpoint` column stores the checkpoint request ID generated by the client. +Follow [Setting Up Custom Checkpoint Requests](/handling-writes/custom-write-checkpoints#setting-up-custom-checkpoint-requests) for the source database, Sync Config, and backend changes. If older app versions still use legacy Custom Write Checkpoints, see [Migrating to Custom Checkpoint Requests](/handling-writes/custom-write-checkpoints#migrating-to-custom-checkpoint-requests). The difference on the client is that the PowerSync Client SDK generates the checkpoint request ID and sends it to your backend through `CustomCheckpointRequestConnector`. diff --git a/configuration/powersync-service/self-hosted-instances.mdx b/configuration/powersync-service/self-hosted-instances.mdx index 2e57d460..d86ff020 100644 --- a/configuration/powersync-service/self-hosted-instances.mdx +++ b/configuration/powersync-service/self-hosted-instances.mdx @@ -467,6 +467,10 @@ The PowerSync Service requires a storage database to store the data and metadata Storage version to use when deploying a Sync Config that does not specify a storage version. To opt in to [storage version 4](/sync/advanced/storage-version-4) (Beta), set `config.storage_version: 4` in your Sync Config. An explicit `config.storage_version` takes precedence over this default. Available since Service version 1.26.0. + + + Postgres bucket storage does not currently support version 4. + #### MongoDB Storage diff --git a/handling-writes/custom-write-checkpoints.mdx b/handling-writes/custom-write-checkpoints.mdx index 3af866e7..106eec80 100644 --- a/handling-writes/custom-write-checkpoints.mdx +++ b/handling-writes/custom-write-checkpoints.mdx @@ -1,217 +1,217 @@ --- title: "Data Pipelines" sidebarTitle: "Data Pipelines" -description: "Use Custom Write Checkpoints to track asynchronous data uploads through chained data pipelines and confirm write completion on the client." +description: "Use custom checkpoints to keep client data stable when your backend applies uploads to the source database asynchronously." --- -**Availability**: -Custom Write Checkpoints are available for customers on our [Team and Enterprise](https://www.powersync.com/pricing) plans. +**Availability**: Custom checkpoints are available for customers on our [Team and Enterprise](https://www.powersync.com/pricing) plans. - -The alpha [Checkpoint Requests](/client-sdks/advanced/checkpoint-requests) API uses the term "custom checkpoint requests" for asynchronous upload backends. Client support is currently available for Swift. This page retains the previous "Custom Write Checkpoints" name for the source-side configuration. - +PowerSync uses write checkpoints to keep client data [consistent](/architecture/consistency). After the client uploads its local writes, it obtains a checkpoint that marks the source database position after those writes. The client applies downloaded data only once the sync checkpoint includes that write checkpoint. This is why your [write endpoint must be synchronous](/handling-writes/writing-client-changes): the default checkpoint marks the source database position when `uploadData()` returns, so the uploaded changes must already be in the source database at that moment. -To ensure [consistency](/architecture/consistency), PowerSync relies on Write Checkpoints. These checkpoints ensure that clients have uploaded their own local changes/mutations to the server before applying downloaded data from the server to the local database. +Some backends cannot process uploads synchronously. In a chained data pipeline, uploads first go to a queue or an intermediate database and reach the source database later. With the default checkpoints, this makes client data flicker: -The essential requirement is that the client must get a Write Checkpoint after uploading its last write/mutation. Then, when downloading data from the server, the client checks whether the Write Checkpoint is part of the largest [sync checkpoint](https://github.com/powersync-ja/powersync-service/blob/main/docs/specs/sync-protocol.md) received from the server (i.e. from the PowerSync Service). If it is, the client applies the server-side state to the local database. +1. The client uploads a change. Your backend accepts it and queues it, and `uploadData()` returns. +2. The client obtains a write checkpoint. The PowerSync Service marks the current source database position, which does not include the queued change. +3. The client receives that checkpoint and applies the server state. The change is missing, so the client reverts it locally. +4. The pipeline writes the change to the source database. The Service syncs it, and the client applies it again. -The default Write Checkpoints implementation relies on uploads being acknowledged _synchronously_, i.e. the change persists in the source database (to which PowerSync is connected) before the [`uploadData` call](/configuration/app-backend/client-side-integration) completes. +Custom checkpoints solve this. Instead of the Service marking the source database position when the client asks, your backend writes a checkpoint record into a table in the source database at the end of the pipeline. The record replicates to the Service through the same replication stream as your data, so the checkpoint always follows the uploaded changes. An event definition in your Sync Config tells the Service how to read the record. -Problems occur if the persistence in the source database happens _asynchronously_. If the client's upload is meant to mutate the source database (and eventually does), but this is delayed, it will effectively seem as if the client's uploaded changes were reverted on the server, and then applied again thereafter. +## Choosing a Flow -Chained *data pipelines* are a common example of asynchronous uploads -- e.g. data uploads are first written to a different upstream database, or a separate queue for processing, and then finally replicated to the 'source database' (to which PowerSync is connected). +PowerSync supports two custom checkpoint flows. Use custom checkpoint requests for new implementations. -For example, consider the following data pipeline: +| | Custom checkpoint requests | Legacy Custom Write Checkpoints | +| --- | --- | --- | +| Checkpoint ID | Generated by the PowerSync Client SDK | Generated by your backend | +| Client integration | Implement `postCheckpointRequest()` on your connector. The SDK calls it after each upload. | Pass the checkpoint number to `transaction.complete()` in `uploadData()`. | +| Event definition | `checkpoint_requests` | `write_checkpoints` | +| Records in the Service | Expire after a retention period | Retained | +| Status | Alpha. Requires PowerSync Service 1.24.0 or later. | Stable | -1. The client makes a change locally and the local database is updated. -2. The client uploads this change to the server. -3. The server resolves the request and writes the change into an intermediate database (not the source database yet). -4. The client thinks the upload is complete (i.e. persisted into the source database). It requests a Write Checkpoint from the PowerSync Service. -5. The PowerSync Service increments the replication `HEAD` in the source database, and creates a Write Checkpoint for the client. The Write Checkpoint number is returned and recorded in the client. -6. The PowerSync Service replicates past the previous replication `HEAD` (but the changes are still not present in the source database). -7. It should be fine for the client to apply the state of the server to the local database. But the server state does not include the client's uploaded changes mentioned in #2. This is the same as if the client's uploaded changes were rejected (not applied) by the server. This results in the client reverting the changes in its local database. -8. Eventually the change is written to the source database, and increments the replication `HEAD`. -9. The PowerSync Service replicates this change and sends it to the client. The client then reapplies the changes to its local database. +Custom checkpoint requests are part of the [Checkpoint Requests](/client-sdks/advanced/checkpoint-requests) API. See that page for the supported client SDKs. The legacy flow continues to work and is documented in [Legacy Custom Write Checkpoints](#legacy-custom-write-checkpoints). -In the above case, the client may see the Write Checkpoint before the data has been replicated. This will cause the client to revert its changes, then apply them again later when it has actually replicated, causing data to "flicker" in the app. +From PowerSync Service 1.26.0, a Sync Config can define only one of the two events. If you have app versions in production that use the legacy flow, see [Migrating to Custom Checkpoint Requests](#migrating-to-custom-checkpoint-requests). -For these use cases, Custom Write Checkpoints should be implemented. +## Setting Up Custom Checkpoint Requests -## Custom Write Checkpoints +With custom checkpoint requests, the PowerSync Client SDK generates an increasing checkpoint request ID and sends it to your backend after each upload. Your backend writes the ID into a checkpoints table in the source database. When the Service replicates the record, the client knows that its uploads are in the source database. -_Custom Write Checkpoints_ allow the developer to define Write Checkpoints and insert them into the replication stream directly, instead of relying on the PowerSync Service to create and return them. An example of this is having the backend persist Write Checkpoints to a dedicated table which is processed as part of the replication stream. + + + Create a table in your source database that stores the latest checkpoint request ID for each PowerSync client: -The PowerSync Service then needs to process the (ordered) replication events and correlate the checkpoint table changes to Write Checkpoint events. + ```sql + CREATE TABLE checkpoints ( + user_id TEXT NOT NULL, + client_id TEXT NOT NULL, + checkpoint BIGINT NOT NULL, + PRIMARY KEY (user_id, client_id) + ); + ``` -## Example Implementation + - `user_id` is the authenticated user. + - `client_id` is the PowerSync client ID. Each local database has its own client ID, so one user can have many clients. + - `checkpoint` is the checkpoint request ID. IDs are 64-bit integers, so use a `BIGINT` or equivalent column. -A self-hosted Node.js demo with Postgres is available here: + Column names can differ. The event definition in the next steps maps your columns to these fields. + + + For Postgres, add the table to the PowerSync [publication](/configuration/source-db/setup): - - + ```sql + CREATE PUBLICATION powersync FOR TABLE lists, todos, checkpoints; + ``` -## Implementation Details + For other source databases, the Service replicates every table that your Sync Config references, including tables in event definitions. Complete the same [table setup](/configuration/source-db/setup) as for your other tables, such as enabling CDC for a SQL Server table. + + + Add a `checkpoint_requests` event definition to your Sync Config. Its payload query must return the fields `user_id`, `client_id`, and `checkpoint`: -This outlines what a Custom Write Checkpoints implementation entails. + ```yaml + config: + edition: 3 -### Custom Write Checkpoint Table + event_definitions: + checkpoint_requests: + payloads: + - SELECT user_id, client_id, checkpoint FROM checkpoints -Create a dedicated `checkpoints` table, which should contain the following checkpoint payload information in some form: + streams: + todos: + query: SELECT * FROM todos WHERE owner_id = auth.user_id() + ``` -```TypeScript -export type CheckpointPayload = { - /** - * The user account id - */ - user_id: string; - /** - * The client id relating to the user account. - * A single user can have multiple clients. - * A client is analogous to a device session. - * Checkpoints are tracked separately for each `user_id` + `client_id`. - */ - client_id: string; - /** - * A strictly increasing Write Checkpoint identifier. - * This number is generated by the application backend. - */ - checkpoint: bigint; -} -``` + Use aliases if your column names differ, for example `SELECT owner AS user_id, device AS client_id, request_id AS checkpoint FROM checkpoints`. + + + Add an endpoint that receives the client ID and checkpoint request ID from the client. Take the user ID from your session or token. The endpoint must: -### Replication Requirements + 1. Store the greater of the submitted ID and the stored ID for that user and client. + 2. Return that value. If the submitted ID was stale, the client uses the returned ID to continue counting from there. -Replication events for the Custom Write Checkpoint table (`checkpoints` in this example) need to enabled. + Write the record through the same pipeline as the uploads, so that it reaches the source database after the changes it confirms. If your backend writes the checkpoint record directly while the uploads are still queued, the client sees the checkpoint before its changes and reverts them. -For Postgres, this involves adding the table to the [PowerSync logical replication publication](/configuration/source-db/setup), for example: + For Postgres, one statement handles both new and existing rows: -```SQL -create publication powersync for table public.lists, public.todos, public.checkpoints; -``` + ```sql + INSERT INTO checkpoints (user_id, client_id, checkpoint) + VALUES ($1, $2, $3) + ON CONFLICT (user_id, client_id) DO UPDATE + SET checkpoint = GREATEST(checkpoints.checkpoint, EXCLUDED.checkpoint) + RETURNING checkpoint; + ``` + + Return the value as a string in JSON to avoid precision loss in JavaScript clients. See [Checkpoint Request IDs](/client-sdks/advanced/checkpoint-requests#checkpoint-request-ids) for the full reconciliation rules. + + + Connect with checkpoint requests enabled and add `postCheckpointRequest()` to your backend connector to call your endpoint. The SDK calls this method after each upload, so `uploadData()` needs no checkpoint handling of its own. See [Prerequisites](/client-sdks/advanced/checkpoint-requests#prerequisites) and [Connector Changes](/client-sdks/advanced/checkpoint-requests#connector-changes) for how to declare the method in each SDK. + + + +### Record Retention + +The Service keeps a replicated checkpoint request for `checkpoint_request_retention_minutes` after it stores the record. The default is 60 minutes. The next compact job then removes it. Clients send their current request ID again when they reconnect, so an expired record is recreated when a client still needs it. On self-hosted instances, you can change the period with [`api.parameters.checkpoint_request_retention_minutes`](/configuration/powersync-service/self-hosted-instances#param-checkpoint-request-retention-minutes). + +Retention applies only to the Service's copy. Rows in your checkpoints table are yours to keep or delete. While a row exists, return its value from your endpoint so that a reconnecting client can resume from it. + +## Legacy Custom Write Checkpoints + +In the legacy flow, your backend generates an increasing checkpoint number for each client, and the client passes that number to `transaction.complete()` after each upload. The Service retains these records because legacy clients wait for a specific number and do not request it again. + + + + Use the same table and replication setup as for [custom checkpoint requests](#setting-up-custom-checkpoint-requests). + + + Add a `write_checkpoints` event definition to your Sync Config: + + ```yaml + config: + edition: 3 + + event_definitions: + write_checkpoints: + payloads: + - SELECT user_id, client_id, checkpoint FROM checkpoints + + streams: + todos: + query: SELECT * FROM todos WHERE owner_id = auth.user_id() + ``` + + + Add an endpoint that increments and returns the checkpoint number for the user and client. Write the record through the same pipeline as the uploads. For Postgres: + + ```sql + INSERT INTO checkpoints (user_id, client_id, checkpoint) + VALUES ($1, $2, 1) + ON CONFLICT (user_id, client_id) DO UPDATE + SET checkpoint = checkpoints.checkpoint + 1 + RETURNING checkpoint; + ``` + + + In `uploadData()`, request a checkpoint from your backend after uploading the transaction and pass it to `complete()`: + + ```typescript + async function uploadData(database: CommonPowerSyncDatabase): Promise { + const transaction = await database.getNextCrudTransaction(); + if (!transaction) { + return; + } + + for (const operation of transaction.crud) { + // Upload the operation to your backend + } + + const clientId = await database.getClientId(); + const checkpoint = await requestWriteCheckpoint(clientId); + await transaction.complete(checkpoint); + } + + async function requestWriteCheckpoint(clientId: string): Promise { + // Call your backend endpoint. It creates the checkpoint record + // and returns the new checkpoint number as a string. + } + ``` + + + +## Migrating to Custom Checkpoint Requests + +While you roll out an updated app version, older versions that still use the legacy flow write legacy checkpoint numbers while updated versions write checkpoint request IDs. Because a Sync Config cannot define both `write_checkpoints` and `checkpoint_requests`, support both kinds of records by defining only `checkpoint_requests` and adding an `is_legacy` field to the payload. The field is available since Service version 1.26.0. -### Sync Rules Requirements +- Set `is_legacy` to `true` for legacy records. The Service retains them. +- Omit `is_legacy`, or set it to `false`, for checkpoint request records. The Service can expire them. -For clients using the [Checkpoint Requests](/client-sdks/advanced/checkpoint-requests) API, enable the `checkpoint_requests` event in your sync configuration. This event maps rows from the `checkpoints` table to the `CheckpointPayload` payload. +Keep both record types in separate tables where possible. Separate tables make the difference in retention visible in the Sync Config: ```yaml event_definitions: - # Note this event is only supported for customers on [Team and Enterprise](https://www.powersync.com/pricing) plans. checkpoint_requests: payloads: - # This defines where the replicated custom Write Checkpoints should be extracted from - - SELECT user_id, checkpoint, client_id FROM checkpoints + # Legacy checkpoints must be retained + - SELECT user_id, client_id, checkpoint, true AS is_legacy FROM legacy_checkpoints + # Checkpoint requests can expire + - SELECT user_id, client_id, checkpoint FROM checkpoint_requests ``` -Use the `write_checkpoints` event only for clients using the legacy Custom Write Checkpoints flow: +If both flows write to one table, distinguish the rows with a column. For example, your checkpoint request endpoint can set a `checkpoint_requested_at` timestamp that the legacy endpoint leaves `NULL`: ```yaml -# sync-rules.yaml - -# Register the custom write_checkpoints event event_definitions: - write_checkpoints: + checkpoint_requests: payloads: - # This defines where the replicated Custom Write Checkpoints should be extracted from - - SELECT user_id, checkpoint, client_id FROM checkpoints - -# Define Sync Rules as usual -bucket_definitions: - global: - data: - ... -``` - -### Application - -Your application should handle Custom Write Checkpoints on both the frontend and backend. - -#### Frontend - -Your client backend connector should make a call to the application backend to create a Custom Write Checkpoint record after uploading items in the `uploadData` method. The Write Checkpoint number should be supplied to the CRUD transactions' `complete` method. - -```TypeScript - async function uploadData(database: CommonPowerSyncDatabase): Promise { - const transaction = await database.getNextCrudTransaction(); - // Get the unique client ID from the PowerSync Database SQLite storage - const clientId = await db.getClientId(); - - for (const operation of transaction.crud) { - // Upload the items to application backend - // .... - } - - await transaction.complete(await getCheckpoint(clientId)); - } - - async function getCheckpoint(clientId: string): string { - /** - * Should perform a request to the application backend which should create the - * Write Checkpoint record and return the corresponding checkpoint number. - */ - return "the Write Checkpoint number from the request"; - } + - SELECT user_id, client_id, checkpoint, checkpoint_requested_at IS NULL AS is_legacy FROM checkpoints ``` -#### Backend +Once no clients use the legacy flow, remove the legacy payload or the `is_legacy` field. With [storage version 4](/sync/advanced/storage-version-4#incremental-reprocessing), changing an event definition reprocesses only that event's data. -The backend should create a Write Checkpoint record when the client requests it. The record should automatically increment the Write Checkpoint number for the associated `user_id` and `client_id`. - -#### Postgres Example - -With the following table defined in the database... - -```SQL -CREATE TABLE checkpoints ( - user_id VARCHAR(255), - client_id VARCHAR(255), - checkpoint INTEGER, - PRIMARY KEY (user_id, client_id) -); -``` - -...the backend should have a route which creates `checkpoints` records: - -```TypeScript -router.put('/checkpoint', async (req, res) => { - if (!req.body) { - res.status(400).send({ - message: 'Invalid body provided' - }); - return; - } - - const client = await pool.connect(); - -// These could be obtained from the session - const { user_id = 'UserID', client_id = '1' } = req.body; - - const response = await client.query( - ` - INSERT - INTO - checkpoints - (user_id, client_id, checkpoint) - VALUES - ($1, $2, '1') - ON - CONFLICT (user_id, client_id) - DO - UPDATE - SET checkpoint = checkpoints.checkpoint + 1 - RETURNING checkpoint; - `, - [user_id, client_id] - ); - client.release(); - - // Return the Write Checkpoint number - res.status(200).send({ - checkpoint: response.rows[0].checkpoint - }); -}); - -``` +## Example Implementations -An example implementation can be seen in the [Node.js backend demo](https://github.com/powersync-ja/powersync-nodejs-backend-todolist-demo/blob/main/src/api/data.js), including examples for [MongoDB](https://github.com/powersync-ja/powersync-nodejs-backend-todolist-demo/blob/main/src/persistance/mongo/mongo-persistance.js) and [MySQL](https://github.com/powersync-ja/powersync-nodejs-backend-todolist-demo/blob/main/src/persistance/mysql/mysql-persistance.js). +- [Swift custom checkpoint demo](https://github.com/powersync-ja/powersync-swift/tree/main/Demos/CustomCheckpointDemo): a client that uses custom checkpoint requests with the Node.js backend demo. +- [Node.js backend demo](https://github.com/powersync-ja/powersync-nodejs-backend-todolist-demo): implements both a checkpoint request endpoint and a legacy checkpoint endpoint, with Postgres, MongoDB, and MySQL persistence. +- [Self-hosted custom checkpoints demo](https://github.com/powersync-ja/self-host-demo/tree/main/demos/nodejs-custom-checkpoints): a Docker Compose setup with Postgres that uses the legacy flow. diff --git a/sync/advanced/compatibility.mdx b/sync/advanced/compatibility.mdx index 98517bca..444e6e9a 100644 --- a/sync/advanced/compatibility.mdx +++ b/sync/advanced/compatibility.mdx @@ -86,6 +86,8 @@ Set `storage_version` when you need to: ### Available Versions +All PowerSync Cloud instances use MongoDB bucket storage, so they are compatible with all available storage versions. Self-hosted instances with Postgres bucket storage can use versions 1 and 2 only. + | Version | Bucket storage | Status | | --- | --- | --- | | `1` | MongoDB or Postgres | Legacy format, retained for existing deployments. | diff --git a/sync/advanced/storage-version-4.mdx b/sync/advanced/storage-version-4.mdx index 0521834d..56d5922b 100644 --- a/sync/advanced/storage-version-4.mdx +++ b/sync/advanced/storage-version-4.mdx @@ -13,7 +13,9 @@ Compared to version 2, it provides: ## Availability -The PowerSync Cloud and self-hosted columns below apply during the Beta only. Once storage version 4 is generally available, it will become the default for all instances. S3 object storage is then also enabled on all PowerSync Cloud instances. For self-hosted deployments, follow the [S3 setup instructions](#self-hosted-s3-setup). +Storage version 4 is compatible with all PowerSync Cloud instances, which already use MongoDB [bucket storage](/architecture/powersync-service#bucket-storage). Self-hosted instances must also use MongoDB bucket storage. Postgres bucket storage is not currently supported. + +The PowerSync Cloud and self-hosted columns below apply during the Beta only. Once storage version 4 is generally available, it will become the default for all supported instances. S3 object storage is then also enabled on all PowerSync Cloud instances. For self-hosted deployments, follow the [S3 setup instructions](#self-hosted-s3-setup). | | Source database | Sync Config | PowerSync Cloud (Beta) | Self-hosted (Beta) | | --- | --- | --- | --- | --- | @@ -21,8 +23,6 @@ The PowerSync Cloud and self-hosted columns below apply during the Beta only. On | Incremental reprocessing | MongoDB | Sync Streams | Included with version 4 | Included with version 4 | | S3 object storage | Any | Sync Streams or Sync Rules | Enabled per instance by PowerSync on request | [Set up S3 object storage](#self-hosted-s3-setup) | -Storage version 4 requires MongoDB as the [bucket storage database](/architecture/powersync-service#bucket-storage). PowerSync Cloud always uses MongoDB. Self-hosted deployments with Postgres bucket storage cannot use version 4. - Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/streams/migration). ## Opt In @@ -53,6 +53,10 @@ streams: ### Self-Hosted + + Postgres bucket storage is not supported with version 4. + + Add `storage_version: 4` to the `config` block of each Sync Config as shown above, then deploy or redeploy it to use version 4. To move a Sync Config back to version 2, set `storage_version: 2` and deploy again. This is another full reprocess. @@ -61,7 +65,11 @@ To also enable S3 object storage, follow the [self-hosted S3 setup instructions] ## Incremental Reprocessing -Incremental reprocessing requires a MongoDB source database, Sync Streams, and storage version 4. It is active whenever all three apply. +Incremental reprocessing is active when you use a MongoDB source database, Sync Streams, and storage version 4. + + + Self-hosted instances with Postgres bucket storage are not supported. + Without it, every deployment reads all data selected by the Sync Config from your source database and prepares a complete new copy. Clients then download all their data again, even if only one stream changed. @@ -111,7 +119,7 @@ For self-hosted instances, offloading bucket data to S3 can reduce storage and d Clients connect only to the PowerSync Service and never to the object store, so no client changes are needed. If the object store becomes unreachable, sync is interrupted until it recovers. Clients reconnect and resume automatically. -S3 object storage requires MongoDB bucket storage and storage version 4. It works with Sync Streams and legacy Sync Rules. +S3 object storage requires storage version 4 and works with Sync Streams and legacy Sync Rules. It is compatible with all PowerSync Cloud instances. S3 object storage holds PowerSync's internal sync data. To store files uploaded by your app, use [Attachments](/client-sdks/advanced/attachments). @@ -123,6 +131,10 @@ During the Beta, PowerSync enables S3 object storage per instance. [Contact us]( ### Self-Hosted S3 Setup + + Self-hosted instances with Postgres bucket storage are not supported. + + Create a bucket. Use the same region as the PowerSync Service where possible, to keep latency low and avoid cross-region data transfer charges. Use a dedicated bucket, or a unique `prefix` per PowerSync instance, so that instances never read or delete each other's files. Give the PowerSync Service permission to list the bucket and to read, write, and delete objects under the prefix.