Use proper result set when evaluating parameters by simolus3 · Pull Request #782 · powersync-ja/powersync-service

simolus3 · 2026-09-01T15:30:52Z

When querying buckets for Sync Streams, we generally try to resolve parameters independently to form a cartesian product in the end. This is correct for most streams, but goes wrong when a parameter index has more than one column. For example, in SELECT a.* FROM a, b WHERE a.c1 = b.c1 AND a.c2 = b.c2 AND b.u = auth.user_id(), the two parameters are b.c1 and b.c2. If we encounter multiple rows of b through a lookup result, we can't assume those to be independent parameters though! We can only pair parameters that originate from the same row.

This is currently implemented by tracking provenance for each parameter value back to the lookup this originally came from. When we build the cartesian product in the end, we ignore values with incompatible provenance from different rows. Unfortunately, tracking provenance is both kind of expensive and very tricky to get right.

Semantically, evaluating bucket parameters involves:

Forming a result set by adding lookups through joins or a cartesian product.
Once we have the full result set, bucket parameters are a projection.

This replaces the previous querier logic with an actual result set implementation: We start out with a unit set of one row without columns, then go through added lookups that are cross-joined (table-valued functions) or inner-joined (parameter lookups). If we end up with an empty intermediate result set at any point, we know there won't be any buckets and bail out early.

Intersection parameters require special consideration now, but can be implemented by going through the result set and deleting rows where the columns don't match.

AI use: The approach is manual, most tests and some implementation details are generated with Claude Code.

changeset-bot · 2026-09-01T15:30:58Z

⚠️ No Changeset found

Latest commit: 07a4ef4

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

simolus3 · 2026-09-01T15:36:36Z

+      }

-    for (let i = 0; i < this.lookupStages.length; i++) {
-      // Within a stage, we can resolve lookups concurrently.


With the implementation from this PR, joins are no longer processed concurrently. It's possible to add that back with minor added complexity, but:

this is only relevant for complex sync streams

we already evaluate other users / queriers concurrently, to the point where bucket storage is likely the bottleneck and not JS

So I don't think this is necessarily something worth doing, but I can change this here / in a follow-up PR if needed.

simolus3 · 2026-09-01T15:39:58Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 68a4c2dc8b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af866d1014

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

rkistner

Codex picked up a performance regression here with intersecting results. Example queries:

-- In this case we can intersect results
SELECT * FROM issues
  WHERE id IN auth.parameter('allowed_issue_ids')
    AND id IN subscription.parameter('visible_issue_ids')
    AND id IN subscription.parameter('selected_issue_ids');

-- In this case we can compute the checks independently
SELECT * FROM issues
  WHERE 'issues' IN auth.parameter('tables')
    AND 'read' IN auth.parameter('permissions');

With the old implementation, the intersection was computed first. The new implementation builds up the Cartesian product of the arrays before filtering them. In a cases like the above, the number of results can explode even with modestly-sized arrays, leading to slow performance and/or OOM-crashes.

simolus3 · 2026-09-08T13:23:53Z

That is a good point. Since the columns of intersections are known statically, we can evaluate them for each row added to the result set instead of removing rows after building the cartesian product.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 849a419d50

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

rkistner

I like the approach of using a ResultSet here, but Codex picked up another couple of performance regressions with the implementation (manually confirmed the findings).

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9bf009cf7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1183c12e16

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-09-15T15:46:57Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-15T15:54:00.339037Z	`07a4ef4`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 07a4ef428c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

rkistner

The further optimizations did help, but there are still some further optimizations needed to avoid performance regressions. See the individual comments, analyzed using Codex.

rkistner · 2026-09-24T15:29:48Z

+              this.#checkInstantiable();
+            } else {
+              // For this parameter to be part of the intersection optimization, it must return exactly one column.
+              for (const [value] of new Set(outputs)) {


Outputs is SqliteParameterValue[][], so this creates a Set<SqliteParameterValue[]>. That will compare arrays by identity, rather than value, so de-duplication doesn't work properly.

Since you're only using one array element, it should be sufficient to just change the order:

for (const value of new Set(outputs.map(([value]) => value))) {

A regression test covering this case would also be good - see this suggestion from Codex:

For a IN subscription.parameter('x') AND a IN auth.parameter('y'): - x = ['unauthorized', 'unauthorized'], y = ['allowed'] → no buckets. - x = ['allowed', 'allowed'], y = ['allowed'] → exactly the allowed bucket.

rkistner · 2026-09-24T15:53:09Z

+      for (const [parameter, intersections] of parameterToIntersection.entries()) {
+        if (intersections.length == 1) {
+          this.resultSet.multiply(parameter.resultSetIndex, intersections[0].materializedRows);
+        } else {
+          const rows = intersections.reduce((acc, intersection) =>
+            intersection.materializedRows.length < acc.materializedRows.length ? intersection : acc
+          ).materializedRows;
+
+          this.resultSet.multiply(
+            parameter.resultSetIndex,
+            rows.filter(([value]) => {
+              return intersections.every((intersection) => {
+                const matchingCount = intersection.rows.get(value);
+                return matchingCount === intersection.amountOfRequestParameters;
+              });
+            })
+          );
+        }
+
+        this.#checkInstantiable();
+      }


If I understand it correctly, the optimization above now removes values that don't intersect. But any remaining values are still multiplied, which can result in exponential computations and memory.

Maybe it can work to have a multiply function that takes into account the intersection, something like this?

resultSet.multiplyCorrelated( parameters.map((p) => p.resultSetIndex), intersection.materializedRows );

rkistner · 2026-09-24T16:05:23Z

+    using add = this.#prepareAddingResultSet(resultSetIndex);
+
+    const originalLength = this.#rows.length;
+    for (let i = 0; i < originalLength; i++) {
+      const lookup = lookupsByRow[i];
+      if (lookup.foundRows.length === 0 || !this.#multiplyAtRow(resultSetIndex, add.filter, i, lookup.foundRows)) {
+        // The row has no matching join partner, so remove it. We can't split it immediately because #multiplyAtRow is
+        // still iterating through rows.
+        add.deletedRows.push(i);
+      }
+    }


This still does O(N*M) comparisons, even though the results are O(N+M). An in-memory index (Map) should be able to reduce the comparisons to O(N+M) as well.

Here it appears that the previous implementation did cover intersection on a single key efficiently. A stretch goal would be to cover multiple intersection parameters, such as x and y in this example:

SELECT items.* FROM items JOIN allowed a ON items.x = a.x AND items.y = a.y JOIN enabled e ON items.x = e.x AND items.y = e.y WHERE a.user_id = auth.user_id()

I believe the previous implementation would efficiently intersect on x, then multiply on filter on y. We could make that even more efficient by intersecting on (x, y), but that's a further optimization rather than covering a performance regression.

simolus3 commented Sep 1, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Sep 1, 2026

View reviewed changes

Comment thread packages/sync-rules/src/sync_plan/evaluator/parameter_evaluator.ts Outdated

simolus3 commented Sep 1, 2026

View reviewed changes

Comment thread packages/sync-rules/src/sync_plan/evaluator/result_set.ts

simolus3 mentioned this pull request Sep 1, 2026

Fix provenance for lookups with multiple inputs #780

Merged

simolus3 marked this pull request as ready for review September 2, 2026 08:41

chatgpt-codex-connector Bot reviewed Sep 2, 2026

View reviewed changes

Comment thread packages/sync-rules/src/sync_plan/evaluator/parameter_evaluator.ts Outdated

simolus3 requested a review from rkistner September 2, 2026 09:11

rkistner reviewed Sep 8, 2026

View reviewed changes

simolus3 added 9 commits September 8, 2026 13:47

Represent instantiated parameters as result set

f577834

Fix most evaluator tests

46d9374

Support intersections

0454ab0

Test empty joins

a17c768

Fix for falsy resolved values

a31864a

Freeze rows before adding them

69caf10

AI feedback

b6ff8f3

Evaluate intersection on table-valued functions eagerly

d5500fe

Build intersection row-by-row

849a419

simolus3 force-pushed the update-evaluator branch from dade362 to 849a419 Compare September 8, 2026 13:22

chatgpt-codex-connector Bot reviewed Sep 8, 2026

View reviewed changes

Comment thread packages/sync-rules/src/sync_plan/evaluator/parameter_evaluator.ts Outdated

Comment thread packages/sync-rules/src/sync_plan/evaluator/result_set.ts Outdated

AI feedback

b5cf4eb

rkistner reviewed Sep 10, 2026

View reviewed changes

Comment thread packages/sync-rules/src/sync_plan/evaluator/result_set.ts

Comment thread packages/sync-rules/src/sync_plan/evaluator/result_set.ts Outdated

simolus3 added 3 commits September 10, 2026 18:25

Avoid quadratic result set cleanup

a573651

Make more efficient because we don't care about order

84a752f

Optimizations

c9bf009

chatgpt-codex-connector Bot reviewed Sep 14, 2026

View reviewed changes

Comment thread packages/sync-rules/src/sync_plan/evaluator/parameter_evaluator.ts

Simplify, AI feedback

1183c12

chatgpt-codex-connector Bot reviewed Sep 14, 2026

View reviewed changes

Comment thread packages/sync-rules/src/sync_plan/evaluator/parameter_evaluator.ts Outdated

simolus3 and others added 2 commits September 14, 2026 12:08

Use strict equality

6f398e7

Merge branch 'main' into update-evaluator

07a4ef4

chatgpt-codex-connector Bot reviewed Sep 15, 2026

View reviewed changes

Comment thread packages/sync-rules/src/sync_plan/evaluator/result_set.ts

Comment thread packages/sync-rules/src/sync_plan/evaluator/parameter_evaluator.ts

simolus3 added this pull request to stack #817 September 23, 2026 09:31

simolus3 mentioned this pull request Sep 23, 2026

Fix parameter equality for numbers #816

Open

simolus3 requested a review from rkistner September 23, 2026 09:32

rkistner reviewed Sep 24, 2026

View reviewed changes

simolus3 marked this pull request as draft September 28, 2026 11:42

Conversation

simolus3 commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

simolus3 Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

simolus3 commented Sep 1, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

rkistner left a comment

Choose a reason for hiding this comment

Uh oh!

simolus3 commented Sep 8, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

rkistner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

rkistner left a comment

Choose a reason for hiding this comment

Uh oh!

rkistner Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

rkistner Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

rkistner Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

simolus3 commented Sep 1, 2026 •

edited

Loading

changeset-bot Bot commented Sep 1, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 15, 2026 •

edited

Loading