feat: collection-scoped activity bucketing

[gaspergrom]

Summary

• Adds collection-scoped activity bucketing (10 hash-based buckets on collectionSlug, mirroring the existing project-side bucketing) so collection queries no longer scan the full unbucketed activity relations table: 10 activityRelations_collection_bucket_MV_N streaming MVs, 10 raw bucket datasources, 10 daily COPY_MODE clean/enrich pipes, and a activityRelations_collection_bucket_union.pipe (10-way UNION ALL) query layer, since collectionSlug isn't known at Jinja-template time and a project can belong to multiple collections.
• Points activities_filtered.pipe, activities_filtered_retention.pipe, and activityTypes_by_project.pipe at the new collection bucket union when collectionSlug is scoped.
• Re-engineers contributors_leaderboard.pipe and contributor_dependency.pipe to support collectionSlug scope directly, using a scalar-total pattern (a separate cheap unfiltered aggregate cross-joined into the grouped/limited result) instead of a window function over the full grouped result — this reaches the same performance as the dedicated collection-specific pipes without needing them.
• Removes the now-redundant collection_contributors_leaderboard.pipe and collection_contributor_dependency.pipe (+ its copy pipe/datasource), superseded by the above.

Deploy status

Already deployed to Tinybird production and validated:

• Bucket data confirmed correct (bucket 0: 111K MV rows / 55K cleaned rows; bucket 6: 75K MV rows / 42K cleaned rows across 53 collections)
• Sample collection query (orm-persistence-frameworks) returned correct counts: 170 activities / 49 unique members / 39 unique projects
• Project-side bucketing confirmed unaffected
• The removed pipes were confirmed to have zero remaining traffic before deletion (gated on the insights consumer PR below being live in production first)

Note: staging was skipped for the initial bucket infra deploy — it was at its 100-datasource capacity limit — and went straight to production after validation looked solid.

Cross-repo dependency

insights PR #2045 updates the Insights app to call contributors_leaderboard/contributor_dependency directly with collectionSlug instead of the old collection-specific pipes. That PR has already been merged and deployed to production, which is what unblocked deleting the old pipes in this PR.

Test plan

• Production Tinybird validation of bucket MV/cleaned row counts and a sample collection query (see above)
• Confirmed project-side (non-collection) bucketing unaffected
• Confirmed zero live traffic against the deleted pipes before removal
• Spot-check a few more collections' Contributors tab widgets post-merge

[cursor]

PR Summary

High Risk
Large production Tinybird pipeline change (new MVs, daily jobs, deleted serving pipes) affecting collection insights correctness and latency; mitigated by prod validation and gated removal of old pipes after the insights consumer shipped.

Overview
Introduces collection-scoped activity bucketing (10 hash buckets on collectionSlug, parallel to project bucketing): streaming MVs fan curated-collection activities into per-bucket tables, daily clean/enrich COPY jobs, on-demand snapshot backfills, and activityRelations_collection_bucket_union as the query layer when collectionSlug is not known at Jinja time.

Collection queries in activities_filtered, activities_filtered_retention, and activityTypes_by_project now read the new collection bucket union instead of the old project-wide cleaned union.

Contributor and aggregate APIs are consolidated: contributors_leaderboard and contributor_dependency accept collectionSlug and use a scalar total (not a window over all members) for percentages; the dedicated collection_contributors_leaderboard, collection_contributor_dependency, and bus-factor copy pipe/datasource are removed. collection_insights_aggregate becomes a join of daily precomputed contributor and project snapshot tables (per-bucket contributor COPY pipes + a projects COPY pipe), replacing live scans over the full activity union.

Adds collection_buckets.pipe for slug→bucket resolution and insights-app-token on key serving pipes.

^{Reviewed by Cursor Bugbot for commit c6d883e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

• feat: add user authentication (CM-123)
• feat: add user authentication (IN-123)

Projects:

• CM: Community Data Platform
• IN: Insights

Please add a Jira issue key to your PR title.

[Copilot]

Pull request overview

Adds collection-scoped Tinybird activity bucketing to improve query performance.

Changes:

• Adds 10 collection bucket ingestion, cleanup, and query pipelines.
• Routes collection activity queries through the new buckets.
• Unifies contributor pipes and removes collection-specific equivalents.

Review blockers include community-collection data loss, stale membership fan-out, dependency-query scalability, and overlapping copy schedules.

Reviewed changes

Copilot reviewed 61 out of 61 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
contributors_leaderboard.pipe	Adds collection scope and scalar totals.
contributor_dependency.pipe	Adds collection scope.
collection_contributors_leaderboard.pipe	Removes legacy collection leaderboard.
collection_contributor_dependency.pipe	Removes collection dependency endpoint.
collection_contributor_dependency_copy.pipe	Removes dependency precomputation.
collection_buckets.pipe	Adds collection hash resolution.
activityTypes_by_project.pipe	Routes collections through new union.
activityRelations_collection_bucket_union.pipe	Unions ten collection buckets.
activityRelations_collection_bucket_MV_snapshot_0.pipe	Adds bucket 0 backfill.
activityRelations_collection_bucket_MV_snapshot_1.pipe	Adds bucket 1 backfill.
activityRelations_collection_bucket_MV_snapshot_2.pipe	Adds bucket 2 backfill.
activityRelations_collection_bucket_MV_snapshot_3.pipe	Adds bucket 3 backfill.
activityRelations_collection_bucket_MV_snapshot_4.pipe	Adds bucket 4 backfill.
activityRelations_collection_bucket_MV_snapshot_5.pipe	Adds bucket 5 backfill.
activityRelations_collection_bucket_MV_snapshot_6.pipe	Adds bucket 6 backfill.
activityRelations_collection_bucket_MV_snapshot_7.pipe	Adds bucket 7 backfill.
activityRelations_collection_bucket_MV_snapshot_8.pipe	Adds bucket 8 backfill.
activityRelations_collection_bucket_MV_snapshot_9.pipe	Adds bucket 9 backfill.
activityRelations_collection_bucket_MV_0.pipe	Adds bucket 0 fan-out MV.
activityRelations_collection_bucket_MV_1.pipe	Adds bucket 1 fan-out MV.
activityRelations_collection_bucket_MV_2.pipe	Adds bucket 2 fan-out MV.
activityRelations_collection_bucket_MV_3.pipe	Adds bucket 3 fan-out MV.
activityRelations_collection_bucket_MV_4.pipe	Adds bucket 4 fan-out MV.
activityRelations_collection_bucket_MV_5.pipe	Adds bucket 5 fan-out MV.
activityRelations_collection_bucket_MV_6.pipe	Adds bucket 6 fan-out MV.
activityRelations_collection_bucket_MV_7.pipe	Adds bucket 7 fan-out MV.
activityRelations_collection_bucket_MV_8.pipe	Adds bucket 8 fan-out MV.
activityRelations_collection_bucket_MV_9.pipe	Adds bucket 9 fan-out MV.
activityRelations_collection_bucket_clean_enrich_copy_pipe_0.pipe	Adds bucket 0 cleanup.
activityRelations_collection_bucket_clean_enrich_copy_pipe_1.pipe	Adds bucket 1 cleanup.
activityRelations_collection_bucket_clean_enrich_copy_pipe_2.pipe	Adds bucket 2 cleanup.
activityRelations_collection_bucket_clean_enrich_copy_pipe_3.pipe	Adds bucket 3 cleanup.
activityRelations_collection_bucket_clean_enrich_copy_pipe_4.pipe	Adds bucket 4 cleanup.
activityRelations_collection_bucket_clean_enrich_copy_pipe_5.pipe	Adds bucket 5 cleanup.
activityRelations_collection_bucket_clean_enrich_copy_pipe_6.pipe	Adds bucket 6 cleanup.
activityRelations_collection_bucket_clean_enrich_copy_pipe_7.pipe	Adds bucket 7 cleanup.
activityRelations_collection_bucket_clean_enrich_copy_pipe_8.pipe	Adds bucket 8 cleanup.
activityRelations_collection_bucket_clean_enrich_copy_pipe_9.pipe	Adds bucket 9 cleanup.
activities_filtered.pipe	Routes collection activity filtering.
activities_filtered_retention.pipe	Routes collection retention queries.
collection_contributor_dependency_copy_ds.datasource	Removes dependency cache schema.
activityRelations_collection_deduplicated_cleaned_bucket_0_ds.datasource	Defines cleaned bucket 0.
activityRelations_collection_deduplicated_cleaned_bucket_1_ds.datasource	Defines cleaned bucket 1.
activityRelations_collection_deduplicated_cleaned_bucket_2_ds.datasource	Defines cleaned bucket 2.
activityRelations_collection_deduplicated_cleaned_bucket_3_ds.datasource	Defines cleaned bucket 3.
activityRelations_collection_deduplicated_cleaned_bucket_4_ds.datasource	Defines cleaned bucket 4.
activityRelations_collection_deduplicated_cleaned_bucket_5_ds.datasource	Defines cleaned bucket 5.
activityRelations_collection_deduplicated_cleaned_bucket_6_ds.datasource	Defines cleaned bucket 6.
activityRelations_collection_deduplicated_cleaned_bucket_7_ds.datasource	Defines cleaned bucket 7.
activityRelations_collection_deduplicated_cleaned_bucket_8_ds.datasource	Defines cleaned bucket 8.
activityRelations_collection_deduplicated_cleaned_bucket_9_ds.datasource	Defines cleaned bucket 9.
activityRelations_collection_bucket_MV_ds_0.datasource	Defines raw bucket 0.
activityRelations_collection_bucket_MV_ds_1.datasource	Defines raw bucket 1.
activityRelations_collection_bucket_MV_ds_2.datasource	Defines raw bucket 2.
activityRelations_collection_bucket_MV_ds_3.datasource	Defines raw bucket 3.
activityRelations_collection_bucket_MV_ds_4.datasource	Defines raw bucket 4.
activityRelations_collection_bucket_MV_ds_5.datasource	Defines raw bucket 5.
activityRelations_collection_bucket_MV_ds_6.datasource	Defines raw bucket 6.
activityRelations_collection_bucket_MV_ds_7.datasource	Defines raw bucket 7.
activityRelations_collection_bucket_MV_ds_8.datasource	Defines raw bucket 8.
activityRelations_collection_bucket_MV_ds_9.datasource	Defines raw bucket 9.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 61 out of 61 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (4)

services/libs/tinybird/pipes/activities_filtered.pipe:31

• This route drops all community-collection activity. segments_filtered_by_collection intentionally resolves both curated and community slugs (see insights_projects_populated_copy.pipe:1-16), but every new collection bucket MV filters its fan-out to ssoUserId IS NULL. A request for an ssoUserId IS NOT NULL collection therefore has valid segments yet reads an empty source. Keep the previous all-bucket source as a fallback for community collections, or include those collections in the new bucket pipeline.

        {% if defined(collectionSlug) %} activityRelations_collection_bucket_union

services/libs/tinybird/pipes/activities_filtered.pipe:31

• Pre-fanning membership only when an activityRelations row is inserted makes historical results stale when an existing project is later added to a collection. insights_projects_populated_ds refreshes memberships every 15 minutes, but these streaming MVs are not triggered by that refresh and the snapshots are on-demand only, so old activities are never copied into the new collection's bucket. Before switching the shared filter to this source, add a recurring membership-aware rebuild/backfill or another path that preserves current-membership semantics.

        {% if defined(collectionSlug) %} activityRelations_collection_bucket_union

services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_0.pipe:57

• Bucket 0 and bucket 1 are both scheduled at 01:00, and activityRelations_snapshot_merger_copy also runs then. This contradicts the stated five-minute staggering and launches multiple full-table jobs together. Give bucket 0 a distinct five-minute slot so only one collection full-replace runs at a time.
  services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_0.pipe:8
• This references a collection-bucketing addendum that does not exist in bucketing-architecture.md; that document currently describes only project-side buckets. Add the promised architecture and operational/backfill guidance, especially because collection membership changes require a different consistency model from deterministic project bucketing.

    up to N rows across up to N different buckets - this is intentional write amplification, see
    bucketing-architecture.md's collection-bucketing addendum.

[Copilot]

Pull request overview

Copilot reviewed 75 out of 75 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (4)

services/libs/tinybird/pipes/activities_filtered.pipe:31

• This routes every collectionSlug request to a datasource populated only for curated collections (ssoUserId IS NULL in each new bucket MV). However, insights_projects_populated_copy.pipe:3-15 intentionally includes both curated and community collections after IN-1195, and segments_filtered_by_collection supports both. Community collection requests will therefore return no activities here (and in the two direct union consumers). Materialize community collections too, or retain a fallback to the project-bucket union for them.

        {% if defined(collectionSlug) %} activityRelations_collection_bucket_union

services/libs/tinybird/pipes/contributor_dependency.pipe:9

• Collection dependency requests now always execute the live per-member aggregation. The deleted collection-specific endpoint documents that this live GROUP BY memberId timed out and triggered Tinybird concurrency alerts for large collections; moving the percentage denominator to a scalar removes the window overhead but not that expensive grouping. Preserve an equivalent precomputed default path before routing production collection requests through this pipe.
  services/libs/tinybird/pipes/collection_insights_aggregate.pipe:37
• A collection with no matching contribution rows now returns NULL because this is a left join, whereas the previous uniq(...) aggregate returned 0. Preserve the endpoint's numeric response contract so consumers do not receive a nullable count.
  services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_0.pipe:57
• Bucket 0 and bucket 1 are both scheduled at 0 1 * * *, so two full-bucket replacement jobs start simultaneously despite the stated five-minute staggering intended to avoid load spikes. Assign distinct slots across all ten jobs.

[Copilot]

Pull request overview

Copilot reviewed 75 out of 75 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (5)

services/libs/tinybird/pipes/contributor_dependency.pipe:9

• Collection requests to this pipe now always execute the live leaderboard aggregation. The deleted collection_contributor_dependency.pipe documents that this same collection-wide GROUP BY af.memberId path caused production timeouts/rate-limit alerts and therefore used collection_contributor_dependency_copy_ds for the default unfiltered limit=100 request. The scalar-total change removes the percentage window, but it does not remove that full member aggregation, so routing the Insights client to this unified pipe reintroduces the known production failure mode. Preserve an equivalent precomputed path (or otherwise demonstrate that the default collection query no longer performs the full live aggregation) before deleting the copy pipeline.
  services/libs/tinybird/pipes/activities_filtered.pipe:31
• This unconditionally routes every collectionSlug through buckets populated only for curated collections: all new fan-out MVs filter collections with isNull(ssoUserId). segments_filtered_by_collection and insights_projects_populated_ds explicitly support community collections too, so those requests now return no activities. Either include community collections in the bucket population or retain a correct fallback for them.

        {% if defined(collectionSlug) %} activityRelations_collection_bucket_union

services/libs/tinybird/pipes/activities_filtered_retention.pipe:27

• Community collections are supported by segments_filtered_by_collection, but the new collection bucket MVs only materialize slugs where ssoUserId IS NULL. Consequently retention requests for community collections are routed to empty tables. Populate all supported collection types or keep a non-curated fallback here.

        {% if defined(collectionSlug) %} activityRelations_collection_bucket_union

services/libs/tinybird/pipes/activityTypes_by_project.pipe:25

• This breaks activity-type discovery for community collections. The collection bucket MVs explicitly exclude collections with a non-null ssoUserId, while the existing collection-scoping pipe accepts those slugs; this branch therefore returns an empty activity-type list for a previously supported collection type. Include community collections in the buckets or use the existing all-activity union for that scope.
  services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_0.pipe:18
• This fan-out captures collection membership only when an activityRelations insert triggers the MV. insights_projects_populated_ds refreshes membership every 15 minutes, but adding or removing a project from a collection does not replay historical activities through these MVs. Added projects therefore lack all historical activity, and the new precomputed collection aggregate can continue counting stale rows after removals. Rebuild the raw collection buckets when membership changes, or resolve membership in a scheduled/query-time layer that is refreshed independently of activity inserts.

    SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug
    FROM activityRelations a
    LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId

[Copilot]

Pull request overview

Copilot reviewed 75 out of 75 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (5)

services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_0.pipe:18

• The materialized view denormalizes collectionsSlugs only when an activity row is inserted. insights_projects_populated_ds is replaced every 15 minutes, but changes to collection membership do not trigger this MV: adding a project omits its historical activity, and removing it leaves old bucket rows indefinitely. Add a scheduled reconciliation/rebuild (or derive collection membership at read time) so membership changes update existing activity rows.

    SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug
    FROM activityRelations a
    LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId

services/libs/tinybird/pipes/collection_insights_aggregate.pipe:40

• Filtering on the global maximum snapshot exposes a partial daily snapshot. Bucket 0 advances max(snapshotId) at 02:05, while buckets 1-9 are appended through 02:50; during that window—and for an entire day if one job fails—collections in unfinished buckets have no matching contributor row. Select the latest snapshot independently per collection so yesterday's value remains available until that collection's bucket succeeds.
  services/libs/tinybird/pipes/contributors_leaderboard.pipe:50
• The scalar denominator removes the window function, but this node still groups and sorts every distinct collection member before applying LIMIT. Because this PR deletes the collection dependency pipe's precomputed path, default collection bus-factor requests now execute this live aggregation—the removed pipe documents that this exact work caused production timeouts and rate-limit/concurrency alerts for large collections. Preserve an equivalent precomputed route before removing that materialization.
  services/libs/tinybird/pipes/collection_insights_aggregate.pipe:33
• A collection with no qualifying activity has no row in the grouped contributor copy, so this left join returns NULL. The previous live aggregate returned 0 from uniq(...), making this a response-contract regression for empty collections. Coalesce the missing count to zero.
  services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_0.pipe:57
• Bucket 0 and bucket 1 are both scheduled at 01:00, so two full-bucket COPY_MODE replace jobs run concurrently despite the stated five-minute staggering; bucket 2 then starts at 01:05. Move one job (and reconcile the documented sequence) so every heavy replacement is actually staggered.

[Copilot]

Pull request overview

Copilot reviewed 75 out of 75 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (3)

services/libs/tinybird/pipes/collection_insights_aggregate.pipe:38

• This contributor source contains only curated collections: every upstream collection bucket MV filters to ssoUserId IS NULL. However, insights_projects_populated_ds and the new projects aggregate intentionally include both curated and community collections (insights_projects_populated_copy.pipe:1-16; projects copy lines 25-35). Thus this endpoint returns no contributor row for every community collection, regressing the IN-1195 behavior the previous implementation supported. Add a community-collection contributor path or include those collections in the materialization.
  services/libs/tinybird/pipes/activityTypes_by_project.pipe:33
• The community fallback bypasses the mandatory latest-snapshot filter for activityRelations_deduplicated_cleaned_ds (services/libs/tinybird/lambda-architecture.md:470-489). Although DISTINCT hides duplicate rows, types that exist only in an older snapshot can still be returned. Filter this source to max(snapshotId).
  services/libs/tinybird/pipes/collection_insights_aggregate.pipe:43
• Selecting rows whose timestamp equals the daily maximum does not enforce the promised one-row-per-collection contract. All copy pipes use append mode and truncate snapshotId to the day, so any same-day retry appends another row with the same key; this subquery then returns every duplicate and the endpoint emits duplicate result rows. Make each run’s version unique and select its value with argMax, or otherwise deduplicate retries before joining.

[cursor]

Snapshot retries can duplicate aggregate rows

Medium Severity

Contributor copy pipes append daily rows with COPY_MODE append and a day-granularity snapshotId, but the serving query only filters to max(snapshotId) and does not collapse duplicates. A same-day retry of a bucket copy creates multiple identical (collectionSlug, snapshotId) rows, so the join can return multiple result rows for one collection.

Additional Locations (1)

• services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_0.pipe#L31-L34

 

^{Reviewed by Cursor Bugbot for commit 3891018. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 75 out of 75 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

services/libs/tinybird/pipes/activityRelations_collection_bucket_union.pipe:23

• Routing collection queries to these tables reduces freshness from hourly to daily. The previous project-bucket clean tables are rebuilt hourly (activityRelations_bucket_clean_enrich_copy_pipe_*.pipe), and bucketing-architecture.md:35-43 explains that hourly replacement is needed because activities, bot status, and repository enablement are mutable. All collection activity widgets now read tables rebuilt only at 01:00–01:45 UTC, so new activity and bot/repository changes can remain stale for nearly 24 hours. Preserve equivalent freshness (or add a hot delta layer) unless this product-level latency regression is explicitly accepted.
  services/libs/tinybird/pipes/collection_insights_aggregate.pipe:44
• Same-day retries are still not deduplicated here. Every copy uses COPY_MODE append with toStartOfInterval(now(), INTERVAL 1 day), and the target is a plain MergeTree, so rerunning a bucket appends another row with the same (collectionSlug, snapshotId). This tuple filter returns every such row, causing duplicate endpoint rows (and potentially different counts with no deterministic winner). Use a distinct run/version timestamp and select one latest row per collection, or make the target a versioned replacing table.

[cursor]

Stale collection membership overcount

High Severity

The per-bucket contributor copy pipes count uniq(memberId) from the collection activity buckets with no check against current collectionsSlugs membership. After a project is removed from a collection, its fan-out rows remain in the MV and cleaned tables, so uniqueContributorCount stays inflated. The previous live query restricted to current member segments, and live widgets still do via segments_filtered_by_collection, so the aggregate widget drifts from them.

Additional Locations (1)

• services/libs/tinybird/pipes/collection_insights_aggregate.pipe#L30-L49

 

^{Reviewed by Cursor Bugbot for commit cd52952. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 75 out of 75 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

services/libs/tinybird/pipes/collection_insights_aggregate.pipe:44

• The latest-snapshot predicate does not collapse retries. All ten producers use COPY_MODE append and write a day-truncated snapshotId, while this target is a plain MergeTree; rerunning a bucket on the same day therefore leaves multiple rows with the same (collectionSlug, snapshotId), and this subquery returns all of them, duplicating the endpoint row after the join. Use a distinct run/version timestamp plus a replacing/aggregating target, or otherwise materialize exactly one latest row per collection.
  services/libs/tinybird/pipes/collection_insights_aggregate.pipe:33
• A collection with enabled projects but no qualifying contribution/collaboration rows has no entry in the contributor aggregate, so this LEFT JOIN now returns NULL. The previous live uniq(...) aggregate returned 0, and the documented response is a contributor count. Coalesce the missing aggregate to zero.
  services/libs/tinybird/pipes/activityRelations_collection_bucket_union.pipe:23
• Every table read by this union is rebuilt only once daily (01:00–01:45), whereas the project cleaned buckets used by the previous collection path are rebuilt hourly (activityRelations_bucket_clean_enrich_copy_pipe_*.pipe). Routing collection widgets here therefore increases activity-data latency from roughly one hour to as much as 24 hours. Preserve the existing freshness SLA—e.g. refresh these buckets hourly with safe staggering or add an incremental current-data layer—before switching all collection queries to this union.
  services/libs/tinybird/pipes/collection_insights_aggregate.pipe:34
• Using the projects aggregate as the driving table drops empty collections entirely. The previous aggregate query always returned one row (projectCount = 0, uniqueContributorCount = 0, avgHealthScore = NULL) even when the collection had no enabled projects; this query returns no rows because the daily projects copy cannot materialize a group for that slug. Drive the response from the requested slug (or the collections table) and left-join both aggregates so the endpoint contract remains one row per valid collection.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit c1ee75d. Configure here.}

[cursor]

Aggregate counts disabled project contributors

Medium Severity

The per-bucket contributor copy pipes aggregate uniqueContributorCount from collection bucket rows with no check that the activity's project is still enabled = 1. The previous live collection_insights_aggregate restricted members via segments from enabled projects only, so disabled projects still present in the bucket can inflate the precomputed count served to the UI.

Additional Locations (2)

• services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_1.pipe#L16-L30
• services/libs/tinybird/pipes/collection_insights_aggregate.pipe#L30-L49

 

^{Reviewed by Cursor Bugbot for commit c1ee75d. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 75 out of 75 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (10)

services/libs/tinybird/pipes/collection_insights_aggregate.pipe:38

• The two precomputed inputs cover different populations: this contributor table is populated only from curated-collection buckets, while collection_insights_aggregate_projects_copy_pipe.pipe:31-33 expands every collection slug, including community collections. A community slug therefore returns a project row with uniqueContributorCount = NULL, regressing the live aggregate that previously resolved its member segments. Align both materializations to the same collection population or retain a contributor fallback for community collections.
  services/libs/tinybird/pipes/collection_insights_aggregate.pipe:44
• This latest-per-collection filter does not collapse duplicate rows with the same daily snapshotId. Every bucket copy uses append mode and toStartOfInterval(now(), INTERVAL 1 day), so retrying a copy on the same day appends another identical key; both rows satisfy this tuple predicate and the endpoint returns duplicate aggregate rows. Use a unique run/version timestamp and select one latest row per collection, or make the target/query deterministically deduplicate retries.
  services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_2.pipe:8
• The documented run time is stale: this bucket is scheduled at 01:10 on line 57, not :05. Update the description so operators do not use the wrong schedule when diagnosing copy jobs.
  services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_3.pipe:8
• The documented run time is stale: this bucket is scheduled at 01:15 on line 57, not :10.
  services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_4.pipe:8
• The documented run time is stale: this bucket is scheduled at 01:20 on line 57, not :15.
  services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_5.pipe:8
• The documented run time is stale: this bucket is scheduled at 01:25 on line 57, not :20.
  services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_6.pipe:8
• The documented run time is stale: this bucket is scheduled at 01:30 on line 57, not :25.
  services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_7.pipe:8
• The documented run time is stale: this bucket is scheduled at 01:35 on line 57, not :30.
  services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_8.pipe:8
• The documented run time is stale: this bucket is scheduled at 01:40 on line 57, not :35.
  services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_9.pipe:8
• The documented run time is stale: this bucket is scheduled at 01:45 on line 57, not :40.

[Copilot]

Pull request overview

Copilot reviewed 75 out of 75 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

services/libs/tinybird/pipes/collection_insights_aggregate.pipe:38

• This still joins different collection populations. collection_insights_aggregate_projects_copy_pipe expands every slug in insights_projects_populated_ds, whose producer explicitly includes curated and community collections (insights_projects_populated_copy.pipe:1-16), while all contributor bucket MVs require ssoUserId IS NULL (for example activityRelations_collection_bucket_MV_0.pipe:10-20). Community collections therefore have a projects row but can never have a contributor row, regressing the aggregate that IN-1195 explicitly restored. The contributor materialization needs to cover the same population, or this endpoint needs a correct community path.
  services/libs/tinybird/pipes/collection_insights_aggregate.pipe:44
• Same-day retries are not collapsed here. Every aggregate job appends with snapshotId = toStartOfInterval(now(), INTERVAL 1 day), and the target is a plain MergeTree, so rerunning a bucket appends another row with the same (collectionSlug, snapshotId). This tuple predicate selects every duplicate latest row and the LEFT JOIN then returns duplicate endpoint rows. Add a distinct run/version column with replacing semantics, or aggregate to exactly one deterministic latest row per collection.
  services/libs/tinybird/pipes/collection_insights_aggregate.pipe:33
• Because this is a LEFT JOIN, a collection with no qualifying contributor aggregate (for example, no code/collaboration activity or an expired/failed snapshot) returns uniqueContributorCount = NULL; the previous live uniq(...) aggregate returned 0, and the response documents this field as a count. Coalesce the missing aggregate to zero to preserve the endpoint contract.