From 499d36cf039178276a674f5a19a0ee21689a6777 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 28 Jul 2026 15:56:21 +0100 Subject: [PATCH 01/10] feat: route collection-scoped activity queries to collection buckets MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mirrors project-side activityRelations bucketing (hash-based, 10 buckets) for collections. Adds 10 streaming MVs, 10 raw MV datasources, 10 daily enrich/dedup copy pipes, 10 cleaned bucket datasources, the collection_buckets resolver, and the activityRelations_collection_bucket_union query pipe. Points activities_filtered/activities_filtered_retention/activityTypes_by_project at the new collection bucket union instead of the full unbucketed scan. Adds collectionSlug support to contributors_leaderboard and contributor_dependency via a scalar-total pattern instead of a window function, avoiding the same collection-scale timeout that motivated collection_contributors_leaderboard. Signed-off-by: Gašper Grom --- ...tions_collection_bucket_MV_ds_0.datasource | 30 ++++++++++ ...tions_collection_bucket_MV_ds_1.datasource | 30 ++++++++++ ...tions_collection_bucket_MV_ds_2.datasource | 30 ++++++++++ ...tions_collection_bucket_MV_ds_3.datasource | 30 ++++++++++ ...tions_collection_bucket_MV_ds_4.datasource | 30 ++++++++++ ...tions_collection_bucket_MV_ds_5.datasource | 30 ++++++++++ ...tions_collection_bucket_MV_ds_6.datasource | 30 ++++++++++ ...tions_collection_bucket_MV_ds_7.datasource | 30 ++++++++++ ...tions_collection_bucket_MV_ds_8.datasource | 30 ++++++++++ ...tions_collection_bucket_MV_ds_9.datasource | 30 ++++++++++ ...eduplicated_cleaned_bucket_0_ds.datasource | 33 +++++++++++ ...eduplicated_cleaned_bucket_1_ds.datasource | 33 +++++++++++ ...eduplicated_cleaned_bucket_2_ds.datasource | 33 +++++++++++ ...eduplicated_cleaned_bucket_3_ds.datasource | 33 +++++++++++ ...eduplicated_cleaned_bucket_4_ds.datasource | 33 +++++++++++ ...eduplicated_cleaned_bucket_5_ds.datasource | 33 +++++++++++ ...eduplicated_cleaned_bucket_6_ds.datasource | 33 +++++++++++ ...eduplicated_cleaned_bucket_7_ds.datasource | 33 +++++++++++ ...eduplicated_cleaned_bucket_8_ds.datasource | 33 +++++++++++ ...eduplicated_cleaned_bucket_9_ds.datasource | 33 +++++++++++ .../tinybird/pipes/activities_filtered.pipe | 5 +- .../pipes/activities_filtered_retention.pipe | 2 +- ...ivityRelations_collection_bucket_MV_0.pipe | 24 ++++++++ ...ivityRelations_collection_bucket_MV_1.pipe | 24 ++++++++ ...ivityRelations_collection_bucket_MV_2.pipe | 24 ++++++++ ...ivityRelations_collection_bucket_MV_3.pipe | 24 ++++++++ ...ivityRelations_collection_bucket_MV_4.pipe | 24 ++++++++ ...ivityRelations_collection_bucket_MV_5.pipe | 24 ++++++++ ...ivityRelations_collection_bucket_MV_6.pipe | 24 ++++++++ ...ivityRelations_collection_bucket_MV_7.pipe | 24 ++++++++ ...ivityRelations_collection_bucket_MV_8.pipe | 24 ++++++++ ...ivityRelations_collection_bucket_MV_9.pipe | 24 ++++++++ ...tions_collection_bucket_MV_snapshot_0.pipe | 25 ++++++++ ...tions_collection_bucket_MV_snapshot_1.pipe | 25 ++++++++ ...tions_collection_bucket_MV_snapshot_2.pipe | 25 ++++++++ ...tions_collection_bucket_MV_snapshot_3.pipe | 25 ++++++++ ...tions_collection_bucket_MV_snapshot_4.pipe | 25 ++++++++ ...tions_collection_bucket_MV_snapshot_5.pipe | 25 ++++++++ ...tions_collection_bucket_MV_snapshot_6.pipe | 25 ++++++++ ...tions_collection_bucket_MV_snapshot_7.pipe | 25 ++++++++ ...tions_collection_bucket_MV_snapshot_8.pipe | 25 ++++++++ ...tions_collection_bucket_MV_snapshot_9.pipe | 25 ++++++++ ...ction_bucket_clean_enrich_copy_pipe_0.pipe | 57 ++++++++++++++++++ ...ction_bucket_clean_enrich_copy_pipe_1.pipe | 57 ++++++++++++++++++ ...ction_bucket_clean_enrich_copy_pipe_2.pipe | 57 ++++++++++++++++++ ...ction_bucket_clean_enrich_copy_pipe_3.pipe | 57 ++++++++++++++++++ ...ction_bucket_clean_enrich_copy_pipe_4.pipe | 57 ++++++++++++++++++ ...ction_bucket_clean_enrich_copy_pipe_5.pipe | 57 ++++++++++++++++++ ...ction_bucket_clean_enrich_copy_pipe_6.pipe | 57 ++++++++++++++++++ ...ction_bucket_clean_enrich_copy_pipe_7.pipe | 57 ++++++++++++++++++ ...ction_bucket_clean_enrich_copy_pipe_8.pipe | 57 ++++++++++++++++++ ...ction_bucket_clean_enrich_copy_pipe_9.pipe | 57 ++++++++++++++++++ ...vityRelations_collection_bucket_union.pipe | 59 +++++++++++++++++++ .../pipes/activityTypes_by_project.pipe | 4 +- .../tinybird/pipes/collection_buckets.pipe | 16 +++++ .../pipes/contributor_dependency.pipe | 10 +++- .../pipes/contributors_leaderboard.pipe | 43 ++++++++++---- 57 files changed, 1811 insertions(+), 18 deletions(-) create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_0.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_1.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_2.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_3.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_4.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_5.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_6.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_7.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_8.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_9.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_0_ds.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_1_ds.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_2_ds.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_3_ds.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_4_ds.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_5_ds.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_6_ds.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_7_ds.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_8_ds.datasource create mode 100644 services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_9_ds.datasource create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_0.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_1.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_2.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_3.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_4.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_5.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_6.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_7.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_8.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_9.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_0.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_1.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_2.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_3.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_4.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_5.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_6.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_7.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_8.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_9.pipe create mode 100644 services/libs/tinybird/pipes/activityRelations_collection_bucket_union.pipe create mode 100644 services/libs/tinybird/pipes/collection_buckets.pipe diff --git a/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_0.datasource b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_0.datasource new file mode 100644 index 0000000000..a7615d526c --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_0.datasource @@ -0,0 +1,30 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String + +ENGINE ReplacingMergeTree +ENGINE_PARTITION_KEY toYear(createdAt) +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, channel, sourceId +ENGINE_VER updatedAt diff --git a/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_1.datasource b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_1.datasource new file mode 100644 index 0000000000..a7615d526c --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_1.datasource @@ -0,0 +1,30 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String + +ENGINE ReplacingMergeTree +ENGINE_PARTITION_KEY toYear(createdAt) +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, channel, sourceId +ENGINE_VER updatedAt diff --git a/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_2.datasource b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_2.datasource new file mode 100644 index 0000000000..a7615d526c --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_2.datasource @@ -0,0 +1,30 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String + +ENGINE ReplacingMergeTree +ENGINE_PARTITION_KEY toYear(createdAt) +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, channel, sourceId +ENGINE_VER updatedAt diff --git a/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_3.datasource b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_3.datasource new file mode 100644 index 0000000000..a7615d526c --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_3.datasource @@ -0,0 +1,30 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String + +ENGINE ReplacingMergeTree +ENGINE_PARTITION_KEY toYear(createdAt) +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, channel, sourceId +ENGINE_VER updatedAt diff --git a/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_4.datasource b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_4.datasource new file mode 100644 index 0000000000..a7615d526c --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_4.datasource @@ -0,0 +1,30 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String + +ENGINE ReplacingMergeTree +ENGINE_PARTITION_KEY toYear(createdAt) +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, channel, sourceId +ENGINE_VER updatedAt diff --git a/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_5.datasource b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_5.datasource new file mode 100644 index 0000000000..a7615d526c --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_5.datasource @@ -0,0 +1,30 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String + +ENGINE ReplacingMergeTree +ENGINE_PARTITION_KEY toYear(createdAt) +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, channel, sourceId +ENGINE_VER updatedAt diff --git a/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_6.datasource b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_6.datasource new file mode 100644 index 0000000000..a7615d526c --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_6.datasource @@ -0,0 +1,30 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String + +ENGINE ReplacingMergeTree +ENGINE_PARTITION_KEY toYear(createdAt) +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, channel, sourceId +ENGINE_VER updatedAt diff --git a/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_7.datasource b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_7.datasource new file mode 100644 index 0000000000..a7615d526c --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_7.datasource @@ -0,0 +1,30 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String + +ENGINE ReplacingMergeTree +ENGINE_PARTITION_KEY toYear(createdAt) +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, channel, sourceId +ENGINE_VER updatedAt diff --git a/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_8.datasource b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_8.datasource new file mode 100644 index 0000000000..a7615d526c --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_8.datasource @@ -0,0 +1,30 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String + +ENGINE ReplacingMergeTree +ENGINE_PARTITION_KEY toYear(createdAt) +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, channel, sourceId +ENGINE_VER updatedAt diff --git a/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_9.datasource b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_9.datasource new file mode 100644 index 0000000000..a7615d526c --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_bucket_MV_ds_9.datasource @@ -0,0 +1,30 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String + +ENGINE ReplacingMergeTree +ENGINE_PARTITION_KEY toYear(createdAt) +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, channel, sourceId +ENGINE_VER updatedAt diff --git a/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_0_ds.datasource b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_0_ds.datasource new file mode 100644 index 0000000000..f0a054f467 --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_0_ds.datasource @@ -0,0 +1,33 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String, + `gitChangedLines` UInt64, + `gitChangedLinesBucket` String, + `organizationCountryCode` LowCardinality(String), + `organizationName` String, + `snapshotId` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, memberId, organizationId diff --git a/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_1_ds.datasource b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_1_ds.datasource new file mode 100644 index 0000000000..f0a054f467 --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_1_ds.datasource @@ -0,0 +1,33 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String, + `gitChangedLines` UInt64, + `gitChangedLinesBucket` String, + `organizationCountryCode` LowCardinality(String), + `organizationName` String, + `snapshotId` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, memberId, organizationId diff --git a/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_2_ds.datasource b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_2_ds.datasource new file mode 100644 index 0000000000..f0a054f467 --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_2_ds.datasource @@ -0,0 +1,33 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String, + `gitChangedLines` UInt64, + `gitChangedLinesBucket` String, + `organizationCountryCode` LowCardinality(String), + `organizationName` String, + `snapshotId` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, memberId, organizationId diff --git a/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_3_ds.datasource b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_3_ds.datasource new file mode 100644 index 0000000000..f0a054f467 --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_3_ds.datasource @@ -0,0 +1,33 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String, + `gitChangedLines` UInt64, + `gitChangedLinesBucket` String, + `organizationCountryCode` LowCardinality(String), + `organizationName` String, + `snapshotId` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, memberId, organizationId diff --git a/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_4_ds.datasource b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_4_ds.datasource new file mode 100644 index 0000000000..f0a054f467 --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_4_ds.datasource @@ -0,0 +1,33 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String, + `gitChangedLines` UInt64, + `gitChangedLinesBucket` String, + `organizationCountryCode` LowCardinality(String), + `organizationName` String, + `snapshotId` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, memberId, organizationId diff --git a/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_5_ds.datasource b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_5_ds.datasource new file mode 100644 index 0000000000..f0a054f467 --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_5_ds.datasource @@ -0,0 +1,33 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String, + `gitChangedLines` UInt64, + `gitChangedLinesBucket` String, + `organizationCountryCode` LowCardinality(String), + `organizationName` String, + `snapshotId` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, memberId, organizationId diff --git a/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_6_ds.datasource b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_6_ds.datasource new file mode 100644 index 0000000000..f0a054f467 --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_6_ds.datasource @@ -0,0 +1,33 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String, + `gitChangedLines` UInt64, + `gitChangedLinesBucket` String, + `organizationCountryCode` LowCardinality(String), + `organizationName` String, + `snapshotId` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, memberId, organizationId diff --git a/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_7_ds.datasource b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_7_ds.datasource new file mode 100644 index 0000000000..f0a054f467 --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_7_ds.datasource @@ -0,0 +1,33 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String, + `gitChangedLines` UInt64, + `gitChangedLinesBucket` String, + `organizationCountryCode` LowCardinality(String), + `organizationName` String, + `snapshotId` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, memberId, organizationId diff --git a/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_8_ds.datasource b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_8_ds.datasource new file mode 100644 index 0000000000..f0a054f467 --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_8_ds.datasource @@ -0,0 +1,33 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String, + `gitChangedLines` UInt64, + `gitChangedLinesBucket` String, + `organizationCountryCode` LowCardinality(String), + `organizationName` String, + `snapshotId` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, memberId, organizationId diff --git a/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_9_ds.datasource b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_9_ds.datasource new file mode 100644 index 0000000000..f0a054f467 --- /dev/null +++ b/services/libs/tinybird/datasources/activityRelations_collection_deduplicated_cleaned_bucket_9_ds.datasource @@ -0,0 +1,33 @@ +SCHEMA > + `activityId` String, + `conversationId` String, + `createdAt` DateTime64(3), + `updatedAt` DateTime64(3), + `memberId` String, + `objectMemberId` String, + `objectMemberUsername` String, + `organizationId` String, + `parentId` String, + `platform` LowCardinality(String), + `segmentId` String, + `username` String, + `sourceId` String, + `type` LowCardinality(String), + `timestamp` DateTime64(3), + `sourceParentId` String, + `channel` String, + `sentimentScore` Int8, + `gitInsertions` UInt32, + `gitDeletions` UInt32, + `score` Int8, + `isContribution` UInt8, + `pullRequestReviewState` LowCardinality(String), + `collectionSlug` String, + `gitChangedLines` UInt64, + `gitChangedLinesBucket` String, + `organizationCountryCode` LowCardinality(String), + `organizationName` String, + `snapshotId` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug, segmentId, timestamp, type, platform, memberId, organizationId diff --git a/services/libs/tinybird/pipes/activities_filtered.pipe b/services/libs/tinybird/pipes/activities_filtered.pipe index f7bec9eab2..d18137909c 100644 --- a/services/libs/tinybird/pipes/activities_filtered.pipe +++ b/services/libs/tinybird/pipes/activities_filtered.pipe @@ -20,12 +20,15 @@ DESCRIPTION > - This pipe is consumed by many of downstream pipes and widgets across the platform for consistent activity filtering. - Performance is optimized through proper sorting keys on `segmentId`, `timestamp`, `type`, `platform`, and `memberId` in the source datasource. +TOKEN "insights-app-token" READ +TOKEN "activities_filtered_endpoint_read_8708" READ + NODE activities_filtered_bucket_routing SQL > % SELECT activityId as id, timestamp, type, platform, memberId, organizationId, segmentId FROM - {% if defined(collectionSlug) %} activityRelations_deduplicated_cleaned_bucket_union + {% if defined(collectionSlug) %} activityRelations_collection_bucket_union {% else %} activityRelations_bucket_routing {% end %} as a where diff --git a/services/libs/tinybird/pipes/activities_filtered_retention.pipe b/services/libs/tinybird/pipes/activities_filtered_retention.pipe index c919df4003..17bfb92214 100644 --- a/services/libs/tinybird/pipes/activities_filtered_retention.pipe +++ b/services/libs/tinybird/pipes/activities_filtered_retention.pipe @@ -24,7 +24,7 @@ SQL > % SELECT activityId as id, timestamp, type, platform, memberId, organizationId, segmentId FROM - {% if defined(collectionSlug) %} activityRelations_deduplicated_cleaned_bucket_union + {% if defined(collectionSlug) %} activityRelations_collection_bucket_union {% else %} activityRelations_bucket_routing {% end %} as a where diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_0.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_0.pipe new file mode 100644 index 0000000000..c7283f7a06 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_0.pipe @@ -0,0 +1,24 @@ +DESCRIPTION > + Fan-out materialized view for curated-collection bucket 0. Joins each inserted + activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 0. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. + +NODE curated_collection_slugs_0 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_0 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_0) + AND cityHash64(collectionSlug) % 10 = 0 + +TYPE MATERIALIZED +DATASOURCE activityRelations_collection_bucket_MV_ds_0 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_1.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_1.pipe new file mode 100644 index 0000000000..aa6b24cf25 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_1.pipe @@ -0,0 +1,24 @@ +DESCRIPTION > + Fan-out materialized view for curated-collection bucket 1. Joins each inserted + activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 1. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. + +NODE curated_collection_slugs_1 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_1 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_1) + AND cityHash64(collectionSlug) % 10 = 1 + +TYPE MATERIALIZED +DATASOURCE activityRelations_collection_bucket_MV_ds_1 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_2.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_2.pipe new file mode 100644 index 0000000000..55af3a5a0a --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_2.pipe @@ -0,0 +1,24 @@ +DESCRIPTION > + Fan-out materialized view for curated-collection bucket 2. Joins each inserted + activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 2. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. + +NODE curated_collection_slugs_2 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_2 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_2) + AND cityHash64(collectionSlug) % 10 = 2 + +TYPE MATERIALIZED +DATASOURCE activityRelations_collection_bucket_MV_ds_2 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_3.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_3.pipe new file mode 100644 index 0000000000..5bb4919436 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_3.pipe @@ -0,0 +1,24 @@ +DESCRIPTION > + Fan-out materialized view for curated-collection bucket 3. Joins each inserted + activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 3. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. + +NODE curated_collection_slugs_3 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_3 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_3) + AND cityHash64(collectionSlug) % 10 = 3 + +TYPE MATERIALIZED +DATASOURCE activityRelations_collection_bucket_MV_ds_3 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_4.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_4.pipe new file mode 100644 index 0000000000..f2e75f80df --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_4.pipe @@ -0,0 +1,24 @@ +DESCRIPTION > + Fan-out materialized view for curated-collection bucket 4. Joins each inserted + activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 4. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. + +NODE curated_collection_slugs_4 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_4 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_4) + AND cityHash64(collectionSlug) % 10 = 4 + +TYPE MATERIALIZED +DATASOURCE activityRelations_collection_bucket_MV_ds_4 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_5.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_5.pipe new file mode 100644 index 0000000000..f8a7bad1e5 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_5.pipe @@ -0,0 +1,24 @@ +DESCRIPTION > + Fan-out materialized view for curated-collection bucket 5. Joins each inserted + activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 5. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. + +NODE curated_collection_slugs_5 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_5 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_5) + AND cityHash64(collectionSlug) % 10 = 5 + +TYPE MATERIALIZED +DATASOURCE activityRelations_collection_bucket_MV_ds_5 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_6.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_6.pipe new file mode 100644 index 0000000000..b86ad03b4e --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_6.pipe @@ -0,0 +1,24 @@ +DESCRIPTION > + Fan-out materialized view for curated-collection bucket 6. Joins each inserted + activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 6. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. + +NODE curated_collection_slugs_6 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_6 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_6) + AND cityHash64(collectionSlug) % 10 = 6 + +TYPE MATERIALIZED +DATASOURCE activityRelations_collection_bucket_MV_ds_6 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_7.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_7.pipe new file mode 100644 index 0000000000..e2587f92b9 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_7.pipe @@ -0,0 +1,24 @@ +DESCRIPTION > + Fan-out materialized view for curated-collection bucket 7. Joins each inserted + activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 7. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. + +NODE curated_collection_slugs_7 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_7 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_7) + AND cityHash64(collectionSlug) % 10 = 7 + +TYPE MATERIALIZED +DATASOURCE activityRelations_collection_bucket_MV_ds_7 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_8.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_8.pipe new file mode 100644 index 0000000000..16c2d23d01 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_8.pipe @@ -0,0 +1,24 @@ +DESCRIPTION > + Fan-out materialized view for curated-collection bucket 8. Joins each inserted + activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 8. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. + +NODE curated_collection_slugs_8 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_8 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_8) + AND cityHash64(collectionSlug) % 10 = 8 + +TYPE MATERIALIZED +DATASOURCE activityRelations_collection_bucket_MV_ds_8 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_9.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_9.pipe new file mode 100644 index 0000000000..99ff7a28ac --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_9.pipe @@ -0,0 +1,24 @@ +DESCRIPTION > + Fan-out materialized view for curated-collection bucket 9. Joins each inserted + activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 9. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. + +NODE curated_collection_slugs_9 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_9 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_9) + AND cityHash64(collectionSlug) % 10 = 9 + +TYPE MATERIALIZED +DATASOURCE activityRelations_collection_bucket_MV_ds_9 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe new file mode 100644 index 0000000000..ad654d3d59 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe @@ -0,0 +1,25 @@ +DESCRIPTION > + On-demand backfill/bootstrap for curated-collection bucket 0 - populates + activityRelations_collection_bucket_MV_ds_0 from the full historical activityRelations + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + activityRelations_bucket_MV_snapshot_0.pipe's role in the project-bucketing "Adding New + Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV + (activityRelations_collection_bucket_MV_0.pipe) then keeps it current on new inserts. + +NODE curated_collection_slugs_snapshot_0 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_snapshot_0 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_0) + AND cityHash64(collectionSlug) % 10 = 0 + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_0 +COPY_MODE append +COPY_SCHEDULE @on-demand diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe new file mode 100644 index 0000000000..ea107b4e42 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe @@ -0,0 +1,25 @@ +DESCRIPTION > + On-demand backfill/bootstrap for curated-collection bucket 1 - populates + activityRelations_collection_bucket_MV_ds_1 from the full historical activityRelations + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + activityRelations_bucket_MV_snapshot_1.pipe's role in the project-bucketing "Adding New + Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV + (activityRelations_collection_bucket_MV_1.pipe) then keeps it current on new inserts. + +NODE curated_collection_slugs_snapshot_1 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_snapshot_1 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_1) + AND cityHash64(collectionSlug) % 10 = 1 + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_1 +COPY_MODE append +COPY_SCHEDULE @on-demand diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe new file mode 100644 index 0000000000..37715aeb70 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe @@ -0,0 +1,25 @@ +DESCRIPTION > + On-demand backfill/bootstrap for curated-collection bucket 2 - populates + activityRelations_collection_bucket_MV_ds_2 from the full historical activityRelations + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + activityRelations_bucket_MV_snapshot_2.pipe's role in the project-bucketing "Adding New + Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV + (activityRelations_collection_bucket_MV_2.pipe) then keeps it current on new inserts. + +NODE curated_collection_slugs_snapshot_2 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_snapshot_2 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_2) + AND cityHash64(collectionSlug) % 10 = 2 + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_2 +COPY_MODE append +COPY_SCHEDULE @on-demand diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe new file mode 100644 index 0000000000..30d116ede9 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe @@ -0,0 +1,25 @@ +DESCRIPTION > + On-demand backfill/bootstrap for curated-collection bucket 3 - populates + activityRelations_collection_bucket_MV_ds_3 from the full historical activityRelations + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + activityRelations_bucket_MV_snapshot_3.pipe's role in the project-bucketing "Adding New + Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV + (activityRelations_collection_bucket_MV_3.pipe) then keeps it current on new inserts. + +NODE curated_collection_slugs_snapshot_3 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_snapshot_3 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_3) + AND cityHash64(collectionSlug) % 10 = 3 + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_3 +COPY_MODE append +COPY_SCHEDULE @on-demand diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe new file mode 100644 index 0000000000..ce55cc181a --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe @@ -0,0 +1,25 @@ +DESCRIPTION > + On-demand backfill/bootstrap for curated-collection bucket 4 - populates + activityRelations_collection_bucket_MV_ds_4 from the full historical activityRelations + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + activityRelations_bucket_MV_snapshot_4.pipe's role in the project-bucketing "Adding New + Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV + (activityRelations_collection_bucket_MV_4.pipe) then keeps it current on new inserts. + +NODE curated_collection_slugs_snapshot_4 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_snapshot_4 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_4) + AND cityHash64(collectionSlug) % 10 = 4 + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_4 +COPY_MODE append +COPY_SCHEDULE @on-demand diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe new file mode 100644 index 0000000000..efe884cfbf --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe @@ -0,0 +1,25 @@ +DESCRIPTION > + On-demand backfill/bootstrap for curated-collection bucket 5 - populates + activityRelations_collection_bucket_MV_ds_5 from the full historical activityRelations + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + activityRelations_bucket_MV_snapshot_5.pipe's role in the project-bucketing "Adding New + Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV + (activityRelations_collection_bucket_MV_5.pipe) then keeps it current on new inserts. + +NODE curated_collection_slugs_snapshot_5 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_snapshot_5 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_5) + AND cityHash64(collectionSlug) % 10 = 5 + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_5 +COPY_MODE append +COPY_SCHEDULE @on-demand diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe new file mode 100644 index 0000000000..f927170044 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe @@ -0,0 +1,25 @@ +DESCRIPTION > + On-demand backfill/bootstrap for curated-collection bucket 6 - populates + activityRelations_collection_bucket_MV_ds_6 from the full historical activityRelations + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + activityRelations_bucket_MV_snapshot_6.pipe's role in the project-bucketing "Adding New + Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV + (activityRelations_collection_bucket_MV_6.pipe) then keeps it current on new inserts. + +NODE curated_collection_slugs_snapshot_6 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_snapshot_6 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_6) + AND cityHash64(collectionSlug) % 10 = 6 + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_6 +COPY_MODE append +COPY_SCHEDULE @on-demand diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe new file mode 100644 index 0000000000..0cc8d5b40f --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe @@ -0,0 +1,25 @@ +DESCRIPTION > + On-demand backfill/bootstrap for curated-collection bucket 7 - populates + activityRelations_collection_bucket_MV_ds_7 from the full historical activityRelations + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + activityRelations_bucket_MV_snapshot_7.pipe's role in the project-bucketing "Adding New + Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV + (activityRelations_collection_bucket_MV_7.pipe) then keeps it current on new inserts. + +NODE curated_collection_slugs_snapshot_7 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_snapshot_7 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_7) + AND cityHash64(collectionSlug) % 10 = 7 + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_7 +COPY_MODE append +COPY_SCHEDULE @on-demand diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe new file mode 100644 index 0000000000..45d2d9759c --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe @@ -0,0 +1,25 @@ +DESCRIPTION > + On-demand backfill/bootstrap for curated-collection bucket 8 - populates + activityRelations_collection_bucket_MV_ds_8 from the full historical activityRelations + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + activityRelations_bucket_MV_snapshot_8.pipe's role in the project-bucketing "Adding New + Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV + (activityRelations_collection_bucket_MV_8.pipe) then keeps it current on new inserts. + +NODE curated_collection_slugs_snapshot_8 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_snapshot_8 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_8) + AND cityHash64(collectionSlug) % 10 = 8 + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_8 +COPY_MODE append +COPY_SCHEDULE @on-demand diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe new file mode 100644 index 0000000000..807518aa79 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe @@ -0,0 +1,25 @@ +DESCRIPTION > + On-demand backfill/bootstrap for curated-collection bucket 9 - populates + activityRelations_collection_bucket_MV_ds_9 from the full historical activityRelations + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + activityRelations_bucket_MV_snapshot_9.pipe's role in the project-bucketing "Adding New + Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV + (activityRelations_collection_bucket_MV_9.pipe) then keeps it current on new inserts. + +NODE curated_collection_slugs_snapshot_9 +SQL > + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + +NODE collection_fanout_snapshot_9 +SQL > + SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug + FROM activityRelations a + LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId + WHERE + collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_9) + AND cityHash64(collectionSlug) % 10 = 9 + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_9 +COPY_MODE append +COPY_SCHEDULE @on-demand diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_0.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_0.pipe new file mode 100644 index 0000000000..4c40e14d66 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_0.pipe @@ -0,0 +1,57 @@ +DESCRIPTION > + Daily enrichment/cleaning copy pipe for curated-collection bucket 0. Same + country/org-denormalization and bot/repo-enablement filtering logic as the project-bucket + copy pipes (activityRelations_bucket_clean_enrich_copy_pipe_0.pipe) - the result depends on + activityRelations, member bot-status, and repository enable/disable state, none of which are + append-only-safe, so this recomputes the full bucket via COPY_MODE replace rather than + appending. Scheduled daily, staggered 5 minutes apart across all 10 collection buckets (this + bucket at :) so all 10 full-replace jobs don't fire simultaneously. + +NODE country_mapping_array +SQL > + SELECT groupArray((country, country_code, timezone_offset)) AS country_data FROM country_mapping_ds + +NODE activityRelations_collection_deduplicated_cleaned_denormalized_0 +SQL > + WITH + upperUTF8(o.location) AS search_u, + (SELECT arrayMap(x -> upperUTF8(x .1), country_data) FROM country_mapping_array) AS names_u, + (SELECT country_data FROM country_mapping_array) AS mapping, + multiSearchFirstIndexUTF8(search_u, names_u) AS idx, + if(idx = 0, ('Unknown', 'XX', 0), mapping[idx]) AS country_data + SELECT + activityRelations_collection_bucket_MV_ds_0.*, + (gitInsertions + gitDeletions) as gitChangedLines, + case + when gitChangedLines > 0 and gitChangedLines < 10 + then '1-9' + when gitChangedLines > 9 and gitChangedLines < 60 + then '10-59' + when gitChangedLines > 59 and gitChangedLines < 100 + then '60-99' + when gitChangedLines > 99 and gitChangedLines < 500 + then '100-499' + when gitChangedLines > 499 + then '500+' + else '' + end as "gitChangedLinesBucket", + CAST(country_data .2 AS LowCardinality(String)) AS organizationCountryCode, + o.displayName as "organizationName", + toStartOfInterval(now(), INTERVAL 1 hour) as snapshotId + from activityRelations_collection_bucket_MV_ds_0 final + left join organizations o final on o.id = activityRelations_collection_bucket_MV_ds_0.organizationId + where + memberId IN (SELECT id FROM members_sorted) + and ( + ( + platform IN ('git', 'gerrit', 'github', 'gitlab') + AND (lower(channel), activityRelations_collection_bucket_MV_ds_0.segmentId) + IN (SELECT lower(channel), segmentId FROM repos_to_channels) + ) + OR platform NOT IN ('git', 'gerrit', 'github', 'gitlab') + ) + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_0_ds +COPY_MODE replace +COPY_SCHEDULE 0 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_1.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_1.pipe new file mode 100644 index 0000000000..abad7d2ea8 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_1.pipe @@ -0,0 +1,57 @@ +DESCRIPTION > + Daily enrichment/cleaning copy pipe for curated-collection bucket 1. Same + country/org-denormalization and bot/repo-enablement filtering logic as the project-bucket + copy pipes (activityRelations_bucket_clean_enrich_copy_pipe_1.pipe) - the result depends on + activityRelations, member bot-status, and repository enable/disable state, none of which are + append-only-safe, so this recomputes the full bucket via COPY_MODE replace rather than + appending. Scheduled daily, staggered 5 minutes apart across all 10 collection buckets (this + bucket at :0) so all 10 full-replace jobs don't fire simultaneously. + +NODE country_mapping_array +SQL > + SELECT groupArray((country, country_code, timezone_offset)) AS country_data FROM country_mapping_ds + +NODE activityRelations_collection_deduplicated_cleaned_denormalized_1 +SQL > + WITH + upperUTF8(o.location) AS search_u, + (SELECT arrayMap(x -> upperUTF8(x .1), country_data) FROM country_mapping_array) AS names_u, + (SELECT country_data FROM country_mapping_array) AS mapping, + multiSearchFirstIndexUTF8(search_u, names_u) AS idx, + if(idx = 0, ('Unknown', 'XX', 0), mapping[idx]) AS country_data + SELECT + activityRelations_collection_bucket_MV_ds_1.*, + (gitInsertions + gitDeletions) as gitChangedLines, + case + when gitChangedLines > 0 and gitChangedLines < 10 + then '1-9' + when gitChangedLines > 9 and gitChangedLines < 60 + then '10-59' + when gitChangedLines > 59 and gitChangedLines < 100 + then '60-99' + when gitChangedLines > 99 and gitChangedLines < 500 + then '100-499' + when gitChangedLines > 499 + then '500+' + else '' + end as "gitChangedLinesBucket", + CAST(country_data .2 AS LowCardinality(String)) AS organizationCountryCode, + o.displayName as "organizationName", + toStartOfInterval(now(), INTERVAL 1 hour) as snapshotId + from activityRelations_collection_bucket_MV_ds_1 final + left join organizations o final on o.id = activityRelations_collection_bucket_MV_ds_1.organizationId + where + memberId IN (SELECT id FROM members_sorted) + and ( + ( + platform IN ('git', 'gerrit', 'github', 'gitlab') + AND (lower(channel), activityRelations_collection_bucket_MV_ds_1.segmentId) + IN (SELECT lower(channel), segmentId FROM repos_to_channels) + ) + OR platform NOT IN ('git', 'gerrit', 'github', 'gitlab') + ) + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_1_ds +COPY_MODE replace +COPY_SCHEDULE 0 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_2.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_2.pipe new file mode 100644 index 0000000000..3bcffaed5f --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_2.pipe @@ -0,0 +1,57 @@ +DESCRIPTION > + Daily enrichment/cleaning copy pipe for curated-collection bucket 2. Same + country/org-denormalization and bot/repo-enablement filtering logic as the project-bucket + copy pipes (activityRelations_bucket_clean_enrich_copy_pipe_2.pipe) - the result depends on + activityRelations, member bot-status, and repository enable/disable state, none of which are + append-only-safe, so this recomputes the full bucket via COPY_MODE replace rather than + appending. Scheduled daily, staggered 5 minutes apart across all 10 collection buckets (this + bucket at :5) so all 10 full-replace jobs don't fire simultaneously. + +NODE country_mapping_array +SQL > + SELECT groupArray((country, country_code, timezone_offset)) AS country_data FROM country_mapping_ds + +NODE activityRelations_collection_deduplicated_cleaned_denormalized_2 +SQL > + WITH + upperUTF8(o.location) AS search_u, + (SELECT arrayMap(x -> upperUTF8(x .1), country_data) FROM country_mapping_array) AS names_u, + (SELECT country_data FROM country_mapping_array) AS mapping, + multiSearchFirstIndexUTF8(search_u, names_u) AS idx, + if(idx = 0, ('Unknown', 'XX', 0), mapping[idx]) AS country_data + SELECT + activityRelations_collection_bucket_MV_ds_2.*, + (gitInsertions + gitDeletions) as gitChangedLines, + case + when gitChangedLines > 0 and gitChangedLines < 10 + then '1-9' + when gitChangedLines > 9 and gitChangedLines < 60 + then '10-59' + when gitChangedLines > 59 and gitChangedLines < 100 + then '60-99' + when gitChangedLines > 99 and gitChangedLines < 500 + then '100-499' + when gitChangedLines > 499 + then '500+' + else '' + end as "gitChangedLinesBucket", + CAST(country_data .2 AS LowCardinality(String)) AS organizationCountryCode, + o.displayName as "organizationName", + toStartOfInterval(now(), INTERVAL 1 hour) as snapshotId + from activityRelations_collection_bucket_MV_ds_2 final + left join organizations o final on o.id = activityRelations_collection_bucket_MV_ds_2.organizationId + where + memberId IN (SELECT id FROM members_sorted) + and ( + ( + platform IN ('git', 'gerrit', 'github', 'gitlab') + AND (lower(channel), activityRelations_collection_bucket_MV_ds_2.segmentId) + IN (SELECT lower(channel), segmentId FROM repos_to_channels) + ) + OR platform NOT IN ('git', 'gerrit', 'github', 'gitlab') + ) + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_2_ds +COPY_MODE replace +COPY_SCHEDULE 5 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_3.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_3.pipe new file mode 100644 index 0000000000..7657cc7f81 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_3.pipe @@ -0,0 +1,57 @@ +DESCRIPTION > + Daily enrichment/cleaning copy pipe for curated-collection bucket 3. Same + country/org-denormalization and bot/repo-enablement filtering logic as the project-bucket + copy pipes (activityRelations_bucket_clean_enrich_copy_pipe_3.pipe) - the result depends on + activityRelations, member bot-status, and repository enable/disable state, none of which are + append-only-safe, so this recomputes the full bucket via COPY_MODE replace rather than + appending. Scheduled daily, staggered 5 minutes apart across all 10 collection buckets (this + bucket at :10) so all 10 full-replace jobs don't fire simultaneously. + +NODE country_mapping_array +SQL > + SELECT groupArray((country, country_code, timezone_offset)) AS country_data FROM country_mapping_ds + +NODE activityRelations_collection_deduplicated_cleaned_denormalized_3 +SQL > + WITH + upperUTF8(o.location) AS search_u, + (SELECT arrayMap(x -> upperUTF8(x .1), country_data) FROM country_mapping_array) AS names_u, + (SELECT country_data FROM country_mapping_array) AS mapping, + multiSearchFirstIndexUTF8(search_u, names_u) AS idx, + if(idx = 0, ('Unknown', 'XX', 0), mapping[idx]) AS country_data + SELECT + activityRelations_collection_bucket_MV_ds_3.*, + (gitInsertions + gitDeletions) as gitChangedLines, + case + when gitChangedLines > 0 and gitChangedLines < 10 + then '1-9' + when gitChangedLines > 9 and gitChangedLines < 60 + then '10-59' + when gitChangedLines > 59 and gitChangedLines < 100 + then '60-99' + when gitChangedLines > 99 and gitChangedLines < 500 + then '100-499' + when gitChangedLines > 499 + then '500+' + else '' + end as "gitChangedLinesBucket", + CAST(country_data .2 AS LowCardinality(String)) AS organizationCountryCode, + o.displayName as "organizationName", + toStartOfInterval(now(), INTERVAL 1 hour) as snapshotId + from activityRelations_collection_bucket_MV_ds_3 final + left join organizations o final on o.id = activityRelations_collection_bucket_MV_ds_3.organizationId + where + memberId IN (SELECT id FROM members_sorted) + and ( + ( + platform IN ('git', 'gerrit', 'github', 'gitlab') + AND (lower(channel), activityRelations_collection_bucket_MV_ds_3.segmentId) + IN (SELECT lower(channel), segmentId FROM repos_to_channels) + ) + OR platform NOT IN ('git', 'gerrit', 'github', 'gitlab') + ) + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_3_ds +COPY_MODE replace +COPY_SCHEDULE 10 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_4.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_4.pipe new file mode 100644 index 0000000000..b899ef8043 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_4.pipe @@ -0,0 +1,57 @@ +DESCRIPTION > + Daily enrichment/cleaning copy pipe for curated-collection bucket 4. Same + country/org-denormalization and bot/repo-enablement filtering logic as the project-bucket + copy pipes (activityRelations_bucket_clean_enrich_copy_pipe_4.pipe) - the result depends on + activityRelations, member bot-status, and repository enable/disable state, none of which are + append-only-safe, so this recomputes the full bucket via COPY_MODE replace rather than + appending. Scheduled daily, staggered 5 minutes apart across all 10 collection buckets (this + bucket at :15) so all 10 full-replace jobs don't fire simultaneously. + +NODE country_mapping_array +SQL > + SELECT groupArray((country, country_code, timezone_offset)) AS country_data FROM country_mapping_ds + +NODE activityRelations_collection_deduplicated_cleaned_denormalized_4 +SQL > + WITH + upperUTF8(o.location) AS search_u, + (SELECT arrayMap(x -> upperUTF8(x .1), country_data) FROM country_mapping_array) AS names_u, + (SELECT country_data FROM country_mapping_array) AS mapping, + multiSearchFirstIndexUTF8(search_u, names_u) AS idx, + if(idx = 0, ('Unknown', 'XX', 0), mapping[idx]) AS country_data + SELECT + activityRelations_collection_bucket_MV_ds_4.*, + (gitInsertions + gitDeletions) as gitChangedLines, + case + when gitChangedLines > 0 and gitChangedLines < 10 + then '1-9' + when gitChangedLines > 9 and gitChangedLines < 60 + then '10-59' + when gitChangedLines > 59 and gitChangedLines < 100 + then '60-99' + when gitChangedLines > 99 and gitChangedLines < 500 + then '100-499' + when gitChangedLines > 499 + then '500+' + else '' + end as "gitChangedLinesBucket", + CAST(country_data .2 AS LowCardinality(String)) AS organizationCountryCode, + o.displayName as "organizationName", + toStartOfInterval(now(), INTERVAL 1 hour) as snapshotId + from activityRelations_collection_bucket_MV_ds_4 final + left join organizations o final on o.id = activityRelations_collection_bucket_MV_ds_4.organizationId + where + memberId IN (SELECT id FROM members_sorted) + and ( + ( + platform IN ('git', 'gerrit', 'github', 'gitlab') + AND (lower(channel), activityRelations_collection_bucket_MV_ds_4.segmentId) + IN (SELECT lower(channel), segmentId FROM repos_to_channels) + ) + OR platform NOT IN ('git', 'gerrit', 'github', 'gitlab') + ) + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_4_ds +COPY_MODE replace +COPY_SCHEDULE 15 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_5.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_5.pipe new file mode 100644 index 0000000000..da2bfd8b2b --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_5.pipe @@ -0,0 +1,57 @@ +DESCRIPTION > + Daily enrichment/cleaning copy pipe for curated-collection bucket 5. Same + country/org-denormalization and bot/repo-enablement filtering logic as the project-bucket + copy pipes (activityRelations_bucket_clean_enrich_copy_pipe_5.pipe) - the result depends on + activityRelations, member bot-status, and repository enable/disable state, none of which are + append-only-safe, so this recomputes the full bucket via COPY_MODE replace rather than + appending. Scheduled daily, staggered 5 minutes apart across all 10 collection buckets (this + bucket at :20) so all 10 full-replace jobs don't fire simultaneously. + +NODE country_mapping_array +SQL > + SELECT groupArray((country, country_code, timezone_offset)) AS country_data FROM country_mapping_ds + +NODE activityRelations_collection_deduplicated_cleaned_denormalized_5 +SQL > + WITH + upperUTF8(o.location) AS search_u, + (SELECT arrayMap(x -> upperUTF8(x .1), country_data) FROM country_mapping_array) AS names_u, + (SELECT country_data FROM country_mapping_array) AS mapping, + multiSearchFirstIndexUTF8(search_u, names_u) AS idx, + if(idx = 0, ('Unknown', 'XX', 0), mapping[idx]) AS country_data + SELECT + activityRelations_collection_bucket_MV_ds_5.*, + (gitInsertions + gitDeletions) as gitChangedLines, + case + when gitChangedLines > 0 and gitChangedLines < 10 + then '1-9' + when gitChangedLines > 9 and gitChangedLines < 60 + then '10-59' + when gitChangedLines > 59 and gitChangedLines < 100 + then '60-99' + when gitChangedLines > 99 and gitChangedLines < 500 + then '100-499' + when gitChangedLines > 499 + then '500+' + else '' + end as "gitChangedLinesBucket", + CAST(country_data .2 AS LowCardinality(String)) AS organizationCountryCode, + o.displayName as "organizationName", + toStartOfInterval(now(), INTERVAL 1 hour) as snapshotId + from activityRelations_collection_bucket_MV_ds_5 final + left join organizations o final on o.id = activityRelations_collection_bucket_MV_ds_5.organizationId + where + memberId IN (SELECT id FROM members_sorted) + and ( + ( + platform IN ('git', 'gerrit', 'github', 'gitlab') + AND (lower(channel), activityRelations_collection_bucket_MV_ds_5.segmentId) + IN (SELECT lower(channel), segmentId FROM repos_to_channels) + ) + OR platform NOT IN ('git', 'gerrit', 'github', 'gitlab') + ) + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_5_ds +COPY_MODE replace +COPY_SCHEDULE 20 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_6.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_6.pipe new file mode 100644 index 0000000000..16e8482c63 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_6.pipe @@ -0,0 +1,57 @@ +DESCRIPTION > + Daily enrichment/cleaning copy pipe for curated-collection bucket 6. Same + country/org-denormalization and bot/repo-enablement filtering logic as the project-bucket + copy pipes (activityRelations_bucket_clean_enrich_copy_pipe_6.pipe) - the result depends on + activityRelations, member bot-status, and repository enable/disable state, none of which are + append-only-safe, so this recomputes the full bucket via COPY_MODE replace rather than + appending. Scheduled daily, staggered 5 minutes apart across all 10 collection buckets (this + bucket at :25) so all 10 full-replace jobs don't fire simultaneously. + +NODE country_mapping_array +SQL > + SELECT groupArray((country, country_code, timezone_offset)) AS country_data FROM country_mapping_ds + +NODE activityRelations_collection_deduplicated_cleaned_denormalized_6 +SQL > + WITH + upperUTF8(o.location) AS search_u, + (SELECT arrayMap(x -> upperUTF8(x .1), country_data) FROM country_mapping_array) AS names_u, + (SELECT country_data FROM country_mapping_array) AS mapping, + multiSearchFirstIndexUTF8(search_u, names_u) AS idx, + if(idx = 0, ('Unknown', 'XX', 0), mapping[idx]) AS country_data + SELECT + activityRelations_collection_bucket_MV_ds_6.*, + (gitInsertions + gitDeletions) as gitChangedLines, + case + when gitChangedLines > 0 and gitChangedLines < 10 + then '1-9' + when gitChangedLines > 9 and gitChangedLines < 60 + then '10-59' + when gitChangedLines > 59 and gitChangedLines < 100 + then '60-99' + when gitChangedLines > 99 and gitChangedLines < 500 + then '100-499' + when gitChangedLines > 499 + then '500+' + else '' + end as "gitChangedLinesBucket", + CAST(country_data .2 AS LowCardinality(String)) AS organizationCountryCode, + o.displayName as "organizationName", + toStartOfInterval(now(), INTERVAL 1 hour) as snapshotId + from activityRelations_collection_bucket_MV_ds_6 final + left join organizations o final on o.id = activityRelations_collection_bucket_MV_ds_6.organizationId + where + memberId IN (SELECT id FROM members_sorted) + and ( + ( + platform IN ('git', 'gerrit', 'github', 'gitlab') + AND (lower(channel), activityRelations_collection_bucket_MV_ds_6.segmentId) + IN (SELECT lower(channel), segmentId FROM repos_to_channels) + ) + OR platform NOT IN ('git', 'gerrit', 'github', 'gitlab') + ) + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_6_ds +COPY_MODE replace +COPY_SCHEDULE 25 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_7.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_7.pipe new file mode 100644 index 0000000000..8d380e1fd4 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_7.pipe @@ -0,0 +1,57 @@ +DESCRIPTION > + Daily enrichment/cleaning copy pipe for curated-collection bucket 7. Same + country/org-denormalization and bot/repo-enablement filtering logic as the project-bucket + copy pipes (activityRelations_bucket_clean_enrich_copy_pipe_7.pipe) - the result depends on + activityRelations, member bot-status, and repository enable/disable state, none of which are + append-only-safe, so this recomputes the full bucket via COPY_MODE replace rather than + appending. Scheduled daily, staggered 5 minutes apart across all 10 collection buckets (this + bucket at :30) so all 10 full-replace jobs don't fire simultaneously. + +NODE country_mapping_array +SQL > + SELECT groupArray((country, country_code, timezone_offset)) AS country_data FROM country_mapping_ds + +NODE activityRelations_collection_deduplicated_cleaned_denormalized_7 +SQL > + WITH + upperUTF8(o.location) AS search_u, + (SELECT arrayMap(x -> upperUTF8(x .1), country_data) FROM country_mapping_array) AS names_u, + (SELECT country_data FROM country_mapping_array) AS mapping, + multiSearchFirstIndexUTF8(search_u, names_u) AS idx, + if(idx = 0, ('Unknown', 'XX', 0), mapping[idx]) AS country_data + SELECT + activityRelations_collection_bucket_MV_ds_7.*, + (gitInsertions + gitDeletions) as gitChangedLines, + case + when gitChangedLines > 0 and gitChangedLines < 10 + then '1-9' + when gitChangedLines > 9 and gitChangedLines < 60 + then '10-59' + when gitChangedLines > 59 and gitChangedLines < 100 + then '60-99' + when gitChangedLines > 99 and gitChangedLines < 500 + then '100-499' + when gitChangedLines > 499 + then '500+' + else '' + end as "gitChangedLinesBucket", + CAST(country_data .2 AS LowCardinality(String)) AS organizationCountryCode, + o.displayName as "organizationName", + toStartOfInterval(now(), INTERVAL 1 hour) as snapshotId + from activityRelations_collection_bucket_MV_ds_7 final + left join organizations o final on o.id = activityRelations_collection_bucket_MV_ds_7.organizationId + where + memberId IN (SELECT id FROM members_sorted) + and ( + ( + platform IN ('git', 'gerrit', 'github', 'gitlab') + AND (lower(channel), activityRelations_collection_bucket_MV_ds_7.segmentId) + IN (SELECT lower(channel), segmentId FROM repos_to_channels) + ) + OR platform NOT IN ('git', 'gerrit', 'github', 'gitlab') + ) + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_7_ds +COPY_MODE replace +COPY_SCHEDULE 30 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_8.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_8.pipe new file mode 100644 index 0000000000..f19f702038 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_8.pipe @@ -0,0 +1,57 @@ +DESCRIPTION > + Daily enrichment/cleaning copy pipe for curated-collection bucket 8. Same + country/org-denormalization and bot/repo-enablement filtering logic as the project-bucket + copy pipes (activityRelations_bucket_clean_enrich_copy_pipe_8.pipe) - the result depends on + activityRelations, member bot-status, and repository enable/disable state, none of which are + append-only-safe, so this recomputes the full bucket via COPY_MODE replace rather than + appending. Scheduled daily, staggered 5 minutes apart across all 10 collection buckets (this + bucket at :35) so all 10 full-replace jobs don't fire simultaneously. + +NODE country_mapping_array +SQL > + SELECT groupArray((country, country_code, timezone_offset)) AS country_data FROM country_mapping_ds + +NODE activityRelations_collection_deduplicated_cleaned_denormalized_8 +SQL > + WITH + upperUTF8(o.location) AS search_u, + (SELECT arrayMap(x -> upperUTF8(x .1), country_data) FROM country_mapping_array) AS names_u, + (SELECT country_data FROM country_mapping_array) AS mapping, + multiSearchFirstIndexUTF8(search_u, names_u) AS idx, + if(idx = 0, ('Unknown', 'XX', 0), mapping[idx]) AS country_data + SELECT + activityRelations_collection_bucket_MV_ds_8.*, + (gitInsertions + gitDeletions) as gitChangedLines, + case + when gitChangedLines > 0 and gitChangedLines < 10 + then '1-9' + when gitChangedLines > 9 and gitChangedLines < 60 + then '10-59' + when gitChangedLines > 59 and gitChangedLines < 100 + then '60-99' + when gitChangedLines > 99 and gitChangedLines < 500 + then '100-499' + when gitChangedLines > 499 + then '500+' + else '' + end as "gitChangedLinesBucket", + CAST(country_data .2 AS LowCardinality(String)) AS organizationCountryCode, + o.displayName as "organizationName", + toStartOfInterval(now(), INTERVAL 1 hour) as snapshotId + from activityRelations_collection_bucket_MV_ds_8 final + left join organizations o final on o.id = activityRelations_collection_bucket_MV_ds_8.organizationId + where + memberId IN (SELECT id FROM members_sorted) + and ( + ( + platform IN ('git', 'gerrit', 'github', 'gitlab') + AND (lower(channel), activityRelations_collection_bucket_MV_ds_8.segmentId) + IN (SELECT lower(channel), segmentId FROM repos_to_channels) + ) + OR platform NOT IN ('git', 'gerrit', 'github', 'gitlab') + ) + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_8_ds +COPY_MODE replace +COPY_SCHEDULE 35 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_9.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_9.pipe new file mode 100644 index 0000000000..7f56eb9fb4 --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_9.pipe @@ -0,0 +1,57 @@ +DESCRIPTION > + Daily enrichment/cleaning copy pipe for curated-collection bucket 9. Same + country/org-denormalization and bot/repo-enablement filtering logic as the project-bucket + copy pipes (activityRelations_bucket_clean_enrich_copy_pipe_9.pipe) - the result depends on + activityRelations, member bot-status, and repository enable/disable state, none of which are + append-only-safe, so this recomputes the full bucket via COPY_MODE replace rather than + appending. Scheduled daily, staggered 5 minutes apart across all 10 collection buckets (this + bucket at :40) so all 10 full-replace jobs don't fire simultaneously. + +NODE country_mapping_array +SQL > + SELECT groupArray((country, country_code, timezone_offset)) AS country_data FROM country_mapping_ds + +NODE activityRelations_collection_deduplicated_cleaned_denormalized_9 +SQL > + WITH + upperUTF8(o.location) AS search_u, + (SELECT arrayMap(x -> upperUTF8(x .1), country_data) FROM country_mapping_array) AS names_u, + (SELECT country_data FROM country_mapping_array) AS mapping, + multiSearchFirstIndexUTF8(search_u, names_u) AS idx, + if(idx = 0, ('Unknown', 'XX', 0), mapping[idx]) AS country_data + SELECT + activityRelations_collection_bucket_MV_ds_9.*, + (gitInsertions + gitDeletions) as gitChangedLines, + case + when gitChangedLines > 0 and gitChangedLines < 10 + then '1-9' + when gitChangedLines > 9 and gitChangedLines < 60 + then '10-59' + when gitChangedLines > 59 and gitChangedLines < 100 + then '60-99' + when gitChangedLines > 99 and gitChangedLines < 500 + then '100-499' + when gitChangedLines > 499 + then '500+' + else '' + end as "gitChangedLinesBucket", + CAST(country_data .2 AS LowCardinality(String)) AS organizationCountryCode, + o.displayName as "organizationName", + toStartOfInterval(now(), INTERVAL 1 hour) as snapshotId + from activityRelations_collection_bucket_MV_ds_9 final + left join organizations o final on o.id = activityRelations_collection_bucket_MV_ds_9.organizationId + where + memberId IN (SELECT id FROM members_sorted) + and ( + ( + platform IN ('git', 'gerrit', 'github', 'gitlab') + AND (lower(channel), activityRelations_collection_bucket_MV_ds_9.segmentId) + IN (SELECT lower(channel), segmentId FROM repos_to_channels) + ) + OR platform NOT IN ('git', 'gerrit', 'github', 'gitlab') + ) + +TYPE COPY +TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_9_ds +COPY_MODE replace +COPY_SCHEDULE 40 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_union.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_union.pipe new file mode 100644 index 0000000000..5ff7bddbba --- /dev/null +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_union.pipe @@ -0,0 +1,59 @@ +DESCRIPTION > + - Query entrypoint for collection-scoped activity queries. Unlike the project-side + activityRelations_bucket_routing.pipe, this does NOT resolve to a single physical table via + Jinja branching - the bucket for a given collectionSlug is only known at SQL-execution time + (cityHash64(collectionSlug) % 10), and Tinybird's Jinja templating renders before any SQL + runs, so it cannot branch on a value only ClickHouse can compute. + - Instead this is a 10-way UNION ALL filtered by `collectionSlug =` on every branch. Each + activityRelations_collection_deduplicated_cleaned_bucket_N_ds is sorted with collectionSlug as + the first key (see its ENGINE_SORTING_KEY), so ClickHouse's sort-key pruning skips the other 9 + tables' data at read time - in practice this behaves like a single-bucket lookup without + needing a bucketId resolution step, while staying correct for any collectionSlug without + requiring the caller to pre-resolve a bucket. + - Callers must always filter on collectionSlug (via the WHERE clause below or their own), + otherwise this degrades into a genuine 10-way full scan - the same failure mode the old + single-datasource union pipe had for every query, which this design avoids only when + collectionSlug is present. + +NODE activityRelations_collection_bucket_union_0 +SQL > + % + SELECT * + FROM activityRelations_collection_deduplicated_cleaned_bucket_0_ds + WHERE collectionSlug = {{ String(collectionSlug, required=True) }} + UNION ALL + SELECT * + FROM activityRelations_collection_deduplicated_cleaned_bucket_1_ds + WHERE collectionSlug = {{ String(collectionSlug, required=True) }} + UNION ALL + SELECT * + FROM activityRelations_collection_deduplicated_cleaned_bucket_2_ds + WHERE collectionSlug = {{ String(collectionSlug, required=True) }} + UNION ALL + SELECT * + FROM activityRelations_collection_deduplicated_cleaned_bucket_3_ds + WHERE collectionSlug = {{ String(collectionSlug, required=True) }} + UNION ALL + SELECT * + FROM activityRelations_collection_deduplicated_cleaned_bucket_4_ds + WHERE collectionSlug = {{ String(collectionSlug, required=True) }} + UNION ALL + SELECT * + FROM activityRelations_collection_deduplicated_cleaned_bucket_5_ds + WHERE collectionSlug = {{ String(collectionSlug, required=True) }} + UNION ALL + SELECT * + FROM activityRelations_collection_deduplicated_cleaned_bucket_6_ds + WHERE collectionSlug = {{ String(collectionSlug, required=True) }} + UNION ALL + SELECT * + FROM activityRelations_collection_deduplicated_cleaned_bucket_7_ds + WHERE collectionSlug = {{ String(collectionSlug, required=True) }} + UNION ALL + SELECT * + FROM activityRelations_collection_deduplicated_cleaned_bucket_8_ds + WHERE collectionSlug = {{ String(collectionSlug, required=True) }} + UNION ALL + SELECT * + FROM activityRelations_collection_deduplicated_cleaned_bucket_9_ds + WHERE collectionSlug = {{ String(collectionSlug, required=True) }} diff --git a/services/libs/tinybird/pipes/activityTypes_by_project.pipe b/services/libs/tinybird/pipes/activityTypes_by_project.pipe index 7074602911..d850ebd683 100644 --- a/services/libs/tinybird/pipes/activityTypes_by_project.pipe +++ b/services/libs/tinybird/pipes/activityTypes_by_project.pipe @@ -6,7 +6,7 @@ DESCRIPTION > - Parameters: - `project`: String for project slug (e.g., 'k8s', 'tensorflow'). Passed to `segments_filtered`. Required unless `collectionSlug` is given. - `collectionSlug`: Collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`. Required unless `project` is given. - Collection requests read from `activityRelations_deduplicated_cleaned_bucket_union` (all 10 buckets) instead + Collection requests read from `activityRelations_collection_bucket_union` (all 10 collection-scoped buckets) instead of `activityRelations_bucket_routing` (a single bucket, defaulting to bucket 0 when `bucketId` is not injected), since a collection's member projects' segments can be routed to any of the 10 buckets - same pattern as `activities_filtered.pipe`. @@ -22,7 +22,7 @@ SQL > % SELECT DISTINCT a.type as activityType, a.platform, at.label FROM - {% if defined(collectionSlug) %} activityRelations_deduplicated_cleaned_bucket_union + {% if defined(collectionSlug) %} activityRelations_collection_bucket_union {% else %} activityRelations_bucket_routing {% end %} a INNER JOIN activityTypes at ON a.type = at.activityType AND a.platform = at.platform diff --git a/services/libs/tinybird/pipes/collection_buckets.pipe b/services/libs/tinybird/pipes/collection_buckets.pipe new file mode 100644 index 0000000000..fd39cd76b5 --- /dev/null +++ b/services/libs/tinybird/pipes/collection_buckets.pipe @@ -0,0 +1,16 @@ +DESCRIPTION > + - `collection_buckets.pipe` resolves a curated collection's slug to its bucketId, mirroring + `project_buckets.pipe`'s role for segments. + - Bucket assignment is `cityHash64(collectionSlug) % 10` - collections_filtered already + restricts to curated collections (`ssoUserId IS NULL`), so non-curated collections never + resolve a bucket here. + - Unlike project bucketing, a project can belong to multiple curated collections, so a + project's activities may be duplicated into multiple collection buckets - this pipe only + resolves the bucket for a single collection slug, it does not describe the fan-out. + +NODE get_collection_bucket +SQL > + % + SELECT cityHash64(slug) % 10 as bucketId + FROM collections_filtered + WHERE slug = {{ String(collectionSlug, description="Collection slug to resolve bucketId for", required=True) }} diff --git a/services/libs/tinybird/pipes/contributor_dependency.pipe b/services/libs/tinybird/pipes/contributor_dependency.pipe index 7b03efe2ef..fc7c5d9b45 100644 --- a/services/libs/tinybird/pipes/contributor_dependency.pipe +++ b/services/libs/tinybird/pipes/contributor_dependency.pipe @@ -3,9 +3,14 @@ DESCRIPTION > - Identifies contributors whose combined contributions reach or exceed 51% of total project activity (bus factor analysis). - Uses running total calculation to find the minimum set of contributors representing majority contribution share. - Combines data from `contributors_leaderboard` and `active_contributors` to provide dependency risk assessment. - - Primary use case: measuring project sustainability and identifying key contributor dependencies for risk management. + - Scopes to either a single project or every project in a collection - exactly one of `project`/`collectionSlug` + must be provided, inherited transitively from `contributors_leaderboard`/`active_contributors`'s own + `collectionSlug` branch. Callers doing bus-factor analysis should pass `limit=100` explicitly (same + convention regardless of scope) to reliably reach the 51% cutoff. + - Primary use case: measuring project (or collection) sustainability and identifying key contributor dependencies for risk management. - Parameters: - - `project`: Required string for project slug (e.g., 'k8s', 'tensorflow') - inherited from `contributors_leaderboard` + - `project`: Project slug (e.g., 'k8s', 'tensorflow') - inherited from `contributors_leaderboard`. Required unless `collectionSlug` is given. + - `collectionSlug`: Collection slug (e.g., 'cncf') - inherited from `contributors_leaderboard`. Required unless `project` is given. - `repos`: Optional array of repository URLs for filtering (inherited from `contributors_leaderboard`) - `startDate`: Optional DateTime filter for activities after timestamp (e.g., '2024-01-01 00:00:00') - `endDate`: Optional DateTime filter for activities before timestamp (e.g., '2024-12-31 23:59:59') @@ -14,6 +19,7 @@ DESCRIPTION > - `activity_type`: Optional string filter for single activity type (e.g., 'authored-commit') - `activity_types`: Optional array of activity types (e.g., ['authored-commit', 'co-authored-commit']) - `onlyContributions`: Optional boolean, defaults to 1 (contributions only), set to 0 for all activities + - `limit`: Optional integer for the number of top contributors to rank before finding the 51% cutoff. Inherited from `contributors_leaderboard` (default 10 if omitted) - pass 100 explicitly for bus-factor analysis. - Response: `id`, `displayName`, `githubHandleArray`, `contributionCount`, `contributionPercentage`, `roles`, `contributionPercentageRunningTotal`, `totalContributorCount` NODE contributions_percentage_running_total diff --git a/services/libs/tinybird/pipes/contributors_leaderboard.pipe b/services/libs/tinybird/pipes/contributors_leaderboard.pipe index eb931b50a2..f34e16836a 100644 --- a/services/libs/tinybird/pipes/contributors_leaderboard.pipe +++ b/services/libs/tinybird/pipes/contributors_leaderboard.pipe @@ -2,9 +2,22 @@ DESCRIPTION > - `contributors_leaderboard.pipe` serves the contributors leaderboard widget showing top contributors ranked by activity count. - Supports both count mode (`count=true`) and data mode for pagination-friendly leaderboard display. - Calculates contribution counts and percentages, showing different display names based on project type (LF vs non-LF). + - Scopes to either a single project (via `segments_filtered`) or every project in a collection (via + `segments_filtered_by_collection`, through `activities_filtered`'s own `collectionSlug` branch) - exactly + one of `project`/`collectionSlug` must be provided by the caller, same convention as `activities_filtered.pipe`. + - The LF/non-LF display-name toggle (`is_request_from_non_lf_project`) only applies in project scope - a + collection can mix LF and non-LF projects, so collection-scope requests always use `displayName`. + - Total contribution count is computed as a separate scalar (`contributors_total`) and cross-joined into + the per-member percentage calculation, instead of a window function over the full grouped result. A + window function forces ClickHouse to fully materialize every distinct `memberId` group before it can + rank and LIMIT - fine at single-project scale, but this is the same pattern that caused a documented + production timeout at collection scale (`cncf`: 149,677 distinct contributors) before + `collection_contributors_leaderboard.pipe` was introduced with the scalar-total design. This pipe now + uses that same design in both scopes. - Primary use case: powering contributor ranking widgets in development insights. - Parameters: - - `project`: Required string for project slug (e.g., 'k8s', 'tensorflow') - inherited from `segments_filtered` + - `project`: Project slug (e.g., 'k8s', 'tensorflow') - inherited from `segments_filtered`. Required unless `collectionSlug` is given. + - `collectionSlug`: Collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`. Required unless `project` is given. - `repos`: Optional array of repository URLs for filtering (inherited from `segments_filtered`) - `startDate`: Optional DateTime filter for activities after timestamp (e.g., '2024-01-01 00:00:00') - `endDate`: Optional DateTime filter for activities before timestamp (e.g., '2024-12-31 23:59:59') @@ -20,15 +33,17 @@ DESCRIPTION > - Count mode (`count=true`): `count` (total number of contributors) - Data mode (default): `id`, `avatar`, `displayName`, `githubHandleArray`, `contributionCount`, `contributionPercentage`, `roles` +NODE contributors_total +SQL > + % + SELECT count(af.id) AS total FROM activities_filtered af + NODE memberId_aggregates SQL > % {% if Boolean(count, false) %} SELECT count(distinct af.memberId) FROM activities_filtered af {% else %} - SELECT - af.memberId, - count(af.id) as "contributionCount", - ROUND(COUNT(af.id) * 100.0 / SUM(COUNT(af.id)) OVER (), 2) as "contributionPercentage" + SELECT af.memberId, count(af.id) as "contributionCount" FROM activities_filtered af group by af.memberId order by contributionCount desc, af.memberId desc @@ -39,12 +54,15 @@ SQL > NODE is_request_from_non_lf_project SQL > % - SELECT not isLF AS result - FROM insightsProjects final - WHERE - isNull (deletedAt) - AND enabled = 1 - and slug = {{ String(project, description="Filter by project slug", required=True) }} + {% if defined(project) %} + SELECT not isLF AS result + FROM insightsProjects final + WHERE + isNull (deletedAt) + AND enabled = 1 + and slug = {{ String(project, description="Filter by project slug", required=False) }} + {% else %} SELECT 0 AS result + {% end %} NODE contributors_leaderboard_2 SQL > @@ -58,12 +76,13 @@ SQL > case when is_non_lf.result then m.publicName else m.displayName end as displayName, m.githubHandleArray, ma.contributionCount, - ma.contributionPercentage, + ROUND(ma.contributionCount * 100.0 / total.total, 2) as contributionPercentage, mr.roles FROM members_sorted AS m any inner join memberId_aggregates ma on ma.memberId = m.id left join member_roles mr on mr.memberId = m.id cross join is_request_from_non_lf_project is_non_lf + cross join contributors_total total where m.id in (select memberId from memberId_aggregates) order by contributionCount desc {% end %} From 2920aeb0a2d9c083a245f0f6eaf2f6c9c594cc90 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 28 Jul 2026 16:44:12 +0100 Subject: [PATCH 02/10] chore: remove redundant collection-specific contributor pipes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit collection_contributors_leaderboard.pipe and collection_contributor_dependency.pipe (+ its copy pipe/datasource) are superseded now that contributors_leaderboard.pipe and contributor_dependency.pipe support collectionSlug scope directly via a scalar-total pattern. Insights has been updated to call the unified pipes and that change is deployed to production; these pipes had zero remaining traffic and were removed from Tinybird staging/production before this commit. Signed-off-by: Gašper Grom --- ..._contributor_dependency_copy_ds.datasource | 53 ------ .../collection_contributor_dependency.pipe | 109 ------------ ...ollection_contributor_dependency_copy.pipe | 155 ------------------ .../collection_contributors_leaderboard.pipe | 84 ---------- 4 files changed, 401 deletions(-) delete mode 100644 services/libs/tinybird/datasources/collection_contributor_dependency_copy_ds.datasource delete mode 100644 services/libs/tinybird/pipes/collection_contributor_dependency.pipe delete mode 100644 services/libs/tinybird/pipes/collection_contributor_dependency_copy.pipe delete mode 100644 services/libs/tinybird/pipes/collection_contributors_leaderboard.pipe diff --git a/services/libs/tinybird/datasources/collection_contributor_dependency_copy_ds.datasource b/services/libs/tinybird/datasources/collection_contributor_dependency_copy_ds.datasource deleted file mode 100644 index 33b09479bc..0000000000 --- a/services/libs/tinybird/datasources/collection_contributor_dependency_copy_ds.datasource +++ /dev/null @@ -1,53 +0,0 @@ -DESCRIPTION > - - `collection_contributor_dependency_copy_ds` contains precomputed bus-factor (contributor - dependency) results for the DEFAULT, unfiltered view of every collection - i.e. the exact - output `collection_contributor_dependency.pipe` would compute today with no `repos`/`startDate`/ - `endDate`/`platform`/`activity_type`/`includeCollaborations` override and `limit=100`. - - Created via `collection_contributor_dependency_copy.pipe` (`TYPE COPY`, daily schedule) to - avoid recomputing the same expensive per-member GROUP BY over the full activity set on every - request - see that pipe's DESCRIPTION for the full rationale (this materialization exists - because the live computation triggered a Tinybird production rate-limit/concurrency alert on - `lfx_insights` from timeouts under load). - - One row per (collectionSlug, memberId) that made it into that collection's top-100 ranked - contributors, already carrying the running-total percentage and the 51% cutoff membership flag - so the serving pipe only needs a cheap point lookup, not a recompute. - - `collectionSlug` is the collection this row belongs to. - - `id` is the member id (contributor). - - `displayName` is the member's display name. - - `githubHandleArray` is the member's GitHub usernames. - - `contributionCount` is the member's raw contribution count within the collection (all-time, - all-repos, code contributions only - the default filter set). - - `contributionPercentage` is `contributionCount` as a percentage of the collection's total - contribution count. - - `roles` is the member's maintainer roles within the collection, if any. - - `contributionPercentageRunningTotal` is the cumulative percentage when contributors are - ordered by `contributionPercentage DESC, id` - matches the live pipe's window function output. - - `rank` is the 1-indexed rank of this contributor within the collection (1 = highest - contributionCount), preserved so the serving pipe can `ORDER BY rank` cheaply instead of - re-sorting by percentage. - - `totalContributorCount` is the collection's total unique contributor count (same value - repeated on every row for the collection) - matches `active_contributors.contributorCount` in - the live pipe's response. - - `isBusFactorCutoff` marks the rows the live pipe's WHERE clause would return (running total - <= 51% OR running total minus this row's own percentage < 51%) - the serving pipe filters on - this instead of recomputing the cutoff logic. - - `updatedAt` is when this row's collection was last (re)computed - lets the serving pipe or an - operator sanity-check freshness independent of the COPY job's own execution log. - -SCHEMA > - `collectionSlug` String, - `id` String, - `displayName` String, - `githubHandleArray` Array(String), - `contributionCount` UInt64, - `contributionPercentage` Float64, - `roles` Array(String), - `contributionPercentageRunningTotal` Float64, - `rank` UInt32, - `totalContributorCount` UInt64, - `isBusFactorCutoff` UInt8, - `updatedAt` DateTime64(3) - -ENGINE ReplacingMergeTree -ENGINE_SORTING_KEY collectionSlug, rank -ENGINE_VER updatedAt diff --git a/services/libs/tinybird/pipes/collection_contributor_dependency.pipe b/services/libs/tinybird/pipes/collection_contributor_dependency.pipe deleted file mode 100644 index d8d9df6f71..0000000000 --- a/services/libs/tinybird/pipes/collection_contributor_dependency.pipe +++ /dev/null @@ -1,109 +0,0 @@ -DESCRIPTION > - - `collection_contributor_dependency.pipe` serves the contributor dependency widget for a - collection's Contributors tab - identifies contributors whose combined contributions reach or - exceed 51% of total collection activity (bus factor analysis), aggregated across every project - in the collection. - - Collection-scoped counterpart to `contributor_dependency.pipe` - separate because that pipe - references `contributors_leaderboard` (project-only) by name, and the collection-scale - replacement for that table is `collection_contributors_leaderboard` (a differently-shaped - pipe - see its own DESCRIPTION for why a naive collectionSlug branch on the single-project - leaderboard pipe isn't used there either). - - **Two paths, branched on whether any optional filter is present:** - - **Precomputed path (no `repos`/`startDate`/`endDate`/`platform`/`activity_type` and - `includeCollaborations` absent or `false`)** - this is what the real frontend widget always - requests on first paint, before a user touches a filter control. Reads - `collection_contributor_dependency_copy_ds`, a daily `TYPE COPY` materialization (see - `collection_contributor_dependency_copy.pipe`), instead of recomputing live. This exists - because the live computation - via `collection_contributors_leaderboard`'s - `collection_member_aggregates` node - must fully materialize a `GROUP BY af.memberId` over the - entire filtered activity set before it can ORDER BY/LIMIT (LIMIT only reduces output size, not - scan/aggregation cost), which at collection scale (e.g. `cncf` has 149,677 distinct - contributors) triggered a Tinybird production rate-limit/concurrency alert on `lfx_insights` - from timeouts under load. - - **Live path (any of those filters present)** - unavoidable, since filters are genuine dynamic - user input and the precomputed table only covers the unfiltered default. Unchanged from the - prior implementation: combines `collection_contributors_leaderboard` and `active_contributors` - (both already collectionSlug-aware). - - Precomputed path only supports `limit=100` (the only value the frontend ever sends for this - widget, and the only value the copy pipe materializes) - if `limit` is explicitly set to - anything else, falls through to the live path even with no other filters present. - - Parameters: - - `collectionSlug`: Required collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`. - - `repos`: Optional array of repository URLs for filtering (inherited from `segments_filtered_by_collection`). Presence routes to the live path. - - `startDate`: Optional DateTime filter for activities after timestamp (e.g., '2024-01-01 00:00:00'). Presence routes to the live path. - - `endDate`: Optional DateTime filter for activities before timestamp (e.g., '2024-12-31 23:59:59'). Presence routes to the live path. - - `platform`: Optional string filter for source platform (e.g., 'github', 'discord', 'slack'). Presence routes to the live path. - - `activity_type`: Optional string filter for single activity type (e.g., 'authored-commit'). Presence routes to the live path. - - `includeCollaborations`: Optional boolean, defaults to false - same toggle as the single-project pipe. `true` routes to the live path. - - `limit`: Optional integer for the number of top contributors to rank before finding the 51% - cutoff. Inherited by `collection_contributors_leaderboard` (this pipe has no default of its - own), whose own default is 10 if omitted - callers doing bus-factor analysis should pass - limit=100 explicitly (same convention as the single-project `contributor_dependency.pipe` / - `contributors_leaderboard.pipe` pair) to reliably reach the 51% cutoff. A value other than 100 - routes to the live path. - - Response: `id`, `displayName`, `githubHandleArray`, `contributionCount`, `contributionPercentage`, `roles`, `contributionPercentageRunningTotal`, `totalContributorCount` - -TAGS "Insights, Widget", "Collection", "Contributors" - -NODE collection_contributor_dependency_precomputed -DESCRIPTION > - Precomputed path: point lookup into the daily materialization, filtered to the rows the live - pipe's WHERE clause would itself return (`isBusFactorCutoff = 1`), ordered by rank. - -SQL > - % - SELECT - id, - displayName, - githubHandleArray, - contributionCount, - contributionPercentage, - roles, - contributionPercentageRunningTotal, - totalContributorCount - FROM collection_contributor_dependency_copy_ds - WHERE - collectionSlug - = {{ String(collectionSlug, description="Filter by collection slug", required=True) }} - AND isBusFactorCutoff = 1 - ORDER BY rank ASC - -NODE collection_contributions_percentage_running_total -SQL > - % - SELECT t.*, active_contributors.contributorCount as "totalContributorCount" - FROM - ( - SELECT - id, - displayName, - githubHandleArray, - contributionCount, - contributionPercentage, - roles, - sum(contributionPercentage) OVER ( - ORDER BY contributionPercentage DESC, id - ) AS contributionPercentageRunningTotal - FROM collection_contributors_leaderboard - ) t - cross join active_contributors - WHERE - contributionPercentageRunningTotal <= 51 - OR (contributionPercentageRunningTotal - contributionPercentage < 51) - ORDER BY contributionPercentageRunningTotal ASC - -NODE collection_contributor_dependency_result -DESCRIPTION > - Routes between the precomputed and live paths. Precomputed path applies only when no optional - filter is present and `limit` (if given) is exactly 100 - the exact shape the frontend always - requests on first paint. - -SQL > - % - {% if not defined(repos) and not defined(startDate) and not defined(endDate) and not defined( - platform - ) and not defined(activity_type) and not Boolean(includeCollaborations, false) and Int32( - limit, 100 - ) == 100 %} SELECT * FROM collection_contributor_dependency_precomputed - {% else %} SELECT * FROM collection_contributions_percentage_running_total - {% end %} diff --git a/services/libs/tinybird/pipes/collection_contributor_dependency_copy.pipe b/services/libs/tinybird/pipes/collection_contributor_dependency_copy.pipe deleted file mode 100644 index e46926316f..0000000000 --- a/services/libs/tinybird/pipes/collection_contributor_dependency_copy.pipe +++ /dev/null @@ -1,155 +0,0 @@ -DESCRIPTION > - - `collection_contributor_dependency_copy.pipe` precomputes the DEFAULT (unfiltered) bus-factor - result for every collection on a daily schedule, writing into `collection_contributor_dependency_copy_ds`. - - Exists because `collection_contributor_dependency.pipe`'s live computation - via - `collection_contributors_leaderboard`'s `collection_member_aggregates` node - must fully - materialize a `GROUP BY af.memberId` over the entire filtered activity set before it can - ORDER BY/LIMIT (LIMIT only reduces output size, not scan/aggregation cost). At collection scale - (e.g. `cncf` has 149,677 distinct contributors across 16.7M activities; some collections run far - larger - `my-collection` alone is ~211M activities) this triggered a Tinybird production - rate-limit/concurrency alert on `lfx_insights` from timeouts under load. - - Only covers the case the real frontend actually hits by default: `collectionSlug` with no - `repos`/`startDate`/`endDate`/`platform`/`activity_type` override and `includeCollaborations=false` - (the page's initial-load state before a user touches a filter control) - see - `collection_contributor_dependency.pipe`'s DESCRIPTION for how the serving pipe branches between - this precomputed path and the still-live filtered path. - - `limit=100` is hardcoded here to match the only value the frontend ever passes for this widget - - same convention as the live pipe's own DESCRIPTION notes for its `limit` parameter. - - Uses `LIMIT 100 BY collectionSlug` (ClickHouse's per-group top-N idiom) instead of a window - function over the full unbounded per-member result, and pre-filters `activities_filtered`'s - equivalent scan (`segmentId IN (...)`) to only segments that belong to at least one collection, - before fanning out to per-collection membership - keeps the join cheap relative to a naive - cross-collection materialization. Validated against live prod data for `cncf` (127 projects, - 149,677 contributors) and `agentgateway` (1 project, 468 contributors) - see deploy notes. - - Known, separate, and intentionally NOT fixed here: for very large collections the top-100 - ranked contributors structurally cannot reach the 51% cutoff (e.g. `cncf`'s top 100 only cover - ~20% of total activity) - this is a product-definition question about the `limit`/cutoff - interaction, out of scope for this performance fix. `isBusFactorCutoff` will legitimately be 0 - for every row of such a collection; the serving pipe must not assume at least one row per - collection has `isBusFactorCutoff = 1`. - -NODE collection_contributor_dependency_copy_segments -DESCRIPTION > - Every (collectionSlug, segmentId) pair for enabled projects that belong to at least one - collection - the fan-out point. A project with N collection memberships produces N rows here, - same semantics as `insights_projects_populated_ds.collectionsSlugs`. - -SQL > - SELECT arrayJoin(collectionsSlugs) AS collectionSlug, segmentId - FROM insights_projects_populated_ds - WHERE enabled = 1 AND notEmpty(collectionsSlugs) - -NODE collection_contributor_dependency_copy_activities -DESCRIPTION > - Code-contribution activities scoped to segments that belong to at least one collection - filtered - by segmentId first (mirrors `activities_filtered`'s default, unfiltered-date/repo/platform path) - so the collection fan-out join below only ever joins against an already-narrowed set. - -SQL > - SELECT a.activityId AS id, a.memberId AS memberId, a.segmentId AS segmentId - FROM activityRelations_deduplicated_cleaned_bucket_union a - WHERE - a.segmentId IN (SELECT segmentId FROM collection_contributor_dependency_copy_segments) - AND (a.type, a.platform) - IN (SELECT activityType, platform FROM activityTypes WHERE isCodeContribution = 1) - -NODE collection_contributor_dependency_copy_activities_by_collection -DESCRIPTION > - Fans activities out per collection membership via the segment join, then per-collection totals - and per-member top-100 ranked counts are computed from this in the following two nodes. - -SQL > - SELECT s.collectionSlug AS collectionSlug, a.memberId AS memberId, a.id AS id - FROM collection_contributor_dependency_copy_activities a - INNER JOIN collection_contributor_dependency_copy_segments s ON s.segmentId = a.segmentId - -NODE collection_contributor_dependency_copy_totals -DESCRIPTION > - Per-collection scalar totals: total contribution count (for percentage denominators) and total - unique contributor count (for `totalContributorCount`, matching `active_contributors`'s scalar - in the live pipe). - -SQL > - SELECT - collectionSlug, - count(id) AS totalContributions, - uniqIf(memberId, memberId != '') AS totalContributorCount - FROM collection_contributor_dependency_copy_activities_by_collection - GROUP BY collectionSlug - -NODE collection_contributor_dependency_copy_member_aggregates -DESCRIPTION > - Per-(collection, member) contribution counts, ranked and capped to the top 100 per collection via - `LIMIT 100 BY collectionSlug` - the ClickHouse per-group top-N idiom, avoiding a full - cross-collection window-function materialization. - -SQL > - SELECT collectionSlug, memberId, count(id) AS contributionCount - FROM collection_contributor_dependency_copy_activities_by_collection - GROUP BY collectionSlug, memberId - ORDER BY collectionSlug, contributionCount DESC, memberId DESC - LIMIT 100 BY collectionSlug - -NODE collection_contributor_dependency_copy_ranked -DESCRIPTION > - Adds contributionPercentage, a 1-indexed rank within each collection, and the running-total - percentage (ordered contributionPercentage DESC, id - same tie-break as the live pipe's window - function) needed for the 51% cutoff. - -SQL > - SELECT - ma.collectionSlug AS collectionSlug, - ma.memberId AS id, - ma.contributionCount AS contributionCount, - ROUND(ma.contributionCount * 100.0 / t.totalContributions, 2) AS contributionPercentage, - toUInt32( - row_number() OVER ( - PARTITION BY ma.collectionSlug ORDER BY ma.contributionCount DESC, ma.memberId DESC - ) - ) AS rank, - sum(ROUND(ma.contributionCount * 100.0 / t.totalContributions, 2)) OVER ( - PARTITION BY ma.collectionSlug - ORDER BY ROUND(ma.contributionCount * 100.0 / t.totalContributions, 2) DESC, ma.memberId - ) AS contributionPercentageRunningTotal, - t.totalContributorCount AS totalContributorCount - FROM collection_contributor_dependency_copy_member_aggregates ma - INNER JOIN collection_contributor_dependency_copy_totals t ON t.collectionSlug = ma.collectionSlug - -NODE collection_contributor_dependency_copy_result -DESCRIPTION > - Joins member profile/role data and flags the 51% bus-factor cutoff membership, matching the live - pipe's WHERE clause exactly: `runningTotal <= 51 OR (runningTotal - ownPercentage < 51)`. - -SQL > - SELECT - r.collectionSlug AS collectionSlug, - r.id AS id, - m.displayName AS displayName, - m.githubHandleArray AS githubHandleArray, - r.contributionCount AS contributionCount, - r.contributionPercentage AS contributionPercentage, - coalesce(mr.roles, []) AS roles, - r.contributionPercentageRunningTotal AS contributionPercentageRunningTotal, - r.rank AS rank, - r.totalContributorCount AS totalContributorCount, - if( - r.contributionPercentageRunningTotal <= 51 - OR (r.contributionPercentageRunningTotal - r.contributionPercentage < 51), - 1, - 0 - ) AS isBusFactorCutoff, - now64(3) AS updatedAt - FROM collection_contributor_dependency_copy_ranked r - INNER JOIN members_sorted m ON m.id = r.id - LEFT JOIN - ( - SELECT memberId, groupUniqArray(role) AS roles - FROM maintainers_roles_copy_ds - GROUP BY memberId - ) mr - ON mr.memberId = r.id - -TYPE COPY -TARGET_DATASOURCE collection_contributor_dependency_copy_ds -COPY_MODE replace -COPY_SCHEDULE 30 1 * * * diff --git a/services/libs/tinybird/pipes/collection_contributors_leaderboard.pipe b/services/libs/tinybird/pipes/collection_contributors_leaderboard.pipe deleted file mode 100644 index 71fd463003..0000000000 --- a/services/libs/tinybird/pipes/collection_contributors_leaderboard.pipe +++ /dev/null @@ -1,84 +0,0 @@ -DESCRIPTION > - - `collection_contributors_leaderboard.pipe` serves the contributors leaderboard widget for a - collection's Contributors tab, aggregated across every project in the collection. - - Separate from `contributors_leaderboard.pipe` (the single-project version) because a naive - collection-scoped port of that pipe's `ROUND(COUNT(af.id) * 100.0 / SUM(COUNT(af.id)) OVER (), 2)` - contributionPercentage forces ClickHouse to fully materialize the per-member GROUP BY (potentially - hundreds of thousands of distinct members at collection scale) before it can apply ORDER BY/LIMIT - - confirmed via direct timing: the equivalent single-project pipe took 7-16s (once timing out - entirely at Tinybird's 20s limit) on a 242-project collection, vs 2-3s for the count-only path - with no window function. This pipe computes the total contribution count as a separate scalar - subquery instead of a window function, avoiding that full-materialization cost. - - Supports both count mode (`count=true`) and data mode for pagination-friendly leaderboard display. - - Always shows `displayName` (not `publicName`) since a collection spans many projects with mixed - LF/non-LF status - there's no single project-type answer to substitute on, unlike the single-project - pipe's `is_request_from_non_lf_project` check. - - Primary use case: powering the collection-scoped contributor ranking widget on Collection detail - pages' Contributors tab. - - Parameters: - - `collectionSlug`: Required collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`. - - `repos`: Optional array of repository URLs for filtering (inherited from `segments_filtered_by_collection`) - - `startDate`: Optional DateTime filter for activities after timestamp (e.g., '2024-01-01 00:00:00') - - `endDate`: Optional DateTime filter for activities before timestamp (e.g., '2024-12-31 23:59:59') - - `platform`: Optional string filter for source platform (e.g., 'github', 'discord', 'slack') - - `activity_type`: Optional string filter for single activity type (e.g., 'authored-commit') - - `activity_types`: Optional array of activity types (e.g., ['authored-commit', 'co-authored-commit']) - - `includeCollaborations`: Optional boolean, defaults to false - same toggle as the single-project pipe. - - `count`: Optional boolean, when true returns contributor count instead of leaderboard data - - `limit`: Optional integer for result pagination, defaults to 10 - - `offset`: Optional integer for result pagination, defaults to 0 - - Response: - - Count mode (`count=true`): `count` (total number of contributors) - - Data mode (default): `id`, `avatar`, `displayName`, `githubHandleArray`, `contributionCount`, `contributionPercentage`, `roles` - -TAGS "Insights, Widget", "Collection", "Contributors" - -NODE collection_contributors_total -DESCRIPTION > - Total contribution count across the collection - a plain aggregate with no GROUP BY, so it's - cheap regardless of how many distinct members contributed. Computed once and reused as a scalar - for the percentage calculation instead of a SUM(...) OVER () window function. - -SQL > - % - SELECT count(af.id) AS total FROM activities_filtered af - -NODE collection_member_aggregates -DESCRIPTION > - Per-member contribution counts, ranked and paginated. No window function here - the percentage - is computed in the next node against the scalar total, so ORDER BY/LIMIT can apply directly - after this GROUP BY without waiting on a second full-set pass. - -SQL > - % - {% if Boolean(count, false) %} SELECT count(distinct af.memberId) FROM activities_filtered af - {% else %} - SELECT af.memberId, count(af.id) AS contributionCount - FROM activities_filtered af - GROUP BY af.memberId - ORDER BY contributionCount DESC, af.memberId DESC - LIMIT {{ Int32(limit, 10) }} - OFFSET {{ Int32(offset, 0) }} - {% end %} - -NODE collection_contributors_leaderboard_result -SQL > - % - {% if Boolean(count, false) %} - SELECT count(distinct af.memberId) AS count FROM activities_filtered af - {% else %} - SELECT - m.id, - m.avatar, - m.displayName, - m.githubHandleArray, - ma.contributionCount, - ROUND(ma.contributionCount * 100.0 / total.total, 2) AS contributionPercentage, - mr.roles - FROM members_sorted AS m any - INNER JOIN collection_member_aggregates ma ON ma.memberId = m.id - LEFT JOIN member_roles mr ON mr.memberId = m.id - CROSS JOIN collection_contributors_total total - WHERE m.id IN (SELECT memberId FROM collection_member_aggregates) - ORDER BY contributionCount DESC - {% end %} From 32cffb2efc25920f66319496194597f0f9394792 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 28 Jul 2026 17:51:01 +0100 Subject: [PATCH 03/10] feat: add sub-bucketing to collection snapshot backfill pipes (CM-1164) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds bucket_id/num_buckets params to all 10 activityRelations_collection_bucket_MV_snapshot_N pipes, mirroring pull_request_analysis_initial_snapshot.pipe's chunking pattern. Lets the on-demand historical backfill run in smaller sequential chunks instead of one long-running job, avoiding the copy-job quota/overload issues seen running the full backfill in one shot. Signed-off-by: Gašper Grom --- ...elations_collection_bucket_MV_snapshot_0.pipe | 16 ++++++++++++++++ ...elations_collection_bucket_MV_snapshot_1.pipe | 16 ++++++++++++++++ ...elations_collection_bucket_MV_snapshot_2.pipe | 16 ++++++++++++++++ ...elations_collection_bucket_MV_snapshot_3.pipe | 16 ++++++++++++++++ ...elations_collection_bucket_MV_snapshot_4.pipe | 16 ++++++++++++++++ ...elations_collection_bucket_MV_snapshot_5.pipe | 16 ++++++++++++++++ ...elations_collection_bucket_MV_snapshot_6.pipe | 16 ++++++++++++++++ ...elations_collection_bucket_MV_snapshot_7.pipe | 16 ++++++++++++++++ ...elations_collection_bucket_MV_snapshot_8.pipe | 16 ++++++++++++++++ ...elations_collection_bucket_MV_snapshot_9.pipe | 16 ++++++++++++++++ 10 files changed, 160 insertions(+) diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe index ad654d3d59..209e231050 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe @@ -5,6 +5,9 @@ DESCRIPTION > activityRelations_bucket_MV_snapshot_0.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_0.pipe) then keeps it current on new inserts. + Run with bucket_id (0..num_buckets-1) and num_buckets (default 5) to process a subset of + segments at a time, avoiding long-running/overloaded jobs on the full historical scan - + mirrors pull_request_analysis_initial_snapshot.pipe's chunking pattern. NODE curated_collection_slugs_snapshot_0 SQL > @@ -12,12 +15,25 @@ SQL > NODE collection_fanout_snapshot_0 SQL > + % SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug FROM activityRelations a LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId WHERE collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_0) AND cityHash64(collectionSlug) % 10 = 0 + {% if defined(bucket_id) %} + AND cityHash64(a.segmentId) + % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} + = {{ + UInt8( + bucket_id, + 0, + description="Bucket index to process (0 to num_buckets-1)", + required=False, + ) + }} + {% end %} TYPE COPY TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_0 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe index ea107b4e42..b8ddfd9d60 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe @@ -5,6 +5,9 @@ DESCRIPTION > activityRelations_bucket_MV_snapshot_1.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_1.pipe) then keeps it current on new inserts. + Run with bucket_id (0..num_buckets-1) and num_buckets (default 5) to process a subset of + segments at a time, avoiding long-running/overloaded jobs on the full historical scan - + mirrors pull_request_analysis_initial_snapshot.pipe's chunking pattern. NODE curated_collection_slugs_snapshot_1 SQL > @@ -12,12 +15,25 @@ SQL > NODE collection_fanout_snapshot_1 SQL > + % SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug FROM activityRelations a LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId WHERE collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_1) AND cityHash64(collectionSlug) % 10 = 1 + {% if defined(bucket_id) %} + AND cityHash64(a.segmentId) + % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} + = {{ + UInt8( + bucket_id, + 0, + description="Bucket index to process (0 to num_buckets-1)", + required=False, + ) + }} + {% end %} TYPE COPY TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_1 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe index 37715aeb70..881315413d 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe @@ -5,6 +5,9 @@ DESCRIPTION > activityRelations_bucket_MV_snapshot_2.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_2.pipe) then keeps it current on new inserts. + Run with bucket_id (0..num_buckets-1) and num_buckets (default 5) to process a subset of + segments at a time, avoiding long-running/overloaded jobs on the full historical scan - + mirrors pull_request_analysis_initial_snapshot.pipe's chunking pattern. NODE curated_collection_slugs_snapshot_2 SQL > @@ -12,12 +15,25 @@ SQL > NODE collection_fanout_snapshot_2 SQL > + % SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug FROM activityRelations a LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId WHERE collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_2) AND cityHash64(collectionSlug) % 10 = 2 + {% if defined(bucket_id) %} + AND cityHash64(a.segmentId) + % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} + = {{ + UInt8( + bucket_id, + 0, + description="Bucket index to process (0 to num_buckets-1)", + required=False, + ) + }} + {% end %} TYPE COPY TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_2 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe index 30d116ede9..1c1f283f7c 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe @@ -5,6 +5,9 @@ DESCRIPTION > activityRelations_bucket_MV_snapshot_3.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_3.pipe) then keeps it current on new inserts. + Run with bucket_id (0..num_buckets-1) and num_buckets (default 5) to process a subset of + segments at a time, avoiding long-running/overloaded jobs on the full historical scan - + mirrors pull_request_analysis_initial_snapshot.pipe's chunking pattern. NODE curated_collection_slugs_snapshot_3 SQL > @@ -12,12 +15,25 @@ SQL > NODE collection_fanout_snapshot_3 SQL > + % SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug FROM activityRelations a LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId WHERE collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_3) AND cityHash64(collectionSlug) % 10 = 3 + {% if defined(bucket_id) %} + AND cityHash64(a.segmentId) + % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} + = {{ + UInt8( + bucket_id, + 0, + description="Bucket index to process (0 to num_buckets-1)", + required=False, + ) + }} + {% end %} TYPE COPY TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_3 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe index ce55cc181a..86c2dc8a3d 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe @@ -5,6 +5,9 @@ DESCRIPTION > activityRelations_bucket_MV_snapshot_4.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_4.pipe) then keeps it current on new inserts. + Run with bucket_id (0..num_buckets-1) and num_buckets (default 5) to process a subset of + segments at a time, avoiding long-running/overloaded jobs on the full historical scan - + mirrors pull_request_analysis_initial_snapshot.pipe's chunking pattern. NODE curated_collection_slugs_snapshot_4 SQL > @@ -12,12 +15,25 @@ SQL > NODE collection_fanout_snapshot_4 SQL > + % SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug FROM activityRelations a LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId WHERE collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_4) AND cityHash64(collectionSlug) % 10 = 4 + {% if defined(bucket_id) %} + AND cityHash64(a.segmentId) + % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} + = {{ + UInt8( + bucket_id, + 0, + description="Bucket index to process (0 to num_buckets-1)", + required=False, + ) + }} + {% end %} TYPE COPY TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_4 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe index efe884cfbf..f0f82977a7 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe @@ -5,6 +5,9 @@ DESCRIPTION > activityRelations_bucket_MV_snapshot_5.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_5.pipe) then keeps it current on new inserts. + Run with bucket_id (0..num_buckets-1) and num_buckets (default 5) to process a subset of + segments at a time, avoiding long-running/overloaded jobs on the full historical scan - + mirrors pull_request_analysis_initial_snapshot.pipe's chunking pattern. NODE curated_collection_slugs_snapshot_5 SQL > @@ -12,12 +15,25 @@ SQL > NODE collection_fanout_snapshot_5 SQL > + % SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug FROM activityRelations a LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId WHERE collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_5) AND cityHash64(collectionSlug) % 10 = 5 + {% if defined(bucket_id) %} + AND cityHash64(a.segmentId) + % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} + = {{ + UInt8( + bucket_id, + 0, + description="Bucket index to process (0 to num_buckets-1)", + required=False, + ) + }} + {% end %} TYPE COPY TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_5 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe index f927170044..79e5ed44df 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe @@ -5,6 +5,9 @@ DESCRIPTION > activityRelations_bucket_MV_snapshot_6.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_6.pipe) then keeps it current on new inserts. + Run with bucket_id (0..num_buckets-1) and num_buckets (default 5) to process a subset of + segments at a time, avoiding long-running/overloaded jobs on the full historical scan - + mirrors pull_request_analysis_initial_snapshot.pipe's chunking pattern. NODE curated_collection_slugs_snapshot_6 SQL > @@ -12,12 +15,25 @@ SQL > NODE collection_fanout_snapshot_6 SQL > + % SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug FROM activityRelations a LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId WHERE collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_6) AND cityHash64(collectionSlug) % 10 = 6 + {% if defined(bucket_id) %} + AND cityHash64(a.segmentId) + % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} + = {{ + UInt8( + bucket_id, + 0, + description="Bucket index to process (0 to num_buckets-1)", + required=False, + ) + }} + {% end %} TYPE COPY TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_6 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe index 0cc8d5b40f..b0fee0b6cb 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe @@ -5,6 +5,9 @@ DESCRIPTION > activityRelations_bucket_MV_snapshot_7.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_7.pipe) then keeps it current on new inserts. + Run with bucket_id (0..num_buckets-1) and num_buckets (default 5) to process a subset of + segments at a time, avoiding long-running/overloaded jobs on the full historical scan - + mirrors pull_request_analysis_initial_snapshot.pipe's chunking pattern. NODE curated_collection_slugs_snapshot_7 SQL > @@ -12,12 +15,25 @@ SQL > NODE collection_fanout_snapshot_7 SQL > + % SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug FROM activityRelations a LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId WHERE collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_7) AND cityHash64(collectionSlug) % 10 = 7 + {% if defined(bucket_id) %} + AND cityHash64(a.segmentId) + % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} + = {{ + UInt8( + bucket_id, + 0, + description="Bucket index to process (0 to num_buckets-1)", + required=False, + ) + }} + {% end %} TYPE COPY TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_7 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe index 45d2d9759c..f3c2a35477 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe @@ -5,6 +5,9 @@ DESCRIPTION > activityRelations_bucket_MV_snapshot_8.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_8.pipe) then keeps it current on new inserts. + Run with bucket_id (0..num_buckets-1) and num_buckets (default 5) to process a subset of + segments at a time, avoiding long-running/overloaded jobs on the full historical scan - + mirrors pull_request_analysis_initial_snapshot.pipe's chunking pattern. NODE curated_collection_slugs_snapshot_8 SQL > @@ -12,12 +15,25 @@ SQL > NODE collection_fanout_snapshot_8 SQL > + % SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug FROM activityRelations a LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId WHERE collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_8) AND cityHash64(collectionSlug) % 10 = 8 + {% if defined(bucket_id) %} + AND cityHash64(a.segmentId) + % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} + = {{ + UInt8( + bucket_id, + 0, + description="Bucket index to process (0 to num_buckets-1)", + required=False, + ) + }} + {% end %} TYPE COPY TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_8 diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe index 807518aa79..b99f8bdbfa 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe @@ -5,6 +5,9 @@ DESCRIPTION > activityRelations_bucket_MV_snapshot_9.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_9.pipe) then keeps it current on new inserts. + Run with bucket_id (0..num_buckets-1) and num_buckets (default 5) to process a subset of + segments at a time, avoiding long-running/overloaded jobs on the full historical scan - + mirrors pull_request_analysis_initial_snapshot.pipe's chunking pattern. NODE curated_collection_slugs_snapshot_9 SQL > @@ -12,12 +15,25 @@ SQL > NODE collection_fanout_snapshot_9 SQL > + % SELECT a.*, arrayJoin(p.collectionsSlugs) AS collectionSlug FROM activityRelations a LEFT JOIN insights_projects_populated_ds p ON p.segmentId = a.segmentId WHERE collectionSlug IN (SELECT slug FROM curated_collection_slugs_snapshot_9) AND cityHash64(collectionSlug) % 10 = 9 + {% if defined(bucket_id) %} + AND cityHash64(a.segmentId) + % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} + = {{ + UInt8( + bucket_id, + 0, + description="Bucket index to process (0 to num_buckets-1)", + required=False, + ) + }} + {% end %} TYPE COPY TARGET_DATASOURCE activityRelations_collection_bucket_MV_ds_9 From 8a1c443b8eabc82c93d26fcc5377bb326fd7c270 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Wed, 29 Jul 2026 10:48:08 +0100 Subject: [PATCH 04/10] refactor: consolidate collection insights aggregates (CM-XXXX) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Replace 10 per-bucket datasources with 1 shared consolidated target - All copy pipes append with daily snapshotId (snapshot + TTL pattern) - TTL of 2 days auto-cleans old snapshots - Consume-side pipe filters to max(snapshotId) for latest data - No breaking change to endpoint shape Signed-off-by: Gašper Grom --- ..._insights_aggregate_bucket_0_ds.datasource | 16 +++++ ..._insights_aggregate_bucket_1_ds.datasource | 16 +++++ ..._insights_aggregate_bucket_2_ds.datasource | 16 +++++ ..._insights_aggregate_bucket_3_ds.datasource | 16 +++++ ..._insights_aggregate_bucket_4_ds.datasource | 16 +++++ ..._insights_aggregate_bucket_5_ds.datasource | 16 +++++ ..._insights_aggregate_bucket_6_ds.datasource | 16 +++++ ..._insights_aggregate_bucket_7_ds.datasource | 16 +++++ ..._insights_aggregate_bucket_8_ds.datasource | 16 +++++ ..._insights_aggregate_bucket_9_ds.datasource | 16 +++++ ...ights_aggregate_consolidated_ds.datasource | 23 +++++++ ...ights_aggregate_contributors_ds.datasource | 23 +++++++ ..._insights_aggregate_projects_ds.datasource | 19 ++++++ .../pipes/collection_insights_aggregate.pipe | 68 +++++++++---------- ...ection_insights_aggregate_copy_pipe_0.pipe | 35 ++++++++++ ...ection_insights_aggregate_copy_pipe_1.pipe | 35 ++++++++++ ...ection_insights_aggregate_copy_pipe_2.pipe | 35 ++++++++++ ...ection_insights_aggregate_copy_pipe_3.pipe | 35 ++++++++++ ...ection_insights_aggregate_copy_pipe_4.pipe | 35 ++++++++++ ...ection_insights_aggregate_copy_pipe_5.pipe | 35 ++++++++++ ...ection_insights_aggregate_copy_pipe_6.pipe | 35 ++++++++++ ...ection_insights_aggregate_copy_pipe_7.pipe | 35 ++++++++++ ...ection_insights_aggregate_copy_pipe_8.pipe | 35 ++++++++++ ...ection_insights_aggregate_copy_pipe_9.pipe | 35 ++++++++++ ...insights_aggregate_projects_copy_pipe.pipe | 33 +++++++++ 25 files changed, 639 insertions(+), 37 deletions(-) create mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_0_ds.datasource create mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_1_ds.datasource create mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_2_ds.datasource create mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_3_ds.datasource create mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_4_ds.datasource create mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_5_ds.datasource create mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_6_ds.datasource create mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_7_ds.datasource create mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_8_ds.datasource create mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_9_ds.datasource create mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_consolidated_ds.datasource create mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_contributors_ds.datasource create mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_projects_ds.datasource create mode 100644 services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_0.pipe create mode 100644 services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_1.pipe create mode 100644 services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_2.pipe create mode 100644 services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_3.pipe create mode 100644 services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_4.pipe create mode 100644 services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_5.pipe create mode 100644 services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_6.pipe create mode 100644 services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_7.pipe create mode 100644 services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_8.pipe create mode 100644 services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_9.pipe create mode 100644 services/libs/tinybird/pipes/collection_insights_aggregate_projects_copy_pipe.pipe diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_0_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_0_ds.datasource new file mode 100644 index 0000000000..3f41f76f8b --- /dev/null +++ b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_0_ds.datasource @@ -0,0 +1,16 @@ +DESCRIPTION > + - Per-bucket contributor aggregate — bucket 0 only. + - Built daily by `collection_insights_aggregate_copy_pipe_0.pipe`. + - One row per collection that has activity rows in bucket 0. + - Columns: + - `collectionSlug`: Collection identifier (String). + - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). + - `updatedAt`: Update timestamp (DateTime). + +SCHEMA > + `collectionSlug` String, + `uniqueContributorCount` UInt64, + `updatedAt` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_1_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_1_ds.datasource new file mode 100644 index 0000000000..ce1291833a --- /dev/null +++ b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_1_ds.datasource @@ -0,0 +1,16 @@ +DESCRIPTION > + - Per-bucket contributor aggregate — bucket 1 only. + - Built daily by `collection_insights_aggregate_copy_pipe_1.pipe`. + - One row per collection that has activity rows in bucket 1. + - Columns: + - `collectionSlug`: Collection identifier (String). + - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). + - `updatedAt`: Update timestamp (DateTime). + +SCHEMA > + `collectionSlug` String, + `uniqueContributorCount` UInt64, + `updatedAt` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_2_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_2_ds.datasource new file mode 100644 index 0000000000..41eb740103 --- /dev/null +++ b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_2_ds.datasource @@ -0,0 +1,16 @@ +DESCRIPTION > + - Per-bucket contributor aggregate — bucket 2 only. + - Built daily by `collection_insights_aggregate_copy_pipe_2.pipe`. + - One row per collection that has activity rows in bucket 2. + - Columns: + - `collectionSlug`: Collection identifier (String). + - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). + - `updatedAt`: Update timestamp (DateTime). + +SCHEMA > + `collectionSlug` String, + `uniqueContributorCount` UInt64, + `updatedAt` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_3_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_3_ds.datasource new file mode 100644 index 0000000000..d934c77759 --- /dev/null +++ b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_3_ds.datasource @@ -0,0 +1,16 @@ +DESCRIPTION > + - Per-bucket contributor aggregate — bucket 3 only. + - Built daily by `collection_insights_aggregate_copy_pipe_3.pipe`. + - One row per collection that has activity rows in bucket 3. + - Columns: + - `collectionSlug`: Collection identifier (String). + - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). + - `updatedAt`: Update timestamp (DateTime). + +SCHEMA > + `collectionSlug` String, + `uniqueContributorCount` UInt64, + `updatedAt` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_4_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_4_ds.datasource new file mode 100644 index 0000000000..6ce8b3a1c3 --- /dev/null +++ b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_4_ds.datasource @@ -0,0 +1,16 @@ +DESCRIPTION > + - Per-bucket contributor aggregate — bucket 4 only. + - Built daily by `collection_insights_aggregate_copy_pipe_4.pipe`. + - One row per collection that has activity rows in bucket 4. + - Columns: + - `collectionSlug`: Collection identifier (String). + - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). + - `updatedAt`: Update timestamp (DateTime). + +SCHEMA > + `collectionSlug` String, + `uniqueContributorCount` UInt64, + `updatedAt` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_5_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_5_ds.datasource new file mode 100644 index 0000000000..5da48ccad4 --- /dev/null +++ b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_5_ds.datasource @@ -0,0 +1,16 @@ +DESCRIPTION > + - Per-bucket contributor aggregate — bucket 5 only. + - Built daily by `collection_insights_aggregate_copy_pipe_5.pipe`. + - One row per collection that has activity rows in bucket 5. + - Columns: + - `collectionSlug`: Collection identifier (String). + - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). + - `updatedAt`: Update timestamp (DateTime). + +SCHEMA > + `collectionSlug` String, + `uniqueContributorCount` UInt64, + `updatedAt` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_6_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_6_ds.datasource new file mode 100644 index 0000000000..2bdb2b9654 --- /dev/null +++ b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_6_ds.datasource @@ -0,0 +1,16 @@ +DESCRIPTION > + - Per-bucket contributor aggregate — bucket 6 only. + - Built daily by `collection_insights_aggregate_copy_pipe_6.pipe`. + - One row per collection that has activity rows in bucket 6. + - Columns: + - `collectionSlug`: Collection identifier (String). + - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). + - `updatedAt`: Update timestamp (DateTime). + +SCHEMA > + `collectionSlug` String, + `uniqueContributorCount` UInt64, + `updatedAt` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_7_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_7_ds.datasource new file mode 100644 index 0000000000..4742fe57d9 --- /dev/null +++ b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_7_ds.datasource @@ -0,0 +1,16 @@ +DESCRIPTION > + - Per-bucket contributor aggregate — bucket 7 only. + - Built daily by `collection_insights_aggregate_copy_pipe_7.pipe`. + - One row per collection that has activity rows in bucket 7. + - Columns: + - `collectionSlug`: Collection identifier (String). + - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). + - `updatedAt`: Update timestamp (DateTime). + +SCHEMA > + `collectionSlug` String, + `uniqueContributorCount` UInt64, + `updatedAt` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_8_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_8_ds.datasource new file mode 100644 index 0000000000..f387f9e434 --- /dev/null +++ b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_8_ds.datasource @@ -0,0 +1,16 @@ +DESCRIPTION > + - Per-bucket contributor aggregate — bucket 8 only. + - Built daily by `collection_insights_aggregate_copy_pipe_8.pipe`. + - One row per collection that has activity rows in bucket 8. + - Columns: + - `collectionSlug`: Collection identifier (String). + - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). + - `updatedAt`: Update timestamp (DateTime). + +SCHEMA > + `collectionSlug` String, + `uniqueContributorCount` UInt64, + `updatedAt` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_9_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_9_ds.datasource new file mode 100644 index 0000000000..c893ad2332 --- /dev/null +++ b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_9_ds.datasource @@ -0,0 +1,16 @@ +DESCRIPTION > + - Per-bucket contributor aggregate — bucket 9 only. + - Built daily by `collection_insights_aggregate_copy_pipe_9.pipe`. + - One row per collection that has activity rows in bucket 9. + - Columns: + - `collectionSlug`: Collection identifier (String). + - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). + - `updatedAt`: Update timestamp (DateTime). + +SCHEMA > + `collectionSlug` String, + `uniqueContributorCount` UInt64, + `updatedAt` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_consolidated_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_consolidated_ds.datasource new file mode 100644 index 0000000000..ff3e17ef4e --- /dev/null +++ b/services/libs/tinybird/datasources/collection_insights_aggregate_consolidated_ds.datasource @@ -0,0 +1,23 @@ +DESCRIPTION > + - Consolidated contributors aggregate table — one row per (collectionSlug, snapshotId) pair. + - Built by 10 separate copy pipes (collection_insights_aggregate_copy_pipe_N.pipe for N=0..9), + each scanning only its own bucket table to bound read volume and avoid massive union/join timeouts. + - Each copy pipe (bucket N) appends its rows to this shared table with the same snapshotId timestamp. + Bucket N's enrich finishes ~2:00-2:05 UTC, aggregate copy runs 2:05+5N UTC (staggered). + - Rows are tagged with snapshotId for versioning; queries filter to max(snapshotId) for latest data. + - TTL of 2 days cleans up old snapshots automatically (argMax-filtering on snapshotId is the + correctness mechanism; TTL is just to prevent unbounded growth). + - Columns: + - `collectionSlug`: The collection identifier (String). + - `uniqueContributorCount`: Unique contributors in this collection across all bucket scans (UInt64). + - `snapshotId`: Timestamp marking when this snapshot was created (daily granularity) (DateTime). + +SCHEMA > + `collectionSlug` String, + `uniqueContributorCount` UInt64, + `snapshotId` DateTime + +ENGINE "MergeTree" +ENGINE_PARTITION_KEY "toYear(snapshotId)" +ENGINE_SORTING_KEY "collectionSlug, snapshotId" +ENGINE_TTL "toDateTime(snapshotId) + toIntervalDay(2)" diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_contributors_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_contributors_ds.datasource new file mode 100644 index 0000000000..ea07d220db --- /dev/null +++ b/services/libs/tinybird/datasources/collection_insights_aggregate_contributors_ds.datasource @@ -0,0 +1,23 @@ +DESCRIPTION > + - Consolidated contributors aggregate table — one row per (collectionSlug, snapshotId) pair. + - Built by 10 separate copy pipes (collection_insights_aggregate_copy_pipe_N.pipe for N=0..9), + each scanning only its own bucket table to bound read volume and avoid massive union/join timeouts. + - Each copy pipe (bucket N) appends its rows to this shared table with the same snapshotId timestamp. + Bucket N's enrich finishes ~2:00-2:05 UTC, aggregate copy runs 2:05+5N UTC (staggered). + - Rows are tagged with snapshotId for versioning; queries filter to max(snapshotId) for latest data. + - TTL of 2 days cleans up old snapshots automatically (argMax-filtering on snapshotId is the + correctness mechanism; TTL is just to prevent unbounded growth). + - Columns: + - `collectionSlug`: The collection identifier (String). + - `uniqueContributorCount`: Unique contributors in this collection across all bucket scans (UInt64). + - `snapshotId`: Timestamp marking when this snapshot was created (daily granularity) (DateTime). + +SCHEMA > + `collectionSlug` String, + `uniqueContributorCount` UInt64, + `snapshotId` DateTime + +ENGINE MergeTree +ENGINE_PARTITION_KEY toYear(snapshotId) +ENGINE_SORTING_KEY collectionSlug, snapshotId +ENGINE_TTL toDateTime(snapshotId) + toIntervalDay(2) diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_projects_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_projects_ds.datasource new file mode 100644 index 0000000000..eb879832b6 --- /dev/null +++ b/services/libs/tinybird/datasources/collection_insights_aggregate_projects_ds.datasource @@ -0,0 +1,19 @@ +DESCRIPTION > + - Per-collection projects aggregate table — small, fast table with project counts and health scores. + - Built once daily by a single copy pipe (collection_insights_aggregate_projects_copy_pipe.pipe) + that scans only insights_projects_populated_ds (no activity tables, no joins to billions of rows). + - Runs daily at a fixed time after the contributor aggregates are ready. + - One row per collection slug, with: + - `collectionSlug`: The collection identifier (String). + - `projectCount`: Total number of onboarded projects in the collection (UInt64). + - `avgHealthScore`: Average health score across onboarded projects, rounded (Nullable Float64). + - `updatedAt`: Timestamp of the last update (DateTime). + +SCHEMA > + `collectionSlug` String, + `projectCount` UInt64, + `avgHealthScore` Nullable(Float64), + `updatedAt` DateTime + +ENGINE MergeTree +ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate.pipe index 0edaa91cf0..9b57b4fc84 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate.pipe @@ -1,11 +1,13 @@ DESCRIPTION > - `collection_insights_aggregate.pipe` serves aggregate insights metrics for a collection. - Returns a single row with metrics aggregated across every project in the collection, not one row per project. + - Now a lightweight join of two precomputed tables: + - `collection_insights_aggregate_contributors_ds`: consolidated unique contributor counts (1 row per collection per snapshot) + - `collection_insights_aggregate_projects_ds`: project count and average health score (1 row per collection) + - Updated daily: contributor aggregates run 2:05-2:50 UTC (staggered per-bucket, all append to shared table), projects aggregate runs 3:00 UTC (after all contributors finish). + - Filters to latest snapshotId for current data (argMax-per-collection-slug pattern). - Parameters: - - `collectionSlug`: Required collection slug. insights_projects_populated_ds.collectionsSlugs already lists - every collection each project belongs to (see insightsProjects_filtered.pipe for the same filter pattern), - so this pipe resolves collection membership directly rather than requiring the caller to first look up a - project id list. + - `collectionSlug`: Required collection slug. Joins both tables on this slug. - Response: - `projectCount`: count of distinct projects in the collection. - `uniqueContributorCount`: total unique contributors across all projects in the collection, deduplicated (a contributor active on multiple projects in the collection is counted once). @@ -15,44 +17,36 @@ DESCRIPTION > check) — projects with no attributed contributors/organizations have no meaningful health signal yet and would otherwise pull the average down artificially. -TAGS "Insights, Widget", "Collection" -NODE collection_insights_aggregate_projects -DESCRIPTION > - Resolves the collection's member projects to their segment ids and health scores directly via - collectionsSlugs, matching the same has(collectionsSlugs, ...) filter already used by - insightsProjects_filtered.pipe. +TOKEN "insights-app-token" READ -SQL > - % - SELECT id, segmentId, healthScore, organizationCount, contributorCount - FROM insights_projects_populated_ds - WHERE - enabled = 1 - AND has( - collectionsSlugs, - {{ String(collectionSlug, description="Filter by collection slug", required=True) }} - ) +TAGS "Insights, Widget", "Collection" NODE collection_insights_aggregate_results DESCRIPTION > - Aggregates health score across matched projects and counts unique contributors across their segments in one pass + Lightweight join of precomputed contributors and projects tables for the requested collection slug. + Filters contributors to latest snapshot using max(snapshotId) and retrieves project metrics. + Collections are bucketed deterministically (cityHash64(slug) % 10) for per-bucket read efficiency; + all buckets' output is consolidated into a single target datasource. SQL > + +% SELECT - (SELECT count(distinct id) FROM collection_insights_aggregate_projects) AS projectCount, - uniq(ar.memberId) AS uniqueContributorCount, - ( - SELECT round(avg(healthScore)) - FROM collection_insights_aggregate_projects - WHERE NOT (organizationCount = 0 AND contributorCount = 0) - ) AS avgHealthScore - FROM activityRelations_deduplicated_cleaned_bucket_union AS ar - WHERE - ar.memberId != '' - AND ar.segmentId IN (SELECT segmentId FROM collection_insights_aggregate_projects) - AND (ar.type, ar.platform) IN ( - SELECT activityType, platform - FROM activityTypes - WHERE isCodeContribution = 1 OR isCollaboration = 1 - ) + p.projectCount, + c.uniqueContributorCount, + p.avgHealthScore + FROM collection_insights_aggregate_projects_ds p + LEFT JOIN ( + SELECT + collectionSlug, + uniqueContributorCount + FROM collection_insights_aggregate_consolidated_ds + WHERE snapshotId = (SELECT max(snapshotId) FROM collection_insights_aggregate_consolidated_ds) + ) c + ON p.collectionSlug = c.collectionSlug + WHERE p.collectionSlug = {{ String(collectionSlug, description="Filter by collection slug", required=True) }} + +TYPE endpoint + + diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_0.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_0.pipe new file mode 100644 index 0000000000..98a225d506 --- /dev/null +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_0.pipe @@ -0,0 +1,35 @@ +DESCRIPTION > + - Per-bucket contributor aggregation copy pipe for bucket 0. + - Scans ONLY activityRelations_collection_deduplicated_cleaned_bucket_0_ds (no 10-way union). + - Computes uniqueContributorCount per collection slug for all collections that route to bucket 0. + - Runs after bucket 0's enrich job completes (1:00 UTC + ~20min runtime typically finishes by 1:20 UTC). + - Scheduled at 2:05 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + +TAGS "Insights, Widget", "Collection" + +NODE collection_insights_aggregate_bucket_0_contributors +DESCRIPTION > + Aggregates unique contributor counts per collection from bucket 0's activity relations. + Matches the activity type filtering (code contributions + collaboration) from the original logic. + Tags each row with snapshotId (start of current day) for versioning. + +SQL > + SELECT + ar.collectionSlug, + uniq(ar.memberId) as uniqueContributorCount, + toStartOfInterval(now(), INTERVAL 1 day) as snapshotId + FROM activityRelations_collection_deduplicated_cleaned_bucket_0_ds ar + WHERE + ar.memberId != '' + AND (ar.type, ar.platform) IN ( + SELECT activityType, platform + FROM activityTypes + WHERE isCodeContribution = 1 OR isCollaboration = 1 + ) + GROUP BY ar.collectionSlug + +TYPE COPY +TARGET_DATASOURCE collection_insights_aggregate_consolidated_ds +COPY_MODE append +COPY_SCHEDULE 5 2 * * * diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_1.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_1.pipe new file mode 100644 index 0000000000..5f2248e514 --- /dev/null +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_1.pipe @@ -0,0 +1,35 @@ +DESCRIPTION > + - Per-bucket contributor aggregation copy pipe for bucket 1. + - Scans ONLY activityRelations_collection_deduplicated_cleaned_bucket_1_ds (no 10-way union). + - Computes uniqueContributorCount per collection slug for all collections that route to bucket 1. + - Runs after bucket 1's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). + - Scheduled at 2:10 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + +TAGS "Insights, Widget", "Collection" + +NODE collection_insights_aggregate_bucket_1_contributors +DESCRIPTION > + Aggregates unique contributor counts per collection from bucket 1's activity relations. + Matches the activity type filtering (code contributions + collaboration) from the original logic. + Tags each row with snapshotId (start of current day) for versioning. + +SQL > + SELECT + ar.collectionSlug, + uniq(ar.memberId) as uniqueContributorCount, + toStartOfInterval(now(), INTERVAL 1 day) as snapshotId + FROM activityRelations_collection_deduplicated_cleaned_bucket_1_ds ar + WHERE + ar.memberId != '' + AND (ar.type, ar.platform) IN ( + SELECT activityType, platform + FROM activityTypes + WHERE isCodeContribution = 1 OR isCollaboration = 1 + ) + GROUP BY ar.collectionSlug + +TYPE COPY +TARGET_DATASOURCE collection_insights_aggregate_consolidated_ds +COPY_MODE append +COPY_SCHEDULE 10 2 * * * diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_2.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_2.pipe new file mode 100644 index 0000000000..ce159e3e4d --- /dev/null +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_2.pipe @@ -0,0 +1,35 @@ +DESCRIPTION > + - Per-bucket contributor aggregation copy pipe for bucket 2. + - Scans ONLY activityRelations_collection_deduplicated_cleaned_bucket_2_ds (no 10-way union). + - Computes uniqueContributorCount per collection slug for all collections that route to bucket 2. + - Runs after bucket 2's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). + - Scheduled at 2:15 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + +TAGS "Insights, Widget", "Collection" + +NODE collection_insights_aggregate_bucket_2_contributors +DESCRIPTION > + Aggregates unique contributor counts per collection from bucket 2's activity relations. + Matches the activity type filtering (code contributions + collaboration) from the original logic. + Tags each row with snapshotId (start of current day) for versioning. + +SQL > + SELECT + ar.collectionSlug, + uniq(ar.memberId) as uniqueContributorCount, + toStartOfInterval(now(), INTERVAL 1 day) as snapshotId + FROM activityRelations_collection_deduplicated_cleaned_bucket_2_ds ar + WHERE + ar.memberId != '' + AND (ar.type, ar.platform) IN ( + SELECT activityType, platform + FROM activityTypes + WHERE isCodeContribution = 1 OR isCollaboration = 1 + ) + GROUP BY ar.collectionSlug + +TYPE COPY +TARGET_DATASOURCE collection_insights_aggregate_consolidated_ds +COPY_MODE append +COPY_SCHEDULE 15 2 * * * diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_3.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_3.pipe new file mode 100644 index 0000000000..b07797fdd2 --- /dev/null +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_3.pipe @@ -0,0 +1,35 @@ +DESCRIPTION > + - Per-bucket contributor aggregation copy pipe for bucket 3. + - Scans ONLY activityRelations_collection_deduplicated_cleaned_bucket_3_ds (no 10-way union). + - Computes uniqueContributorCount per collection slug for all collections that route to bucket 3. + - Runs after bucket 3's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). + - Scheduled at 2:20 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + +TAGS "Insights, Widget", "Collection" + +NODE collection_insights_aggregate_bucket_3_contributors +DESCRIPTION > + Aggregates unique contributor counts per collection from bucket 3's activity relations. + Matches the activity type filtering (code contributions + collaboration) from the original logic. + Tags each row with snapshotId (start of current day) for versioning. + +SQL > + SELECT + ar.collectionSlug, + uniq(ar.memberId) as uniqueContributorCount, + toStartOfInterval(now(), INTERVAL 1 day) as snapshotId + FROM activityRelations_collection_deduplicated_cleaned_bucket_3_ds ar + WHERE + ar.memberId != '' + AND (ar.type, ar.platform) IN ( + SELECT activityType, platform + FROM activityTypes + WHERE isCodeContribution = 1 OR isCollaboration = 1 + ) + GROUP BY ar.collectionSlug + +TYPE COPY +TARGET_DATASOURCE collection_insights_aggregate_consolidated_ds +COPY_MODE append +COPY_SCHEDULE 20 2 * * * diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_4.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_4.pipe new file mode 100644 index 0000000000..ae44ca1ad4 --- /dev/null +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_4.pipe @@ -0,0 +1,35 @@ +DESCRIPTION > + - Per-bucket contributor aggregation copy pipe for bucket 4. + - Scans ONLY activityRelations_collection_deduplicated_cleaned_bucket_4_ds (no 10-way union). + - Computes uniqueContributorCount per collection slug for all collections that route to bucket 4. + - Runs after bucket 4's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). + - Scheduled at 2:25 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + +TAGS "Insights, Widget", "Collection" + +NODE collection_insights_aggregate_bucket_4_contributors +DESCRIPTION > + Aggregates unique contributor counts per collection from bucket 4's activity relations. + Matches the activity type filtering (code contributions + collaboration) from the original logic. + Tags each row with snapshotId (start of current day) for versioning. + +SQL > + SELECT + ar.collectionSlug, + uniq(ar.memberId) as uniqueContributorCount, + toStartOfInterval(now(), INTERVAL 1 day) as snapshotId + FROM activityRelations_collection_deduplicated_cleaned_bucket_4_ds ar + WHERE + ar.memberId != '' + AND (ar.type, ar.platform) IN ( + SELECT activityType, platform + FROM activityTypes + WHERE isCodeContribution = 1 OR isCollaboration = 1 + ) + GROUP BY ar.collectionSlug + +TYPE COPY +TARGET_DATASOURCE collection_insights_aggregate_consolidated_ds +COPY_MODE append +COPY_SCHEDULE 25 2 * * * diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_5.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_5.pipe new file mode 100644 index 0000000000..b3bb3864e0 --- /dev/null +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_5.pipe @@ -0,0 +1,35 @@ +DESCRIPTION > + - Per-bucket contributor aggregation copy pipe for bucket 5. + - Scans ONLY activityRelations_collection_deduplicated_cleaned_bucket_5_ds (no 10-way union). + - Computes uniqueContributorCount per collection slug for all collections that route to bucket 5. + - Runs after bucket 5's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). + - Scheduled at 2:30 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + +TAGS "Insights, Widget", "Collection" + +NODE collection_insights_aggregate_bucket_5_contributors +DESCRIPTION > + Aggregates unique contributor counts per collection from bucket 5's activity relations. + Matches the activity type filtering (code contributions + collaboration) from the original logic. + Tags each row with snapshotId (start of current day) for versioning. + +SQL > + SELECT + ar.collectionSlug, + uniq(ar.memberId) as uniqueContributorCount, + toStartOfInterval(now(), INTERVAL 1 day) as snapshotId + FROM activityRelations_collection_deduplicated_cleaned_bucket_5_ds ar + WHERE + ar.memberId != '' + AND (ar.type, ar.platform) IN ( + SELECT activityType, platform + FROM activityTypes + WHERE isCodeContribution = 1 OR isCollaboration = 1 + ) + GROUP BY ar.collectionSlug + +TYPE COPY +TARGET_DATASOURCE collection_insights_aggregate_consolidated_ds +COPY_MODE append +COPY_SCHEDULE 30 2 * * * diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_6.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_6.pipe new file mode 100644 index 0000000000..eab928287e --- /dev/null +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_6.pipe @@ -0,0 +1,35 @@ +DESCRIPTION > + - Per-bucket contributor aggregation copy pipe for bucket 6. + - Scans ONLY activityRelations_collection_deduplicated_cleaned_bucket_6_ds (no 10-way union). + - Computes uniqueContributorCount per collection slug for all collections that route to bucket 6. + - Runs after bucket 6's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). + - Scheduled at 2:35 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + +TAGS "Insights, Widget", "Collection" + +NODE collection_insights_aggregate_bucket_6_contributors +DESCRIPTION > + Aggregates unique contributor counts per collection from bucket 6's activity relations. + Matches the activity type filtering (code contributions + collaboration) from the original logic. + Tags each row with snapshotId (start of current day) for versioning. + +SQL > + SELECT + ar.collectionSlug, + uniq(ar.memberId) as uniqueContributorCount, + toStartOfInterval(now(), INTERVAL 1 day) as snapshotId + FROM activityRelations_collection_deduplicated_cleaned_bucket_6_ds ar + WHERE + ar.memberId != '' + AND (ar.type, ar.platform) IN ( + SELECT activityType, platform + FROM activityTypes + WHERE isCodeContribution = 1 OR isCollaboration = 1 + ) + GROUP BY ar.collectionSlug + +TYPE COPY +TARGET_DATASOURCE collection_insights_aggregate_consolidated_ds +COPY_MODE append +COPY_SCHEDULE 35 2 * * * diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_7.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_7.pipe new file mode 100644 index 0000000000..f3976ccfe2 --- /dev/null +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_7.pipe @@ -0,0 +1,35 @@ +DESCRIPTION > + - Per-bucket contributor aggregation copy pipe for bucket 7. + - Scans ONLY activityRelations_collection_deduplicated_cleaned_bucket_7_ds (no 10-way union). + - Computes uniqueContributorCount per collection slug for all collections that route to bucket 7. + - Runs after bucket 7's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). + - Scheduled at 2:40 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + +TAGS "Insights, Widget", "Collection" + +NODE collection_insights_aggregate_bucket_7_contributors +DESCRIPTION > + Aggregates unique contributor counts per collection from bucket 7's activity relations. + Matches the activity type filtering (code contributions + collaboration) from the original logic. + Tags each row with snapshotId (start of current day) for versioning. + +SQL > + SELECT + ar.collectionSlug, + uniq(ar.memberId) as uniqueContributorCount, + toStartOfInterval(now(), INTERVAL 1 day) as snapshotId + FROM activityRelations_collection_deduplicated_cleaned_bucket_7_ds ar + WHERE + ar.memberId != '' + AND (ar.type, ar.platform) IN ( + SELECT activityType, platform + FROM activityTypes + WHERE isCodeContribution = 1 OR isCollaboration = 1 + ) + GROUP BY ar.collectionSlug + +TYPE COPY +TARGET_DATASOURCE collection_insights_aggregate_consolidated_ds +COPY_MODE append +COPY_SCHEDULE 40 2 * * * diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_8.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_8.pipe new file mode 100644 index 0000000000..c2ed97ba52 --- /dev/null +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_8.pipe @@ -0,0 +1,35 @@ +DESCRIPTION > + - Per-bucket contributor aggregation copy pipe for bucket 8. + - Scans ONLY activityRelations_collection_deduplicated_cleaned_bucket_8_ds (no 10-way union). + - Computes uniqueContributorCount per collection slug for all collections that route to bucket 8. + - Runs after bucket 8's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). + - Scheduled at 2:45 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + +TAGS "Insights, Widget", "Collection" + +NODE collection_insights_aggregate_bucket_8_contributors +DESCRIPTION > + Aggregates unique contributor counts per collection from bucket 8's activity relations. + Matches the activity type filtering (code contributions + collaboration) from the original logic. + Tags each row with snapshotId (start of current day) for versioning. + +SQL > + SELECT + ar.collectionSlug, + uniq(ar.memberId) as uniqueContributorCount, + toStartOfInterval(now(), INTERVAL 1 day) as snapshotId + FROM activityRelations_collection_deduplicated_cleaned_bucket_8_ds ar + WHERE + ar.memberId != '' + AND (ar.type, ar.platform) IN ( + SELECT activityType, platform + FROM activityTypes + WHERE isCodeContribution = 1 OR isCollaboration = 1 + ) + GROUP BY ar.collectionSlug + +TYPE COPY +TARGET_DATASOURCE collection_insights_aggregate_consolidated_ds +COPY_MODE append +COPY_SCHEDULE 45 2 * * * diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_9.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_9.pipe new file mode 100644 index 0000000000..f32df84543 --- /dev/null +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_9.pipe @@ -0,0 +1,35 @@ +DESCRIPTION > + - Per-bucket contributor aggregation copy pipe for bucket 9. + - Scans ONLY activityRelations_collection_deduplicated_cleaned_bucket_9_ds (no 10-way union). + - Computes uniqueContributorCount per collection slug for all collections that route to bucket 9. + - Runs after bucket 9's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). + - Scheduled at 2:50 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + +TAGS "Insights, Widget", "Collection" + +NODE collection_insights_aggregate_bucket_9_contributors +DESCRIPTION > + Aggregates unique contributor counts per collection from bucket 9's activity relations. + Matches the activity type filtering (code contributions + collaboration) from the original logic. + Tags each row with snapshotId (start of current day) for versioning. + +SQL > + SELECT + ar.collectionSlug, + uniq(ar.memberId) as uniqueContributorCount, + toStartOfInterval(now(), INTERVAL 1 day) as snapshotId + FROM activityRelations_collection_deduplicated_cleaned_bucket_9_ds ar + WHERE + ar.memberId != '' + AND (ar.type, ar.platform) IN ( + SELECT activityType, platform + FROM activityTypes + WHERE isCodeContribution = 1 OR isCollaboration = 1 + ) + GROUP BY ar.collectionSlug + +TYPE COPY +TARGET_DATASOURCE collection_insights_aggregate_consolidated_ds +COPY_MODE append +COPY_SCHEDULE 50 2 * * * diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_projects_copy_pipe.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_projects_copy_pipe.pipe new file mode 100644 index 0000000000..02ce943e50 --- /dev/null +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_projects_copy_pipe.pipe @@ -0,0 +1,33 @@ +DESCRIPTION > + - Fast, small copy pipe for project-count and average health-score aggregation. + - Scans ONLY insights_projects_populated_ds (no activity tables, no massive joins). + - Produces projectCount and avgHealthScore per collection slug. + - Runs after all per-bucket contributor copy pipes have completed (all run in 2:05-2:50 window). + - Scheduled at 3:00 UTC to clear after all 10 bucket pipes finish, giving a 10-min safety margin. + - COPY_MODE replace: one row per collection per copy run (idempotent daily rebuild). + +TAGS "Insights, Widget", "Collection" + +NODE collection_insights_aggregate_projects_aggregated +DESCRIPTION > + Aggregates project counts and health scores per collection from insights_projects_populated_ds. + Uses same "onboarded" definition as original logic: projects with organizationCount > 0 OR contributorCount > 0. + +SQL > + SELECT + slug as collectionSlug, + count(distinct id) as projectCount, + round(avg(if(NOT ((organizationCount = 0) AND (contributorCount = 0)), healthScore, NULL))) as avgHealthScore, + now() as updatedAt + FROM ( + SELECT id, slug, healthScore, organizationCount, contributorCount + FROM insights_projects_populated_ds + ARRAY JOIN collectionsSlugs as slug + WHERE enabled = 1 + ) + GROUP BY slug + +TYPE COPY +TARGET_DATASOURCE collection_insights_aggregate_projects_ds +COPY_MODE replace +COPY_SCHEDULE 0 3 * * * From 4d4681fdeeffc23fe5f688136a1bcb26480813ba Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Wed, 29 Jul 2026 12:32:08 +0100 Subject: [PATCH 05/10] chore: remove orphaned collection_insights_aggregate datasources (CM-XXXX) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Delete the 10 per-bucket datasources and the abandoned contributors_ds from an earlier consolidation attempt - none are referenced by any pipe after collapsing to the single collection_insights_aggregate_consolidated_ds. Also fixes stale comment references in the 10 copy pipes that still named the old contributors_ds target. Signed-off-by: Gašper Grom --- ..._insights_aggregate_bucket_0_ds.datasource | 16 ------------- ..._insights_aggregate_bucket_1_ds.datasource | 16 ------------- ..._insights_aggregate_bucket_2_ds.datasource | 16 ------------- ..._insights_aggregate_bucket_3_ds.datasource | 16 ------------- ..._insights_aggregate_bucket_4_ds.datasource | 16 ------------- ..._insights_aggregate_bucket_5_ds.datasource | 16 ------------- ..._insights_aggregate_bucket_6_ds.datasource | 16 ------------- ..._insights_aggregate_bucket_7_ds.datasource | 16 ------------- ..._insights_aggregate_bucket_8_ds.datasource | 16 ------------- ..._insights_aggregate_bucket_9_ds.datasource | 16 ------------- ...ights_aggregate_contributors_ds.datasource | 23 ------------------- .../pipes/collection_insights_aggregate.pipe | 2 +- ...ection_insights_aggregate_copy_pipe_0.pipe | 2 +- ...ection_insights_aggregate_copy_pipe_1.pipe | 2 +- ...ection_insights_aggregate_copy_pipe_2.pipe | 2 +- ...ection_insights_aggregate_copy_pipe_3.pipe | 2 +- ...ection_insights_aggregate_copy_pipe_4.pipe | 2 +- ...ection_insights_aggregate_copy_pipe_5.pipe | 2 +- ...ection_insights_aggregate_copy_pipe_6.pipe | 2 +- ...ection_insights_aggregate_copy_pipe_7.pipe | 2 +- ...ection_insights_aggregate_copy_pipe_8.pipe | 2 +- ...ection_insights_aggregate_copy_pipe_9.pipe | 2 +- 22 files changed, 11 insertions(+), 194 deletions(-) delete mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_0_ds.datasource delete mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_1_ds.datasource delete mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_2_ds.datasource delete mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_3_ds.datasource delete mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_4_ds.datasource delete mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_5_ds.datasource delete mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_6_ds.datasource delete mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_7_ds.datasource delete mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_8_ds.datasource delete mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_bucket_9_ds.datasource delete mode 100644 services/libs/tinybird/datasources/collection_insights_aggregate_contributors_ds.datasource diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_0_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_0_ds.datasource deleted file mode 100644 index 3f41f76f8b..0000000000 --- a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_0_ds.datasource +++ /dev/null @@ -1,16 +0,0 @@ -DESCRIPTION > - - Per-bucket contributor aggregate — bucket 0 only. - - Built daily by `collection_insights_aggregate_copy_pipe_0.pipe`. - - One row per collection that has activity rows in bucket 0. - - Columns: - - `collectionSlug`: Collection identifier (String). - - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). - - `updatedAt`: Update timestamp (DateTime). - -SCHEMA > - `collectionSlug` String, - `uniqueContributorCount` UInt64, - `updatedAt` DateTime - -ENGINE MergeTree -ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_1_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_1_ds.datasource deleted file mode 100644 index ce1291833a..0000000000 --- a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_1_ds.datasource +++ /dev/null @@ -1,16 +0,0 @@ -DESCRIPTION > - - Per-bucket contributor aggregate — bucket 1 only. - - Built daily by `collection_insights_aggregate_copy_pipe_1.pipe`. - - One row per collection that has activity rows in bucket 1. - - Columns: - - `collectionSlug`: Collection identifier (String). - - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). - - `updatedAt`: Update timestamp (DateTime). - -SCHEMA > - `collectionSlug` String, - `uniqueContributorCount` UInt64, - `updatedAt` DateTime - -ENGINE MergeTree -ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_2_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_2_ds.datasource deleted file mode 100644 index 41eb740103..0000000000 --- a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_2_ds.datasource +++ /dev/null @@ -1,16 +0,0 @@ -DESCRIPTION > - - Per-bucket contributor aggregate — bucket 2 only. - - Built daily by `collection_insights_aggregate_copy_pipe_2.pipe`. - - One row per collection that has activity rows in bucket 2. - - Columns: - - `collectionSlug`: Collection identifier (String). - - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). - - `updatedAt`: Update timestamp (DateTime). - -SCHEMA > - `collectionSlug` String, - `uniqueContributorCount` UInt64, - `updatedAt` DateTime - -ENGINE MergeTree -ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_3_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_3_ds.datasource deleted file mode 100644 index d934c77759..0000000000 --- a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_3_ds.datasource +++ /dev/null @@ -1,16 +0,0 @@ -DESCRIPTION > - - Per-bucket contributor aggregate — bucket 3 only. - - Built daily by `collection_insights_aggregate_copy_pipe_3.pipe`. - - One row per collection that has activity rows in bucket 3. - - Columns: - - `collectionSlug`: Collection identifier (String). - - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). - - `updatedAt`: Update timestamp (DateTime). - -SCHEMA > - `collectionSlug` String, - `uniqueContributorCount` UInt64, - `updatedAt` DateTime - -ENGINE MergeTree -ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_4_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_4_ds.datasource deleted file mode 100644 index 6ce8b3a1c3..0000000000 --- a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_4_ds.datasource +++ /dev/null @@ -1,16 +0,0 @@ -DESCRIPTION > - - Per-bucket contributor aggregate — bucket 4 only. - - Built daily by `collection_insights_aggregate_copy_pipe_4.pipe`. - - One row per collection that has activity rows in bucket 4. - - Columns: - - `collectionSlug`: Collection identifier (String). - - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). - - `updatedAt`: Update timestamp (DateTime). - -SCHEMA > - `collectionSlug` String, - `uniqueContributorCount` UInt64, - `updatedAt` DateTime - -ENGINE MergeTree -ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_5_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_5_ds.datasource deleted file mode 100644 index 5da48ccad4..0000000000 --- a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_5_ds.datasource +++ /dev/null @@ -1,16 +0,0 @@ -DESCRIPTION > - - Per-bucket contributor aggregate — bucket 5 only. - - Built daily by `collection_insights_aggregate_copy_pipe_5.pipe`. - - One row per collection that has activity rows in bucket 5. - - Columns: - - `collectionSlug`: Collection identifier (String). - - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). - - `updatedAt`: Update timestamp (DateTime). - -SCHEMA > - `collectionSlug` String, - `uniqueContributorCount` UInt64, - `updatedAt` DateTime - -ENGINE MergeTree -ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_6_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_6_ds.datasource deleted file mode 100644 index 2bdb2b9654..0000000000 --- a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_6_ds.datasource +++ /dev/null @@ -1,16 +0,0 @@ -DESCRIPTION > - - Per-bucket contributor aggregate — bucket 6 only. - - Built daily by `collection_insights_aggregate_copy_pipe_6.pipe`. - - One row per collection that has activity rows in bucket 6. - - Columns: - - `collectionSlug`: Collection identifier (String). - - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). - - `updatedAt`: Update timestamp (DateTime). - -SCHEMA > - `collectionSlug` String, - `uniqueContributorCount` UInt64, - `updatedAt` DateTime - -ENGINE MergeTree -ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_7_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_7_ds.datasource deleted file mode 100644 index 4742fe57d9..0000000000 --- a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_7_ds.datasource +++ /dev/null @@ -1,16 +0,0 @@ -DESCRIPTION > - - Per-bucket contributor aggregate — bucket 7 only. - - Built daily by `collection_insights_aggregate_copy_pipe_7.pipe`. - - One row per collection that has activity rows in bucket 7. - - Columns: - - `collectionSlug`: Collection identifier (String). - - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). - - `updatedAt`: Update timestamp (DateTime). - -SCHEMA > - `collectionSlug` String, - `uniqueContributorCount` UInt64, - `updatedAt` DateTime - -ENGINE MergeTree -ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_8_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_8_ds.datasource deleted file mode 100644 index f387f9e434..0000000000 --- a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_8_ds.datasource +++ /dev/null @@ -1,16 +0,0 @@ -DESCRIPTION > - - Per-bucket contributor aggregate — bucket 8 only. - - Built daily by `collection_insights_aggregate_copy_pipe_8.pipe`. - - One row per collection that has activity rows in bucket 8. - - Columns: - - `collectionSlug`: Collection identifier (String). - - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). - - `updatedAt`: Update timestamp (DateTime). - -SCHEMA > - `collectionSlug` String, - `uniqueContributorCount` UInt64, - `updatedAt` DateTime - -ENGINE MergeTree -ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_9_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_9_ds.datasource deleted file mode 100644 index c893ad2332..0000000000 --- a/services/libs/tinybird/datasources/collection_insights_aggregate_bucket_9_ds.datasource +++ /dev/null @@ -1,16 +0,0 @@ -DESCRIPTION > - - Per-bucket contributor aggregate — bucket 9 only. - - Built daily by `collection_insights_aggregate_copy_pipe_9.pipe`. - - One row per collection that has activity rows in bucket 9. - - Columns: - - `collectionSlug`: Collection identifier (String). - - `uniqueContributorCount`: Unique contributors in this bucket (UInt64). - - `updatedAt`: Update timestamp (DateTime). - -SCHEMA > - `collectionSlug` String, - `uniqueContributorCount` UInt64, - `updatedAt` DateTime - -ENGINE MergeTree -ENGINE_SORTING_KEY collectionSlug diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_contributors_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_contributors_ds.datasource deleted file mode 100644 index ea07d220db..0000000000 --- a/services/libs/tinybird/datasources/collection_insights_aggregate_contributors_ds.datasource +++ /dev/null @@ -1,23 +0,0 @@ -DESCRIPTION > - - Consolidated contributors aggregate table — one row per (collectionSlug, snapshotId) pair. - - Built by 10 separate copy pipes (collection_insights_aggregate_copy_pipe_N.pipe for N=0..9), - each scanning only its own bucket table to bound read volume and avoid massive union/join timeouts. - - Each copy pipe (bucket N) appends its rows to this shared table with the same snapshotId timestamp. - Bucket N's enrich finishes ~2:00-2:05 UTC, aggregate copy runs 2:05+5N UTC (staggered). - - Rows are tagged with snapshotId for versioning; queries filter to max(snapshotId) for latest data. - - TTL of 2 days cleans up old snapshots automatically (argMax-filtering on snapshotId is the - correctness mechanism; TTL is just to prevent unbounded growth). - - Columns: - - `collectionSlug`: The collection identifier (String). - - `uniqueContributorCount`: Unique contributors in this collection across all bucket scans (UInt64). - - `snapshotId`: Timestamp marking when this snapshot was created (daily granularity) (DateTime). - -SCHEMA > - `collectionSlug` String, - `uniqueContributorCount` UInt64, - `snapshotId` DateTime - -ENGINE MergeTree -ENGINE_PARTITION_KEY toYear(snapshotId) -ENGINE_SORTING_KEY collectionSlug, snapshotId -ENGINE_TTL toDateTime(snapshotId) + toIntervalDay(2) diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate.pipe index 9b57b4fc84..5f9acb5d88 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate.pipe @@ -2,7 +2,7 @@ DESCRIPTION > - `collection_insights_aggregate.pipe` serves aggregate insights metrics for a collection. - Returns a single row with metrics aggregated across every project in the collection, not one row per project. - Now a lightweight join of two precomputed tables: - - `collection_insights_aggregate_contributors_ds`: consolidated unique contributor counts (1 row per collection per snapshot) + - `collection_insights_aggregate_consolidated_ds`: consolidated unique contributor counts (1 row per collection per snapshot) - `collection_insights_aggregate_projects_ds`: project count and average health score (1 row per collection) - Updated daily: contributor aggregates run 2:05-2:50 UTC (staggered per-bucket, all append to shared table), projects aggregate runs 3:00 UTC (after all contributors finish). - Filters to latest snapshotId for current data (argMax-per-collection-slug pattern). diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_0.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_0.pipe index 98a225d506..e87cc4814e 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_0.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_0.pipe @@ -4,7 +4,7 @@ DESCRIPTION > - Computes uniqueContributorCount per collection slug for all collections that route to bucket 0. - Runs after bucket 0's enrich job completes (1:00 UTC + ~20min runtime typically finishes by 1:20 UTC). - Scheduled at 2:05 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). - - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_consolidated_ds. TAGS "Insights, Widget", "Collection" diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_1.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_1.pipe index 5f2248e514..a347c8f90c 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_1.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_1.pipe @@ -4,7 +4,7 @@ DESCRIPTION > - Computes uniqueContributorCount per collection slug for all collections that route to bucket 1. - Runs after bucket 1's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). - Scheduled at 2:10 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). - - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_consolidated_ds. TAGS "Insights, Widget", "Collection" diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_2.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_2.pipe index ce159e3e4d..22127d75c7 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_2.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_2.pipe @@ -4,7 +4,7 @@ DESCRIPTION > - Computes uniqueContributorCount per collection slug for all collections that route to bucket 2. - Runs after bucket 2's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). - Scheduled at 2:15 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). - - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_consolidated_ds. TAGS "Insights, Widget", "Collection" diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_3.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_3.pipe index b07797fdd2..32ffdd7a77 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_3.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_3.pipe @@ -4,7 +4,7 @@ DESCRIPTION > - Computes uniqueContributorCount per collection slug for all collections that route to bucket 3. - Runs after bucket 3's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). - Scheduled at 2:20 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). - - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_consolidated_ds. TAGS "Insights, Widget", "Collection" diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_4.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_4.pipe index ae44ca1ad4..ea53124d5e 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_4.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_4.pipe @@ -4,7 +4,7 @@ DESCRIPTION > - Computes uniqueContributorCount per collection slug for all collections that route to bucket 4. - Runs after bucket 4's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). - Scheduled at 2:25 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). - - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_consolidated_ds. TAGS "Insights, Widget", "Collection" diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_5.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_5.pipe index b3bb3864e0..c2fdfdd88f 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_5.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_5.pipe @@ -4,7 +4,7 @@ DESCRIPTION > - Computes uniqueContributorCount per collection slug for all collections that route to bucket 5. - Runs after bucket 5's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). - Scheduled at 2:30 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). - - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_consolidated_ds. TAGS "Insights, Widget", "Collection" diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_6.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_6.pipe index eab928287e..225524ec00 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_6.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_6.pipe @@ -4,7 +4,7 @@ DESCRIPTION > - Computes uniqueContributorCount per collection slug for all collections that route to bucket 6. - Runs after bucket 6's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). - Scheduled at 2:35 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). - - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_consolidated_ds. TAGS "Insights, Widget", "Collection" diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_7.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_7.pipe index f3976ccfe2..4c03bad973 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_7.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_7.pipe @@ -4,7 +4,7 @@ DESCRIPTION > - Computes uniqueContributorCount per collection slug for all collections that route to bucket 7. - Runs after bucket 7's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). - Scheduled at 2:40 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). - - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_consolidated_ds. TAGS "Insights, Widget", "Collection" diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_8.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_8.pipe index c2ed97ba52..f5e420fe54 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_8.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_8.pipe @@ -4,7 +4,7 @@ DESCRIPTION > - Computes uniqueContributorCount per collection slug for all collections that route to bucket 8. - Runs after bucket 8's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). - Scheduled at 2:45 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). - - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_consolidated_ds. TAGS "Insights, Widget", "Collection" diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_9.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_9.pipe index f32df84543..fb5b0fd565 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_9.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_copy_pipe_9.pipe @@ -4,7 +4,7 @@ DESCRIPTION > - Computes uniqueContributorCount per collection slug for all collections that route to bucket 9. - Runs after bucket 9's enrich job completes (1:05 UTC + ~20min runtime typically finishes by 1:25 UTC). - Scheduled at 2:50 UTC to comfortably clear after enrich job (5min margin, allows for timing variance). - - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_contributors_ds. + - COPY_MODE append: appends rows tagged with snapshotId to shared collection_insights_aggregate_consolidated_ds. TAGS "Insights, Widget", "Collection" From b8033094a71885601347229576386d2dc20e271c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Wed, 29 Jul 2026 12:59:20 +0100 Subject: [PATCH 06/10] feat: formatting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- ...ights_aggregate_consolidated_ds.datasource | 20 +++++------ ..._insights_aggregate_projects_ds.datasource | 10 +++--- .../tinybird/pipes/collection_buckets.pipe | 8 ++++- .../pipes/collection_insights_aggregate.pipe | 35 ++++++++----------- ...insights_aggregate_projects_copy_pipe.pipe | 17 +++++---- 5 files changed, 46 insertions(+), 44 deletions(-) diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_consolidated_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_consolidated_ds.datasource index ff3e17ef4e..a3301de41f 100644 --- a/services/libs/tinybird/datasources/collection_insights_aggregate_consolidated_ds.datasource +++ b/services/libs/tinybird/datasources/collection_insights_aggregate_consolidated_ds.datasource @@ -1,23 +1,23 @@ DESCRIPTION > - Consolidated contributors aggregate table — one row per (collectionSlug, snapshotId) pair. - Built by 10 separate copy pipes (collection_insights_aggregate_copy_pipe_N.pipe for N=0..9), - each scanning only its own bucket table to bound read volume and avoid massive union/join timeouts. + each scanning only its own bucket table to bound read volume and avoid massive union/join timeouts. - Each copy pipe (bucket N) appends its rows to this shared table with the same snapshotId timestamp. - Bucket N's enrich finishes ~2:00-2:05 UTC, aggregate copy runs 2:05+5N UTC (staggered). + Bucket N's enrich finishes ~2:00-2:05 UTC, aggregate copy runs 2:05+5N UTC (staggered). - Rows are tagged with snapshotId for versioning; queries filter to max(snapshotId) for latest data. - TTL of 2 days cleans up old snapshots automatically (argMax-filtering on snapshotId is the - correctness mechanism; TTL is just to prevent unbounded growth). + correctness mechanism; TTL is just to prevent unbounded growth). - Columns: - - `collectionSlug`: The collection identifier (String). - - `uniqueContributorCount`: Unique contributors in this collection across all bucket scans (UInt64). - - `snapshotId`: Timestamp marking when this snapshot was created (daily granularity) (DateTime). + - `collectionSlug`: The collection identifier (String). + - `uniqueContributorCount`: Unique contributors in this collection across all bucket scans (UInt64). + - `snapshotId`: Timestamp marking when this snapshot was created (daily granularity) (DateTime). SCHEMA > `collectionSlug` String, `uniqueContributorCount` UInt64, `snapshotId` DateTime -ENGINE "MergeTree" -ENGINE_PARTITION_KEY "toYear(snapshotId)" -ENGINE_SORTING_KEY "collectionSlug, snapshotId" -ENGINE_TTL "toDateTime(snapshotId) + toIntervalDay(2)" +ENGINE MergeTree +ENGINE_PARTITION_KEY toYear(snapshotId) +ENGINE_SORTING_KEY collectionSlug, snapshotId +ENGINE_TTL toDateTime(snapshotId) + toIntervalDay(2) diff --git a/services/libs/tinybird/datasources/collection_insights_aggregate_projects_ds.datasource b/services/libs/tinybird/datasources/collection_insights_aggregate_projects_ds.datasource index eb879832b6..b571789b86 100644 --- a/services/libs/tinybird/datasources/collection_insights_aggregate_projects_ds.datasource +++ b/services/libs/tinybird/datasources/collection_insights_aggregate_projects_ds.datasource @@ -1,13 +1,13 @@ DESCRIPTION > - Per-collection projects aggregate table — small, fast table with project counts and health scores. - Built once daily by a single copy pipe (collection_insights_aggregate_projects_copy_pipe.pipe) - that scans only insights_projects_populated_ds (no activity tables, no joins to billions of rows). + that scans only insights_projects_populated_ds (no activity tables, no joins to billions of rows). - Runs daily at a fixed time after the contributor aggregates are ready. - One row per collection slug, with: - - `collectionSlug`: The collection identifier (String). - - `projectCount`: Total number of onboarded projects in the collection (UInt64). - - `avgHealthScore`: Average health score across onboarded projects, rounded (Nullable Float64). - - `updatedAt`: Timestamp of the last update (DateTime). + - `collectionSlug`: The collection identifier (String). + - `projectCount`: Total number of onboarded projects in the collection (UInt64). + - `avgHealthScore`: Average health score across onboarded projects, rounded (Nullable Float64). + - `updatedAt`: Timestamp of the last update (DateTime). SCHEMA > `collectionSlug` String, diff --git a/services/libs/tinybird/pipes/collection_buckets.pipe b/services/libs/tinybird/pipes/collection_buckets.pipe index fd39cd76b5..93fee7bba9 100644 --- a/services/libs/tinybird/pipes/collection_buckets.pipe +++ b/services/libs/tinybird/pipes/collection_buckets.pipe @@ -13,4 +13,10 @@ SQL > % SELECT cityHash64(slug) % 10 as bucketId FROM collections_filtered - WHERE slug = {{ String(collectionSlug, description="Collection slug to resolve bucketId for", required=True) }} + WHERE + slug + = {{ + String( + collectionSlug, description="Collection slug to resolve bucketId for", required=True + ) + }} diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate.pipe index 5f9acb5d88..957777f619 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate.pipe @@ -2,8 +2,8 @@ DESCRIPTION > - `collection_insights_aggregate.pipe` serves aggregate insights metrics for a collection. - Returns a single row with metrics aggregated across every project in the collection, not one row per project. - Now a lightweight join of two precomputed tables: - - `collection_insights_aggregate_consolidated_ds`: consolidated unique contributor counts (1 row per collection per snapshot) - - `collection_insights_aggregate_projects_ds`: project count and average health score (1 row per collection) + - `collection_insights_aggregate_consolidated_ds`: consolidated unique contributor counts (1 row per collection per snapshot) + - `collection_insights_aggregate_projects_ds`: project count and average health score (1 row per collection) - Updated daily: contributor aggregates run 2:05-2:50 UTC (staggered per-bucket, all append to shared table), projects aggregate runs 3:00 UTC (after all contributors finish). - Filters to latest snapshotId for current data (argMax-per-collection-slug pattern). - Parameters: @@ -17,7 +17,6 @@ DESCRIPTION > check) — projects with no attributed contributors/organizations have no meaningful health signal yet and would otherwise pull the average down artificially. - TOKEN "insights-app-token" READ TAGS "Insights, Widget", "Collection" @@ -30,23 +29,17 @@ DESCRIPTION > all buckets' output is consolidated into a single target datasource. SQL > - -% - SELECT - p.projectCount, - c.uniqueContributorCount, - p.avgHealthScore + % + SELECT p.projectCount, c.uniqueContributorCount, p.avgHealthScore FROM collection_insights_aggregate_projects_ds p - LEFT JOIN ( - SELECT - collectionSlug, - uniqueContributorCount - FROM collection_insights_aggregate_consolidated_ds - WHERE snapshotId = (SELECT max(snapshotId) FROM collection_insights_aggregate_consolidated_ds) - ) c + LEFT JOIN + ( + SELECT collectionSlug, uniqueContributorCount + FROM collection_insights_aggregate_consolidated_ds + WHERE + snapshotId = (SELECT max(snapshotId) FROM collection_insights_aggregate_consolidated_ds) + ) c ON p.collectionSlug = c.collectionSlug - WHERE p.collectionSlug = {{ String(collectionSlug, description="Filter by collection slug", required=True) }} - -TYPE endpoint - - + WHERE + p.collectionSlug + = {{ String(collectionSlug, description="Filter by collection slug", required=True) }} diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_projects_copy_pipe.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_projects_copy_pipe.pipe index 02ce943e50..d241ef64ff 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate_projects_copy_pipe.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_projects_copy_pipe.pipe @@ -17,14 +17,17 @@ SQL > SELECT slug as collectionSlug, count(distinct id) as projectCount, - round(avg(if(NOT ((organizationCount = 0) AND (contributorCount = 0)), healthScore, NULL))) as avgHealthScore, + round( + avg(if(NOT ((organizationCount = 0) AND (contributorCount = 0)), healthScore, NULL)) + ) as avgHealthScore, now() as updatedAt - FROM ( - SELECT id, slug, healthScore, organizationCount, contributorCount - FROM insights_projects_populated_ds - ARRAY JOIN collectionsSlugs as slug - WHERE enabled = 1 - ) + FROM + ( + SELECT id, slug, healthScore, organizationCount, contributorCount + FROM insights_projects_populated_ds ARRAY + JOIN collectionsSlugs as slug + WHERE enabled = 1 + ) GROUP BY slug TYPE COPY From 9c832b04151516e3243b7e50453b76115de6d811 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Wed, 29 Jul 2026 13:12:50 +0100 Subject: [PATCH 07/10] fix: restore community collection data in bucketed pipelines (CM-XXXX) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- .../tinybird/pipes/activities_filtered.pipe | 17 +++++++++++++++-- .../pipes/activities_filtered_retention.pipe | 16 +++++++++++++++- ...lection_bucket_clean_enrich_copy_pipe_0.pipe | 2 +- ...lection_bucket_clean_enrich_copy_pipe_1.pipe | 4 ++-- ...lection_bucket_clean_enrich_copy_pipe_2.pipe | 2 +- ...lection_bucket_clean_enrich_copy_pipe_3.pipe | 2 +- ...lection_bucket_clean_enrich_copy_pipe_4.pipe | 2 +- ...lection_bucket_clean_enrich_copy_pipe_5.pipe | 2 +- ...lection_bucket_clean_enrich_copy_pipe_6.pipe | 2 +- ...lection_bucket_clean_enrich_copy_pipe_7.pipe | 2 +- ...lection_bucket_clean_enrich_copy_pipe_8.pipe | 2 +- ...lection_bucket_clean_enrich_copy_pipe_9.pipe | 2 +- .../pipes/activityTypes_by_project.pipe | 16 +++++++++++++++- .../pipes/collection_insights_aggregate.pipe | 8 ++++++-- ...n_insights_aggregate_projects_copy_pipe.pipe | 16 ++++++++++------ 15 files changed, 72 insertions(+), 23 deletions(-) diff --git a/services/libs/tinybird/pipes/activities_filtered.pipe b/services/libs/tinybird/pipes/activities_filtered.pipe index d18137909c..7d847ed671 100644 --- a/services/libs/tinybird/pipes/activities_filtered.pipe +++ b/services/libs/tinybird/pipes/activities_filtered.pipe @@ -21,14 +21,27 @@ DESCRIPTION > - Performance is optimized through proper sorting keys on `segmentId`, `timestamp`, `type`, `platform`, and `memberId` in the source datasource. TOKEN "insights-app-token" READ -TOKEN "activities_filtered_endpoint_read_8708" READ NODE activities_filtered_bucket_routing SQL > % SELECT activityId as id, timestamp, type, platform, memberId, organizationId, segmentId FROM - {% if defined(collectionSlug) %} activityRelations_collection_bucket_union + {% if defined(collectionSlug) %} + ( + SELECT * + FROM activityRelations_collection_bucket_union + UNION ALL + SELECT a.* + FROM activityRelations_deduplicated_cleaned_ds a + WHERE + a.collectionSlug = {{ String(collectionSlug, required=True) }} + AND a.collectionSlug NOT IN ( + SELECT slug + FROM collections FINAL + WHERE isNull (deletedAt) AND isNull (ssoUserId) + ) + ) {% else %} activityRelations_bucket_routing {% end %} as a where diff --git a/services/libs/tinybird/pipes/activities_filtered_retention.pipe b/services/libs/tinybird/pipes/activities_filtered_retention.pipe index 17bfb92214..467e662045 100644 --- a/services/libs/tinybird/pipes/activities_filtered_retention.pipe +++ b/services/libs/tinybird/pipes/activities_filtered_retention.pipe @@ -24,7 +24,21 @@ SQL > % SELECT activityId as id, timestamp, type, platform, memberId, organizationId, segmentId FROM - {% if defined(collectionSlug) %} activityRelations_collection_bucket_union + {% if defined(collectionSlug) %} + ( + SELECT * + FROM activityRelations_collection_bucket_union + UNION ALL + SELECT a.* + FROM activityRelations_deduplicated_cleaned_ds a + WHERE + a.collectionSlug = {{ String(collectionSlug, required=True) }} + AND a.collectionSlug NOT IN ( + SELECT slug + FROM collections FINAL + WHERE isNull (deletedAt) AND isNull (ssoUserId) + ) + ) {% else %} activityRelations_bucket_routing {% end %} as a where diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_0.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_0.pipe index 4c40e14d66..90a7b08200 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_0.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_0.pipe @@ -5,7 +5,7 @@ DESCRIPTION > activityRelations, member bot-status, and repository enable/disable state, none of which are append-only-safe, so this recomputes the full bucket via COPY_MODE replace rather than appending. Scheduled daily, staggered 5 minutes apart across all 10 collection buckets (this - bucket at :) so all 10 full-replace jobs don't fire simultaneously. + bucket at 01:00) so all 10 full-replace jobs don't fire simultaneously. NODE country_mapping_array SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_1.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_1.pipe index abad7d2ea8..8e48f0c7da 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_1.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_1.pipe @@ -5,7 +5,7 @@ DESCRIPTION > activityRelations, member bot-status, and repository enable/disable state, none of which are append-only-safe, so this recomputes the full bucket via COPY_MODE replace rather than appending. Scheduled daily, staggered 5 minutes apart across all 10 collection buckets (this - bucket at :0) so all 10 full-replace jobs don't fire simultaneously. + bucket at 1:05) so all 10 full-replace jobs don't fire simultaneously. NODE country_mapping_array SQL > @@ -54,4 +54,4 @@ SQL > TYPE COPY TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_1_ds COPY_MODE replace -COPY_SCHEDULE 0 1 * * * +COPY_SCHEDULE 5 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_2.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_2.pipe index 3bcffaed5f..8219be0ad3 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_2.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_2.pipe @@ -54,4 +54,4 @@ SQL > TYPE COPY TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_2_ds COPY_MODE replace -COPY_SCHEDULE 5 1 * * * +COPY_SCHEDULE 10 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_3.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_3.pipe index 7657cc7f81..08e24a70df 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_3.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_3.pipe @@ -54,4 +54,4 @@ SQL > TYPE COPY TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_3_ds COPY_MODE replace -COPY_SCHEDULE 10 1 * * * +COPY_SCHEDULE 15 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_4.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_4.pipe index b899ef8043..f8782be0cc 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_4.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_4.pipe @@ -54,4 +54,4 @@ SQL > TYPE COPY TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_4_ds COPY_MODE replace -COPY_SCHEDULE 15 1 * * * +COPY_SCHEDULE 20 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_5.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_5.pipe index da2bfd8b2b..1a2611e44a 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_5.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_5.pipe @@ -54,4 +54,4 @@ SQL > TYPE COPY TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_5_ds COPY_MODE replace -COPY_SCHEDULE 20 1 * * * +COPY_SCHEDULE 25 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_6.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_6.pipe index 16e8482c63..2cba930e92 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_6.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_6.pipe @@ -54,4 +54,4 @@ SQL > TYPE COPY TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_6_ds COPY_MODE replace -COPY_SCHEDULE 25 1 * * * +COPY_SCHEDULE 30 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_7.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_7.pipe index 8d380e1fd4..0db922c7dc 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_7.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_7.pipe @@ -54,4 +54,4 @@ SQL > TYPE COPY TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_7_ds COPY_MODE replace -COPY_SCHEDULE 30 1 * * * +COPY_SCHEDULE 35 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_8.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_8.pipe index f19f702038..eee2ec0ec6 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_8.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_8.pipe @@ -54,4 +54,4 @@ SQL > TYPE COPY TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_8_ds COPY_MODE replace -COPY_SCHEDULE 35 1 * * * +COPY_SCHEDULE 40 1 * * * diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_9.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_9.pipe index 7f56eb9fb4..a7ae56ebdb 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_9.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_clean_enrich_copy_pipe_9.pipe @@ -54,4 +54,4 @@ SQL > TYPE COPY TARGET_DATASOURCE activityRelations_collection_deduplicated_cleaned_bucket_9_ds COPY_MODE replace -COPY_SCHEDULE 40 1 * * * +COPY_SCHEDULE 45 1 * * * diff --git a/services/libs/tinybird/pipes/activityTypes_by_project.pipe b/services/libs/tinybird/pipes/activityTypes_by_project.pipe index d850ebd683..6ff1f8475b 100644 --- a/services/libs/tinybird/pipes/activityTypes_by_project.pipe +++ b/services/libs/tinybird/pipes/activityTypes_by_project.pipe @@ -22,7 +22,21 @@ SQL > % SELECT DISTINCT a.type as activityType, a.platform, at.label FROM - {% if defined(collectionSlug) %} activityRelations_collection_bucket_union + {% if defined(collectionSlug) %} + ( + SELECT * + FROM activityRelations_collection_bucket_union + UNION ALL + SELECT a.* + FROM activityRelations_deduplicated_cleaned_ds a + WHERE + a.collectionSlug = {{ String(collectionSlug, required=True) }} + AND a.collectionSlug NOT IN ( + SELECT slug + FROM collections FINAL + WHERE isNull (deletedAt) AND isNull (ssoUserId) + ) + ) {% else %} activityRelations_bucket_routing {% end %} a INNER JOIN activityTypes at ON a.type = at.activityType AND a.platform = at.platform diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate.pipe index 957777f619..fd1b8756e9 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate.pipe @@ -34,10 +34,14 @@ SQL > FROM collection_insights_aggregate_projects_ds p LEFT JOIN ( - SELECT collectionSlug, uniqueContributorCount + SELECT collectionSlug, uniqueContributorCount, snapshotId FROM collection_insights_aggregate_consolidated_ds WHERE - snapshotId = (SELECT max(snapshotId) FROM collection_insights_aggregate_consolidated_ds) + (collectionSlug, snapshotId) IN ( + SELECT collectionSlug, max(snapshotId) + FROM collection_insights_aggregate_consolidated_ds + GROUP BY collectionSlug + ) ) c ON p.collectionSlug = c.collectionSlug WHERE diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate_projects_copy_pipe.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate_projects_copy_pipe.pipe index d241ef64ff..2972cd4a6a 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate_projects_copy_pipe.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate_projects_copy_pipe.pipe @@ -15,20 +15,24 @@ DESCRIPTION > SQL > SELECT - slug as collectionSlug, - count(distinct id) as projectCount, + collectionSlug, + count(distinct projectId) as projectCount, round( avg(if(NOT ((organizationCount = 0) AND (contributorCount = 0)), healthScore, NULL)) ) as avgHealthScore, now() as updatedAt FROM ( - SELECT id, slug, healthScore, organizationCount, contributorCount - FROM insights_projects_populated_ds ARRAY - JOIN collectionsSlugs as slug + SELECT + id as projectId, + healthScore, + organizationCount, + contributorCount, + arrayJoin(collectionsSlugs) as collectionSlug + FROM insights_projects_populated_ds WHERE enabled = 1 ) - GROUP BY slug + GROUP BY collectionSlug TYPE COPY TARGET_DATASOURCE collection_insights_aggregate_projects_ds From 389101810166904e3644ea1e21c2ee57bbbc83af Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Wed, 29 Jul 2026 13:33:24 +0100 Subject: [PATCH 08/10] fix: remove broken community-collection fallback from activity bucket routing (CM-XXXX) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Activity-bucket routing intentionally scopes to curated collections only (ssoUserId IS NULL) - community collection activities are not used for widgets. The fallback UNION referenced a nonexistent activityRelations_deduplicated_cleaned_ds datasource and was never valid. Signed-off-by: Gašper Grom --- .../libs/tinybird/pipes/activities_filtered.pipe | 16 +--------------- .../pipes/activities_filtered_retention.pipe | 16 +--------------- .../tinybird/pipes/activityTypes_by_project.pipe | 16 +--------------- 3 files changed, 3 insertions(+), 45 deletions(-) diff --git a/services/libs/tinybird/pipes/activities_filtered.pipe b/services/libs/tinybird/pipes/activities_filtered.pipe index 7d847ed671..53344642b7 100644 --- a/services/libs/tinybird/pipes/activities_filtered.pipe +++ b/services/libs/tinybird/pipes/activities_filtered.pipe @@ -27,21 +27,7 @@ SQL > % SELECT activityId as id, timestamp, type, platform, memberId, organizationId, segmentId FROM - {% if defined(collectionSlug) %} - ( - SELECT * - FROM activityRelations_collection_bucket_union - UNION ALL - SELECT a.* - FROM activityRelations_deduplicated_cleaned_ds a - WHERE - a.collectionSlug = {{ String(collectionSlug, required=True) }} - AND a.collectionSlug NOT IN ( - SELECT slug - FROM collections FINAL - WHERE isNull (deletedAt) AND isNull (ssoUserId) - ) - ) + {% if defined(collectionSlug) %} activityRelations_collection_bucket_union {% else %} activityRelations_bucket_routing {% end %} as a where diff --git a/services/libs/tinybird/pipes/activities_filtered_retention.pipe b/services/libs/tinybird/pipes/activities_filtered_retention.pipe index 467e662045..17bfb92214 100644 --- a/services/libs/tinybird/pipes/activities_filtered_retention.pipe +++ b/services/libs/tinybird/pipes/activities_filtered_retention.pipe @@ -24,21 +24,7 @@ SQL > % SELECT activityId as id, timestamp, type, platform, memberId, organizationId, segmentId FROM - {% if defined(collectionSlug) %} - ( - SELECT * - FROM activityRelations_collection_bucket_union - UNION ALL - SELECT a.* - FROM activityRelations_deduplicated_cleaned_ds a - WHERE - a.collectionSlug = {{ String(collectionSlug, required=True) }} - AND a.collectionSlug NOT IN ( - SELECT slug - FROM collections FINAL - WHERE isNull (deletedAt) AND isNull (ssoUserId) - ) - ) + {% if defined(collectionSlug) %} activityRelations_collection_bucket_union {% else %} activityRelations_bucket_routing {% end %} as a where diff --git a/services/libs/tinybird/pipes/activityTypes_by_project.pipe b/services/libs/tinybird/pipes/activityTypes_by_project.pipe index 6ff1f8475b..d850ebd683 100644 --- a/services/libs/tinybird/pipes/activityTypes_by_project.pipe +++ b/services/libs/tinybird/pipes/activityTypes_by_project.pipe @@ -22,21 +22,7 @@ SQL > % SELECT DISTINCT a.type as activityType, a.platform, at.label FROM - {% if defined(collectionSlug) %} - ( - SELECT * - FROM activityRelations_collection_bucket_union - UNION ALL - SELECT a.* - FROM activityRelations_deduplicated_cleaned_ds a - WHERE - a.collectionSlug = {{ String(collectionSlug, required=True) }} - AND a.collectionSlug NOT IN ( - SELECT slug - FROM collections FINAL - WHERE isNull (deletedAt) AND isNull (ssoUserId) - ) - ) + {% if defined(collectionSlug) %} activityRelations_collection_bucket_union {% else %} activityRelations_bucket_routing {% end %} a INNER JOIN activityTypes at ON a.type = at.activityType AND a.platform = at.platform From cd52952282e82a81b7346eefeab3fcb30dfd9901 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Wed, 29 Jul 2026 13:47:24 +0100 Subject: [PATCH 09/10] feat: include community collections in activity bucketing (CM-XXXX) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drop the ssoUserId IS NULL restriction from the collection-bucket fan-out MVs and their snapshot/backfill counterparts, so community collection activity is now bucketed alongside curated collections. Public collection listing/search (collections_filtered.pipe) is intentionally left scoped to curated-only - this only widens activity-bucket routing. Signed-off-by: Gašper Grom --- .../activityRelations_collection_bucket_MV_0.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_1.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_2.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_3.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_4.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_5.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_6.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_7.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_8.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_9.pipe | 14 +++++++------- ...yRelations_collection_bucket_MV_snapshot_0.pipe | 7 ++++--- ...yRelations_collection_bucket_MV_snapshot_1.pipe | 7 ++++--- ...yRelations_collection_bucket_MV_snapshot_2.pipe | 7 ++++--- ...yRelations_collection_bucket_MV_snapshot_3.pipe | 7 ++++--- ...yRelations_collection_bucket_MV_snapshot_4.pipe | 7 ++++--- ...yRelations_collection_bucket_MV_snapshot_5.pipe | 7 ++++--- ...yRelations_collection_bucket_MV_snapshot_6.pipe | 7 ++++--- ...yRelations_collection_bucket_MV_snapshot_7.pipe | 7 ++++--- ...yRelations_collection_bucket_MV_snapshot_8.pipe | 7 ++++--- ...yRelations_collection_bucket_MV_snapshot_9.pipe | 7 ++++--- .../libs/tinybird/pipes/collection_buckets.pipe | 13 ++++++++----- 21 files changed, 118 insertions(+), 105 deletions(-) diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_0.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_0.pipe index c7283f7a06..f6bd6f506d 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_0.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_0.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for curated-collection bucket 0. Joins each inserted + Fan-out materialized view for collection bucket 0. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership, restricts to curated collections only - (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only - rows whose collectionSlug hashes into bucket 0. A project in N curated collections produces - up to N rows across up to N different buckets - this is intentional write amplification, see - bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership (curated and community collections alike - + the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and + keeps only rows whose collectionSlug hashes into bucket 0. A project in N collections + produces up to N rows across up to N different buckets - this is intentional write + amplification, see bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_0 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_0 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_1.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_1.pipe index aa6b24cf25..44976921a8 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_1.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_1.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for curated-collection bucket 1. Joins each inserted + Fan-out materialized view for collection bucket 1. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership, restricts to curated collections only - (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only - rows whose collectionSlug hashes into bucket 1. A project in N curated collections produces - up to N rows across up to N different buckets - this is intentional write amplification, see - bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership (curated and community collections alike - + the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and + keeps only rows whose collectionSlug hashes into bucket 1. A project in N collections + produces up to N rows across up to N different buckets - this is intentional write + amplification, see bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_1 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_1 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_2.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_2.pipe index 55af3a5a0a..b74a9ce28e 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_2.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_2.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for curated-collection bucket 2. Joins each inserted + Fan-out materialized view for collection bucket 2. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership, restricts to curated collections only - (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only - rows whose collectionSlug hashes into bucket 2. A project in N curated collections produces - up to N rows across up to N different buckets - this is intentional write amplification, see - bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership (curated and community collections alike - + the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and + keeps only rows whose collectionSlug hashes into bucket 2. A project in N collections + produces up to N rows across up to N different buckets - this is intentional write + amplification, see bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_2 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_2 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_3.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_3.pipe index 5bb4919436..92ca2f9360 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_3.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_3.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for curated-collection bucket 3. Joins each inserted + Fan-out materialized view for collection bucket 3. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership, restricts to curated collections only - (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only - rows whose collectionSlug hashes into bucket 3. A project in N curated collections produces - up to N rows across up to N different buckets - this is intentional write amplification, see - bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership (curated and community collections alike - + the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and + keeps only rows whose collectionSlug hashes into bucket 3. A project in N collections + produces up to N rows across up to N different buckets - this is intentional write + amplification, see bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_3 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_3 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_4.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_4.pipe index f2e75f80df..dc69c4dee9 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_4.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_4.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for curated-collection bucket 4. Joins each inserted + Fan-out materialized view for collection bucket 4. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership, restricts to curated collections only - (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only - rows whose collectionSlug hashes into bucket 4. A project in N curated collections produces - up to N rows across up to N different buckets - this is intentional write amplification, see - bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership (curated and community collections alike - + the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and + keeps only rows whose collectionSlug hashes into bucket 4. A project in N collections + produces up to N rows across up to N different buckets - this is intentional write + amplification, see bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_4 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_4 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_5.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_5.pipe index f8a7bad1e5..11b4539cd2 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_5.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_5.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for curated-collection bucket 5. Joins each inserted + Fan-out materialized view for collection bucket 5. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership, restricts to curated collections only - (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only - rows whose collectionSlug hashes into bucket 5. A project in N curated collections produces - up to N rows across up to N different buckets - this is intentional write amplification, see - bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership (curated and community collections alike - + the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and + keeps only rows whose collectionSlug hashes into bucket 5. A project in N collections + produces up to N rows across up to N different buckets - this is intentional write + amplification, see bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_5 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_5 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_6.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_6.pipe index b86ad03b4e..68deb70d38 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_6.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_6.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for curated-collection bucket 6. Joins each inserted + Fan-out materialized view for collection bucket 6. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership, restricts to curated collections only - (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only - rows whose collectionSlug hashes into bucket 6. A project in N curated collections produces - up to N rows across up to N different buckets - this is intentional write amplification, see - bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership (curated and community collections alike - + the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and + keeps only rows whose collectionSlug hashes into bucket 6. A project in N collections + produces up to N rows across up to N different buckets - this is intentional write + amplification, see bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_6 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_6 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_7.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_7.pipe index e2587f92b9..74d61b17d5 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_7.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_7.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for curated-collection bucket 7. Joins each inserted + Fan-out materialized view for collection bucket 7. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership, restricts to curated collections only - (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only - rows whose collectionSlug hashes into bucket 7. A project in N curated collections produces - up to N rows across up to N different buckets - this is intentional write amplification, see - bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership (curated and community collections alike - + the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and + keeps only rows whose collectionSlug hashes into bucket 7. A project in N collections + produces up to N rows across up to N different buckets - this is intentional write + amplification, see bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_7 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_7 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_8.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_8.pipe index 16c2d23d01..58b993ce40 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_8.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_8.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for curated-collection bucket 8. Joins each inserted + Fan-out materialized view for collection bucket 8. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership, restricts to curated collections only - (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only - rows whose collectionSlug hashes into bucket 8. A project in N curated collections produces - up to N rows across up to N different buckets - this is intentional write amplification, see - bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership (curated and community collections alike - + the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and + keeps only rows whose collectionSlug hashes into bucket 8. A project in N collections + produces up to N rows across up to N different buckets - this is intentional write + amplification, see bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_8 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_8 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_9.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_9.pipe index 99ff7a28ac..210f8fd095 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_9.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_9.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for curated-collection bucket 9. Joins each inserted + Fan-out materialized view for collection bucket 9. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership, restricts to curated collections only - (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only - rows whose collectionSlug hashes into bucket 9. A project in N curated collections produces - up to N rows across up to N different buckets - this is intentional write amplification, see - bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership (curated and community collections alike - + the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and + keeps only rows whose collectionSlug hashes into bucket 9. A project in N collections + produces up to N rows across up to N different buckets - this is intentional write + amplification, see bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_9 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_9 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe index 209e231050..7a01a34f70 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe @@ -1,7 +1,8 @@ DESCRIPTION > - On-demand backfill/bootstrap for curated-collection bucket 0 - populates + On-demand backfill/bootstrap for collection bucket 0 - populates activityRelations_collection_bucket_MV_ds_0 from the full historical activityRelations - table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + table via the same fan-out logic as the steady-state MV (curated and community collections + alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors activityRelations_bucket_MV_snapshot_0.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_0.pipe) then keeps it current on new inserts. @@ -11,7 +12,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_0 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_snapshot_0 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe index b8ddfd9d60..19eb9c8e97 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe @@ -1,7 +1,8 @@ DESCRIPTION > - On-demand backfill/bootstrap for curated-collection bucket 1 - populates + On-demand backfill/bootstrap for collection bucket 1 - populates activityRelations_collection_bucket_MV_ds_1 from the full historical activityRelations - table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + table via the same fan-out logic as the steady-state MV (curated and community collections + alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors activityRelations_bucket_MV_snapshot_1.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_1.pipe) then keeps it current on new inserts. @@ -11,7 +12,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_1 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_snapshot_1 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe index 881315413d..d46c93956e 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe @@ -1,7 +1,8 @@ DESCRIPTION > - On-demand backfill/bootstrap for curated-collection bucket 2 - populates + On-demand backfill/bootstrap for collection bucket 2 - populates activityRelations_collection_bucket_MV_ds_2 from the full historical activityRelations - table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + table via the same fan-out logic as the steady-state MV (curated and community collections + alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors activityRelations_bucket_MV_snapshot_2.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_2.pipe) then keeps it current on new inserts. @@ -11,7 +12,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_2 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_snapshot_2 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe index 1c1f283f7c..e2fcce6958 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe @@ -1,7 +1,8 @@ DESCRIPTION > - On-demand backfill/bootstrap for curated-collection bucket 3 - populates + On-demand backfill/bootstrap for collection bucket 3 - populates activityRelations_collection_bucket_MV_ds_3 from the full historical activityRelations - table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + table via the same fan-out logic as the steady-state MV (curated and community collections + alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors activityRelations_bucket_MV_snapshot_3.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_3.pipe) then keeps it current on new inserts. @@ -11,7 +12,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_3 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_snapshot_3 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe index 86c2dc8a3d..66bf4e0fbd 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe @@ -1,7 +1,8 @@ DESCRIPTION > - On-demand backfill/bootstrap for curated-collection bucket 4 - populates + On-demand backfill/bootstrap for collection bucket 4 - populates activityRelations_collection_bucket_MV_ds_4 from the full historical activityRelations - table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + table via the same fan-out logic as the steady-state MV (curated and community collections + alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors activityRelations_bucket_MV_snapshot_4.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_4.pipe) then keeps it current on new inserts. @@ -11,7 +12,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_4 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_snapshot_4 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe index f0f82977a7..18504488ce 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe @@ -1,7 +1,8 @@ DESCRIPTION > - On-demand backfill/bootstrap for curated-collection bucket 5 - populates + On-demand backfill/bootstrap for collection bucket 5 - populates activityRelations_collection_bucket_MV_ds_5 from the full historical activityRelations - table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + table via the same fan-out logic as the steady-state MV (curated and community collections + alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors activityRelations_bucket_MV_snapshot_5.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_5.pipe) then keeps it current on new inserts. @@ -11,7 +12,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_5 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_snapshot_5 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe index 79e5ed44df..893a3a11b8 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe @@ -1,7 +1,8 @@ DESCRIPTION > - On-demand backfill/bootstrap for curated-collection bucket 6 - populates + On-demand backfill/bootstrap for collection bucket 6 - populates activityRelations_collection_bucket_MV_ds_6 from the full historical activityRelations - table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + table via the same fan-out logic as the steady-state MV (curated and community collections + alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors activityRelations_bucket_MV_snapshot_6.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_6.pipe) then keeps it current on new inserts. @@ -11,7 +12,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_6 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_snapshot_6 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe index b0fee0b6cb..4f9b3e8461 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe @@ -1,7 +1,8 @@ DESCRIPTION > - On-demand backfill/bootstrap for curated-collection bucket 7 - populates + On-demand backfill/bootstrap for collection bucket 7 - populates activityRelations_collection_bucket_MV_ds_7 from the full historical activityRelations - table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + table via the same fan-out logic as the steady-state MV (curated and community collections + alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors activityRelations_bucket_MV_snapshot_7.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_7.pipe) then keeps it current on new inserts. @@ -11,7 +12,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_7 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_snapshot_7 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe index f3c2a35477..5b501add6a 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe @@ -1,7 +1,8 @@ DESCRIPTION > - On-demand backfill/bootstrap for curated-collection bucket 8 - populates + On-demand backfill/bootstrap for collection bucket 8 - populates activityRelations_collection_bucket_MV_ds_8 from the full historical activityRelations - table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + table via the same fan-out logic as the steady-state MV (curated and community collections + alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors activityRelations_bucket_MV_snapshot_8.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_8.pipe) then keeps it current on new inserts. @@ -11,7 +12,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_8 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_snapshot_8 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe index b99f8bdbfa..5647a1032c 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe @@ -1,7 +1,8 @@ DESCRIPTION > - On-demand backfill/bootstrap for curated-collection bucket 9 - populates + On-demand backfill/bootstrap for collection bucket 9 - populates activityRelations_collection_bucket_MV_ds_9 from the full historical activityRelations - table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors + table via the same fan-out logic as the steady-state MV (curated and community collections + alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors activityRelations_bucket_MV_snapshot_9.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_9.pipe) then keeps it current on new inserts. @@ -11,7 +12,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_9 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) NODE collection_fanout_snapshot_9 SQL > diff --git a/services/libs/tinybird/pipes/collection_buckets.pipe b/services/libs/tinybird/pipes/collection_buckets.pipe index 93fee7bba9..88c708eab6 100644 --- a/services/libs/tinybird/pipes/collection_buckets.pipe +++ b/services/libs/tinybird/pipes/collection_buckets.pipe @@ -1,10 +1,13 @@ DESCRIPTION > - - `collection_buckets.pipe` resolves a curated collection's slug to its bucketId, mirroring + - `collection_buckets.pipe` resolves a collection's slug to its bucketId, mirroring `project_buckets.pipe`'s role for segments. - - Bucket assignment is `cityHash64(collectionSlug) % 10` - collections_filtered already - restricts to curated collections (`ssoUserId IS NULL`), so non-curated collections never - resolve a bucket here. - - Unlike project bucketing, a project can belong to multiple curated collections, so a + - Bucket assignment is `cityHash64(collectionSlug) % 10`. + - NOTE: this pipe reads from `collections_filtered`, which restricts to curated collections + (`ssoUserId IS NULL`) for public listing/search purposes. Community collections still resolve + a deterministic bucketId via the same hash (see the bucket MV pipes, which fan out both + curated and community collections), they just aren't resolvable through this specific lookup + since it's scoped to `collections_filtered`. + - Unlike project bucketing, a project can belong to multiple collections, so a project's activities may be duplicated into multiple collection buckets - this pipe only resolves the bucket for a single collection slug, it does not describe the fan-out. From c1ee75d2a4f3dbf52f7a67a58cc2f6b842d868b2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Wed, 29 Jul 2026 15:10:41 +0100 Subject: [PATCH 10/10] Revert "feat: include community collections in activity bucketing (CM-XXXX)" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This reverts commit cd52952282e82a81b7346eefeab3fcb30dfd9901. Signed-off-by: Gašper Grom --- .../activityRelations_collection_bucket_MV_0.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_1.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_2.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_3.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_4.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_5.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_6.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_7.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_8.pipe | 14 +++++++------- .../activityRelations_collection_bucket_MV_9.pipe | 14 +++++++------- ...yRelations_collection_bucket_MV_snapshot_0.pipe | 7 +++---- ...yRelations_collection_bucket_MV_snapshot_1.pipe | 7 +++---- ...yRelations_collection_bucket_MV_snapshot_2.pipe | 7 +++---- ...yRelations_collection_bucket_MV_snapshot_3.pipe | 7 +++---- ...yRelations_collection_bucket_MV_snapshot_4.pipe | 7 +++---- ...yRelations_collection_bucket_MV_snapshot_5.pipe | 7 +++---- ...yRelations_collection_bucket_MV_snapshot_6.pipe | 7 +++---- ...yRelations_collection_bucket_MV_snapshot_7.pipe | 7 +++---- ...yRelations_collection_bucket_MV_snapshot_8.pipe | 7 +++---- ...yRelations_collection_bucket_MV_snapshot_9.pipe | 7 +++---- .../libs/tinybird/pipes/collection_buckets.pipe | 13 +++++-------- 21 files changed, 105 insertions(+), 118 deletions(-) diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_0.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_0.pipe index f6bd6f506d..c7283f7a06 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_0.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_0.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for collection bucket 0. Joins each inserted + Fan-out materialized view for curated-collection bucket 0. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership (curated and community collections alike - - the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and - keeps only rows whose collectionSlug hashes into bucket 0. A project in N collections - produces up to N rows across up to N different buckets - this is intentional write - amplification, see bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 0. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_0 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_0 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_1.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_1.pipe index 44976921a8..aa6b24cf25 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_1.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_1.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for collection bucket 1. Joins each inserted + Fan-out materialized view for curated-collection bucket 1. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership (curated and community collections alike - - the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and - keeps only rows whose collectionSlug hashes into bucket 1. A project in N collections - produces up to N rows across up to N different buckets - this is intentional write - amplification, see bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 1. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_1 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_1 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_2.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_2.pipe index b74a9ce28e..55af3a5a0a 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_2.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_2.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for collection bucket 2. Joins each inserted + Fan-out materialized view for curated-collection bucket 2. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership (curated and community collections alike - - the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and - keeps only rows whose collectionSlug hashes into bucket 2. A project in N collections - produces up to N rows across up to N different buckets - this is intentional write - amplification, see bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 2. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_2 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_2 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_3.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_3.pipe index 92ca2f9360..5bb4919436 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_3.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_3.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for collection bucket 3. Joins each inserted + Fan-out materialized view for curated-collection bucket 3. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership (curated and community collections alike - - the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and - keeps only rows whose collectionSlug hashes into bucket 3. A project in N collections - produces up to N rows across up to N different buckets - this is intentional write - amplification, see bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 3. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_3 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_3 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_4.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_4.pipe index dc69c4dee9..f2e75f80df 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_4.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_4.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for collection bucket 4. Joins each inserted + Fan-out materialized view for curated-collection bucket 4. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership (curated and community collections alike - - the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and - keeps only rows whose collectionSlug hashes into bucket 4. A project in N collections - produces up to N rows across up to N different buckets - this is intentional write - amplification, see bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 4. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_4 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_4 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_5.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_5.pipe index 11b4539cd2..f8a7bad1e5 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_5.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_5.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for collection bucket 5. Joins each inserted + Fan-out materialized view for curated-collection bucket 5. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership (curated and community collections alike - - the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and - keeps only rows whose collectionSlug hashes into bucket 5. A project in N collections - produces up to N rows across up to N different buckets - this is intentional write - amplification, see bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 5. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_5 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_5 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_6.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_6.pipe index 68deb70d38..b86ad03b4e 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_6.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_6.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for collection bucket 6. Joins each inserted + Fan-out materialized view for curated-collection bucket 6. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership (curated and community collections alike - - the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and - keeps only rows whose collectionSlug hashes into bucket 6. A project in N collections - produces up to N rows across up to N different buckets - this is intentional write - amplification, see bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 6. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_6 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_6 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_7.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_7.pipe index 74d61b17d5..e2587f92b9 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_7.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_7.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for collection bucket 7. Joins each inserted + Fan-out materialized view for curated-collection bucket 7. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership (curated and community collections alike - - the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and - keeps only rows whose collectionSlug hashes into bucket 7. A project in N collections - produces up to N rows across up to N different buckets - this is intentional write - amplification, see bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 7. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_7 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_7 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_8.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_8.pipe index 58b993ce40..16c2d23d01 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_8.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_8.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for collection bucket 8. Joins each inserted + Fan-out materialized view for curated-collection bucket 8. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership (curated and community collections alike - - the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and - keeps only rows whose collectionSlug hashes into bucket 8. A project in N collections - produces up to N rows across up to N different buckets - this is intentional write - amplification, see bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 8. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_8 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_8 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_9.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_9.pipe index 210f8fd095..99ff7a28ac 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_9.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_9.pipe @@ -1,15 +1,15 @@ DESCRIPTION > - Fan-out materialized view for collection bucket 9. Joins each inserted + Fan-out materialized view for curated-collection bucket 9. Joins each inserted activityRelations row to its project's collectionsSlugs (via insights_projects_populated_ds), - explodes to one row per collection membership (curated and community collections alike - - the curated_collection_slugs node name is legacy, it no longer filters by ssoUserId), and - keeps only rows whose collectionSlug hashes into bucket 9. A project in N collections - produces up to N rows across up to N different buckets - this is intentional write - amplification, see bucketing-architecture.md's collection-bucketing addendum. + explodes to one row per collection membership, restricts to curated collections only + (ssoUserId IS NULL, via the curated_collection_slugs dictionary-like lookup), and keeps only + rows whose collectionSlug hashes into bucket 9. A project in N curated collections produces + up to N rows across up to N different buckets - this is intentional write amplification, see + bucketing-architecture.md's collection-bucketing addendum. NODE curated_collection_slugs_9 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_9 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe index 7a01a34f70..209e231050 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_0.pipe @@ -1,8 +1,7 @@ DESCRIPTION > - On-demand backfill/bootstrap for collection bucket 0 - populates + On-demand backfill/bootstrap for curated-collection bucket 0 - populates activityRelations_collection_bucket_MV_ds_0 from the full historical activityRelations - table via the same fan-out logic as the steady-state MV (curated and community collections - alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors activityRelations_bucket_MV_snapshot_0.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_0.pipe) then keeps it current on new inserts. @@ -12,7 +11,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_0 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_snapshot_0 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe index 19eb9c8e97..b8ddfd9d60 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_1.pipe @@ -1,8 +1,7 @@ DESCRIPTION > - On-demand backfill/bootstrap for collection bucket 1 - populates + On-demand backfill/bootstrap for curated-collection bucket 1 - populates activityRelations_collection_bucket_MV_ds_1 from the full historical activityRelations - table via the same fan-out logic as the steady-state MV (curated and community collections - alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors activityRelations_bucket_MV_snapshot_1.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_1.pipe) then keeps it current on new inserts. @@ -12,7 +11,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_1 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_snapshot_1 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe index d46c93956e..881315413d 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_2.pipe @@ -1,8 +1,7 @@ DESCRIPTION > - On-demand backfill/bootstrap for collection bucket 2 - populates + On-demand backfill/bootstrap for curated-collection bucket 2 - populates activityRelations_collection_bucket_MV_ds_2 from the full historical activityRelations - table via the same fan-out logic as the steady-state MV (curated and community collections - alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors activityRelations_bucket_MV_snapshot_2.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_2.pipe) then keeps it current on new inserts. @@ -12,7 +11,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_2 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_snapshot_2 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe index e2fcce6958..1c1f283f7c 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_3.pipe @@ -1,8 +1,7 @@ DESCRIPTION > - On-demand backfill/bootstrap for collection bucket 3 - populates + On-demand backfill/bootstrap for curated-collection bucket 3 - populates activityRelations_collection_bucket_MV_ds_3 from the full historical activityRelations - table via the same fan-out logic as the steady-state MV (curated and community collections - alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors activityRelations_bucket_MV_snapshot_3.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_3.pipe) then keeps it current on new inserts. @@ -12,7 +11,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_3 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_snapshot_3 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe index 66bf4e0fbd..86c2dc8a3d 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_4.pipe @@ -1,8 +1,7 @@ DESCRIPTION > - On-demand backfill/bootstrap for collection bucket 4 - populates + On-demand backfill/bootstrap for curated-collection bucket 4 - populates activityRelations_collection_bucket_MV_ds_4 from the full historical activityRelations - table via the same fan-out logic as the steady-state MV (curated and community collections - alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors activityRelations_bucket_MV_snapshot_4.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_4.pipe) then keeps it current on new inserts. @@ -12,7 +11,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_4 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_snapshot_4 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe index 18504488ce..f0f82977a7 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_5.pipe @@ -1,8 +1,7 @@ DESCRIPTION > - On-demand backfill/bootstrap for collection bucket 5 - populates + On-demand backfill/bootstrap for curated-collection bucket 5 - populates activityRelations_collection_bucket_MV_ds_5 from the full historical activityRelations - table via the same fan-out logic as the steady-state MV (curated and community collections - alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors activityRelations_bucket_MV_snapshot_5.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_5.pipe) then keeps it current on new inserts. @@ -12,7 +11,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_5 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_snapshot_5 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe index 893a3a11b8..79e5ed44df 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_6.pipe @@ -1,8 +1,7 @@ DESCRIPTION > - On-demand backfill/bootstrap for collection bucket 6 - populates + On-demand backfill/bootstrap for curated-collection bucket 6 - populates activityRelations_collection_bucket_MV_ds_6 from the full historical activityRelations - table via the same fan-out logic as the steady-state MV (curated and community collections - alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors activityRelations_bucket_MV_snapshot_6.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_6.pipe) then keeps it current on new inserts. @@ -12,7 +11,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_6 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_snapshot_6 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe index 4f9b3e8461..b0fee0b6cb 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_7.pipe @@ -1,8 +1,7 @@ DESCRIPTION > - On-demand backfill/bootstrap for collection bucket 7 - populates + On-demand backfill/bootstrap for curated-collection bucket 7 - populates activityRelations_collection_bucket_MV_ds_7 from the full historical activityRelations - table via the same fan-out logic as the steady-state MV (curated and community collections - alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors activityRelations_bucket_MV_snapshot_7.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_7.pipe) then keeps it current on new inserts. @@ -12,7 +11,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_7 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_snapshot_7 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe index 5b501add6a..f3c2a35477 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_8.pipe @@ -1,8 +1,7 @@ DESCRIPTION > - On-demand backfill/bootstrap for collection bucket 8 - populates + On-demand backfill/bootstrap for curated-collection bucket 8 - populates activityRelations_collection_bucket_MV_ds_8 from the full historical activityRelations - table via the same fan-out logic as the steady-state MV (curated and community collections - alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors activityRelations_bucket_MV_snapshot_8.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_8.pipe) then keeps it current on new inserts. @@ -12,7 +11,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_8 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_snapshot_8 SQL > diff --git a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe index 5647a1032c..b99f8bdbfa 100644 --- a/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe +++ b/services/libs/tinybird/pipes/activityRelations_collection_bucket_MV_snapshot_9.pipe @@ -1,8 +1,7 @@ DESCRIPTION > - On-demand backfill/bootstrap for collection bucket 9 - populates + On-demand backfill/bootstrap for curated-collection bucket 9 - populates activityRelations_collection_bucket_MV_ds_9 from the full historical activityRelations - table via the same fan-out logic as the steady-state MV (curated and community collections - alike - the curated_collection_slugs_snapshot node name is legacy). Mirrors + table via the same fan-out/curated-filter logic as the steady-state MV. Mirrors activityRelations_bucket_MV_snapshot_9.pipe's role in the project-bucketing "Adding New Buckets" playbook. Run once manually after deploy to backfill history; the steady-state MV (activityRelations_collection_bucket_MV_9.pipe) then keeps it current on new inserts. @@ -12,7 +11,7 @@ DESCRIPTION > NODE curated_collection_slugs_snapshot_9 SQL > - SELECT slug FROM collections FINAL WHERE isNull (deletedAt) + SELECT slug FROM collections FINAL WHERE isNull (deletedAt) AND isNull (ssoUserId) NODE collection_fanout_snapshot_9 SQL > diff --git a/services/libs/tinybird/pipes/collection_buckets.pipe b/services/libs/tinybird/pipes/collection_buckets.pipe index 88c708eab6..93fee7bba9 100644 --- a/services/libs/tinybird/pipes/collection_buckets.pipe +++ b/services/libs/tinybird/pipes/collection_buckets.pipe @@ -1,13 +1,10 @@ DESCRIPTION > - - `collection_buckets.pipe` resolves a collection's slug to its bucketId, mirroring + - `collection_buckets.pipe` resolves a curated collection's slug to its bucketId, mirroring `project_buckets.pipe`'s role for segments. - - Bucket assignment is `cityHash64(collectionSlug) % 10`. - - NOTE: this pipe reads from `collections_filtered`, which restricts to curated collections - (`ssoUserId IS NULL`) for public listing/search purposes. Community collections still resolve - a deterministic bucketId via the same hash (see the bucket MV pipes, which fan out both - curated and community collections), they just aren't resolvable through this specific lookup - since it's scoped to `collections_filtered`. - - Unlike project bucketing, a project can belong to multiple collections, so a + - Bucket assignment is `cityHash64(collectionSlug) % 10` - collections_filtered already + restricts to curated collections (`ssoUserId IS NULL`), so non-curated collections never + resolve a bucket here. + - Unlike project bucketing, a project can belong to multiple curated collections, so a project's activities may be duplicated into multiple collection buckets - this pipe only resolves the bucket for a single collection slug, it does not describe the fan-out.