Skip to content

feat: metrics view time_grains with minute and hour multiples - #10080

Merged
nishantmonu51 merged 9 commits into
mainfrom
nishant/custom-time-grains
Nov 9, 2026
Merged

nishantmonu51 merged 9 commits into
mainfrom
nishant/custom-time-grains

Conversation

@nishantmonu51

Copy link
Copy Markdown
Collaborator

Adds time_grains to metrics views so a project can declare the grains its dashboards offer, including sub-hour and multi-hour buckets.

  • A time_grains list on a metrics view replaces the default grain set in explore (chart, TDD, pivot, scheduled reports) and canvas (filter bar, widget inspector). Omitting it keeps today's behavior: the standard grains at or above smallest_time_grain.
  • Ten TimeGrain multiples join the catalog: 5m, 10m, 15m, 30m, 2h, 3h, 4h, 6h, 8h, 12h. Each divides its parent unit, so buckets stay aligned to local midnight in the query timezone. The canonical token (10m) is accepted by resolver queries, metrics SQL (date_trunc('10m', ts)), explore URLs (grain=10m), canvas time_filters and additional_time_grain.
  • Every dialect buckets multiples on the local wall clock: DuckDB time_bucket, ClickHouse toStartOfInterval, Druid time_floor, Snowflake TIME_SLICE, StarRocks time_slice, Pinot DATETIMECONVERT, and floor-within-parent expressions for Databricks and BigQuery. Spine bins, legacy time-series null filling, comparison offsets and rollup derivability handle them.
  • Adaptive grain selection only picks declared grains; bookmarks, URL state and explore presets round-trip the new values.
  • Multiples are aggregation grains only: smallest_time_grain, rollup time_grain and requires/per/window.order selectors reject them at parse time. The list is a UI contract and is not enforced on API queries, same as smallest_time_grain today.
  • Docs: time_grains reference entry and a "Custom time grains" section in the time series guide.
type: metrics_view
timeseries: ts
smallest_time_grain: minute
time_grains: [10m, 30m, hour, day, week]

Not included: a multi-select for time_grains in the visual metrics editor. For frontend reviewers: protobuf-es stops stripping the TIME_GRAIN_ prefix once a value such as TIME_GRAIN_5_MINUTE exists, so the TS proto enum is now referenced as TimeGrain.TIME_GRAIN_DAY.

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

Lets a metrics view declare the grains its dashboards offer, including
multiples of minutes and hours (5m, 10m, 15m, 30m, 2h, 3h, 4h, 6h, 8h, 12h):

- `TimeGrain` enum gains the ten multiples; `MetricsViewSpec.time_grains`
  carries the declared list, parsed from `time_grains: [10m, hour, day]`
- Multiples bucket on the local wall clock in every dialect (DuckDB
  `time_bucket`, ClickHouse `toStartOfInterval`, Druid `time_floor`,
  Databricks/BigQuery floor-within-parent, Snowflake `TIME_SLICE`,
  StarRocks `time_slice`, Pinot `DATETIMECONVERT`); spine bins, null
  filling, comparison offsets and rollup derivability understand them
- `smallest_time_grain`, rollups and selectors stay standard-only
- Explore, TDD, pivot, scheduled reports and canvas pickers list only the
  declared grains; adaptive mode picks among them; the grain travels as
  `grain=10m` in URLs, presets, component `time_filters` and resolver
  queries, and round-trips through bookmarks
- Docs, JSON schema, AI tool text, golden resolver tests, unit tests and
  Playwright specs for explore and canvas
@nishantmonu51
nishantmonu51 requested a review from pjain1 October 29, 2026 11:24

@pjain1 pjain1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few issues come from code paths the PR did not touch but that still assume the standard grains, so I've listed them here rather than inline.

1. Databricks SelectTimeRangeBins was not updated for multiples (runtime/drivers/databricks/dialect.go:170-196). For TIME_GRAIN_10_MINUTE, ConvertToDateTruncSpecifier returns an empty string, so the step expression becomes INTERVAL 1 , which is invalid SQL. The second switch grain also has no case for multiples, so they fall into the day-and-above branch that runs SEQUENCE over dates. Time spine and null-fill queries with a multiple will fail on Databricks. The BigQuery version in this PR already handles this with TimeGrainBase, and the same approach would work here.

2. Rollup selection orders grains with the old grainOrder map (runtime/metricsview/executor/executor_rewrite_rollup.go:490). extractQueryTimeGrain compares grainOrder[g] < grainOrder[smallest], but the map has no entries for multiples, so every multiple reads as order 0, finer than milliseconds. For example, a pivot with both a 6h and a minute time floor picks 6h as the smallest grain. grainDerivableFrom(6h, hour) then returns true, so an hourly rollup is selected and the minute dimension is computed from hourly data. Using timeutil.TimeGrainFromAPI(g).Compare(...), as grainDerivableFrom already does, would fix this.

3. The x-axis drops minutes for minute multiples (MeasureChartXAxis.svelte:114). grainOrder < 1 used to mean "minute grain", but 5m, 10m, 15m and 30m now have orders 1 to 4. With a 15m grain, the ticks at 04:00, 04:15, 04:30 and 04:45 all read "4 AM".

4. Hover labels lose the time for both kinds of multiples (web-common/src/lib/time/ranges/formatter.ts). I confirmed both cases with formatGrainBucket:

  • Minute multiples: formatDateTimeByGrain only shows minutes when grainOrder <= minuteGrainOrder (0), so the 10m buckets at 04:20 and 04:30 both render as "10 Mar 2024, 4 am".
  • Hour multiples: formatGrainBucket only delegates to formatDateTimeByGrain when grainOrder <= hourGrainOrder. Orders 6 to 11 (2h to 12h) fall through to the week, month, quarter and year branch, so the 06:00 and 12:00 buckets of a 6h series both render as "10 Mar 2024", with no time at all.

formatTimePartOfTimeRange makes the same minuteGrainOrder comparison. Comparing the base unit (V1TimeGrainToDateTimeUnit) instead of the raw order would fix all of these.

5. Brush snapping and annotation bucketing use the base unit with a step of 1 (MeasureChart.svelte:325, annotation-utils.ts:44). With a 10m grain, a brush from 04:03 to 04:37 snaps to 04:04 to 04:37 instead of 04:10 to 04:30, so the zoomed range starts and ends mid-bucket. Annotations are also grouped into one-minute buckets rather than the chart's 10-minute buckets. floorToGrain and plusGrain from new-grains.ts already do the right thing here.


Developed in collaboration with Claude Code

Comment thread web-common/src/lib/time/new-grains.ts Outdated
return allowedGrains;
} else {
return [minTimeGrain];
return [validGrains[0] ?? minTimeGrain];

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When no declared grain fits under MAX_BUCKETS, this falls back to the finest grain. Before this PR, that branch only ran for ranges shorter than a minute, because year is exempt from the cap and was always available. Now that time_grains can leave out coarse grains, it also runs for long ranges and picks the densest option.

With time_grains: [10m, hour], adaptive mode chooses 10m for a 90-day range (12,960 buckets) and for a 365-day range (52,560 buckets). I checked this by calling getValidatedTimeGrain directly. A user who has fixed 10m gets the same result, because allowedGrains is [10m] and still contains the requested grain.

I suggest falling back to the coarsest valid grain when the interval is too long for every grain, and keeping the finest only when it is too short.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: when no declared grain fits, allowedGrainsForInterval now falls back to the coarsest one, and keeps the finest only when the range is shorter than a single bucket of every grain. [10m, hour] over 90 or 365 days now gives hour, covered in new-grains.spec.ts.

Comment thread runtime/queries/timeutil.go Outdated
return duration.StandardDuration{Minute: n}
case runtimev1.TimeGrain_TIME_GRAIN_HOUR:
return duration.StandardDuration{Hour: 1}
return duration.StandardDuration{Hour: n}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hour multiples become a StandardDuration{Hour: n}, and addTo and addToNew in metricsview_timeseries.go add that as absolute time. However, the SQL buckets (DuckDB time_bucket with a time zone), TruncateTime and OffsetTime all step hour multiples on the local wall clock. The measure chart still calls MetricsViewTimeSeries, so this affects explore dashboards.

Example: America/New_York on 2024-11-03 with a 6h grain. The SQL rows are at 04:00Z and 11:00Z (local 00:00 and 06:00). After the 04:00Z row, start = addTo(t, dur, tz) gives 10:00Z, so addNulls inserts an extra null bucket at 10:00Z before the real row at 11:00Z. Stepping with timeutil.OffsetTime for multiples would keep the null fill consistent with the queries.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: addTo and addToNew step multiples with timeutil.OffsetTime, and timeGrainToDuration is back to the standard grains. TestMetricsViewsTimeseriesLegacyNullFilling runs your New York fall-back example on both legacy implementations and fails on the previous code.

if tz == "" {
return fmt.Sprintf("TIMESTAMP_ADD(TIMESTAMP_TRUNC(%s, %s), INTERVAL DIV(EXTRACT(%s FROM %s), %d) * %d %s)", expr, parent, specifier, expr, n, n, specifier), nil
}
return fmt.Sprintf("TIMESTAMP_ADD(TIMESTAMP_TRUNC(%s, %s, '%s'), INTERVAL DIV(EXTRACT(%s FROM %s AT TIME ZONE '%s'), %d) * %d %s)", expr, parent, tz, specifier, expr, tz, n, n, specifier), nil

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With a time zone, this adds DIV(local_hour, n) * n hours of absolute time to local midnight. On daylight-saving days, that doesn't match the wall-clock buckets that DuckDB, ClickHouse and TruncateTime produce.

Example: America/New_York on 2024-03-10 (spring forward) with a 6h grain. An event at 06:30 EDT (10:30Z) has local hour 6. Local midnight is 05:00Z, so the bucket comes out as 05:00Z + 6h = 11:00Z, which starts after the event itself. The spine in SelectTimeRangeBins steps in absolute time from a wall-clock-truncated start, so its bins don't line up with these keys either. One option is to truncate with DATETIME_TRUNC on the local DATETIME, adjust the hour there, and convert back with TIMESTAMP(..., tz).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed as you suggested: hour multiples floor the local DATETIME and convert back with TIMESTAMP(..., tz). The spine now steps the local start by a generated index, since BigQuery has no DATETIME array generator.


return aligned
.splitBy({ [unit + "s"]: 1 })
.splitBy({ [unit + "s"]: V1TimeGrainToCount[grain] })

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

splitBy({ hours: n }) (and plusGrain, which uses plus) adds absolute time in luxon, but the runtime steps hour multiples on the wall clock (OffsetTime). For New York with a 6h grain across the 2024-11-03 fall-back, the server returns 04:00Z and then 11:00Z, while the axis bins are 04:00Z and 10:00Z. After the transition, the tick labels and bin count no longer match the data. Stepping with floorToGrain(plusGrain(...)), or adding hours through set({ hour }), would keep them aligned.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: plusGrain now steps hour multiples with set({ hour }), and the axis builds its bins by stepping plusGrain instead of splitBy.

const common = lists[0]?.filter((grain) =>
lists.every((list) => list.includes(grain)),
);
if (common?.length) return common;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the metrics views share no grains, this falls back to the standard grains at or above largestMinTimeGrain. That can offer grains that some of the views don't declare. For example, with view A declaring [10m, hour] and view B declaring [15m, day], the filter bar offers minute, hour, day, week and so on. This conflicts with the documented rule that time_grains replaces the default list. Falling back to the union of the views' lists, or to the coarsest common base unit, would be closer to the documented behavior.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: with no common grain, the canvas offers the union of the views' lists, so it never offers a grain that no view declares.

Comment thread runtime/pkg/timeutil/timeutil.go Outdated
return grainToAPI[tg]
}

// TimeGrainBase returns the base unit of an API grain and the number of those units it spans. Standard grains return themselves and 1.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment puts two sentences on one line, while the repository convention is semantic line breaks at the end of each sentence. The same applies to the comments on Base and ApproxDuration in this file, Base in runtime/metricsview/query.go, and branchOf in rollup_grain.go.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed here and in the other places you listed, plus the TimeGrain enum comment in the proto.

…atting

- DuckDB, Druid and BigQuery bucket hour multiples on the local wall clock with a time zone; DuckDB's time_bucket and Druid's time_floor step them in absolute time, which put New York summer buckets at 01:00, 07:00, 13:00 and 19:00
- Databricks and BigQuery spines step multiples, hour multiples on the wall clock
- Rollup selection orders grains with Compare, so a 6h plus minute pivot no longer picks an hourly rollup
- Legacy time series null filling steps multiples with OffsetTime
- Charts: x-axis bins step with plusGrain and show minutes for minute multiples, hover labels use the base unit, brush snapping and annotation buckets use floorToGrain
- plusGrain steps hour multiples on the wall clock
- allowedGrainsForInterval falls back to the coarsest grain when the range is too long for every declared grain
- Canvas offers the union of the views' grains when they share none
@nishantmonu51

Copy link
Copy Markdown
Collaborator Author

Thanks, all five are fixed in 775aebc:

  1. Databricks SelectTimeRangeBins steps INTERVAL n MINUTE/HOUR and steps hour multiples on the local wall clock.
  2. extractQueryTimeGrain compares with timeutil.Compare; TestExtractQueryTimeGrain covers the 6h plus minute case.
  3. The x-axis shows minutes for every minute-based grain.
  4. The formatter compares the base unit through a new getBaseGrain, so 10m labels show minutes and 6h labels show the hour.
  5. Brush snapping and annotation grouping use floorToGrain and plusGrain.

Checking these turned up the same DST problem on two more engines. DuckDB's time_bucket with a zone and Druid's time_floor both step hour periods in absolute time from a standard-time origin, so New York 6h buckets started at 01:00 local all summer. DuckDB now buckets the local wall-clock time. Druid builds the bucket from local midnight, with a fallback for DST days. Both are covered by golden tests across the spring-forward day: metrics_time_grains.yaml for DuckDB and ClickHouse, and metrics_time_grains_druid.yaml against the Druid test cluster.

@pjain1 pjain1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough fixes. I checked each item against 775aebc, and all eleven are resolved. The new tests pass locally.

One edge case remains in the same family as the DST fixes, noted inline.


Developed in collaboration with Claude Code

Comment thread runtime/pkg/timeutil/timeutil.go Outdated
case TimeGrainMinute:
tm = time.Date(tm.Year(), tm.Month(), tm.Day(), tm.Hour(), tm.Minute()-tm.Minute()%n, 0, 0, tz)
case TimeGrainHour:
tm = time.Date(tm.Year(), tm.Month(), tm.Day(), tm.Hour()-tm.Hour()%n, 0, 0, 0, tz)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hour-multiple bucket starts that fall in a DST gap disagree between Go and the engines. Go's time.Date can resolve a nonexistent local time to before the gap, while DuckDB, ClickHouse-style engines and luxon resolve it to the first instant after it.

Example: America/New_York on 2024-03-10 with a 2h grain, where local 02:00 does not exist. DuckDB's buckets and spine are at 05:00Z, 07:00Z, 08:00Z, 10:00Z and 12:00Z. Starting from TruncateTime and stepping with OffsetTime gives 05:00Z, 06:00Z, 07:00Z, 09:00Z and 11:00Z: there is an extra bin at 06:00Z (01:00 EST), and every bin after the gap lands on an odd local hour, because OffsetTime adds 2h to the resolved 03:00 instead of to the bucket's 02:00.

This matters most for the Go-generated spines in SelectTimeRangeBins for ClickHouse, Druid, Pinot, Snowflake and StarRocks, which then don't match the query's buckets for the rest of the range. It also adds a spurious null bucket in the legacy null fill. It affects the 2h grain in US zones and every hour multiple in zones whose gap starts at midnight, such as America/Havana. The existing DST tests use 6h, whose boundaries never fall in the gap, so they don't catch it.

A fix is to resolve gaps forward and to step OffsetTime from the bucket's grid hour:

// TruncateTime
h := tm.Hour() - tm.Hour()%n
tm = resolveGapForward(time.Date(tm.Year(), tm.Month(), tm.Day(), h, 0, 0, 0, tz), h)

// OffsetTime
h := tm.Hour() - tm.Hour()%k + n*k
tm = resolveGapForward(time.Date(tm.Year(), tm.Month(), tm.Day(), h, tm.Minute(), tm.Second(), tm.Nanosecond(), tz), h)

// resolveGapForward moves t, built by time.Date for the given local hour, past a DST gap.
// time.Date can resolve a nonexistent local time to before the gap,
// while the OLAP engines and luxon resolve it to the first instant after it.
func resolveGapForward(t time.Time, hour int) time.Time {
	if t.Hour() == (hour%24+24)%24 {
		return t
	}
	_, before := t.Zone()
	_, after := t.Add(3 * time.Hour).Zone()
	return t.Add(time.Duration(after-before) * time.Second)
}

With this change, the Go bins match DuckDB's for New York, Havana, Berlin, Lord Howe (30-minute gap) and Kolkata. All existing tests pass unchanged, including the DST resolver tests against ClickHouse and Druid, and a 2h case would be worth adding. By my reading, the Druid fromHour expression has the same problem and returns 06:00Z for an event at 03:30 EDT (I did not run this case against Druid), so it needs the same forward shift when the result's local hour differs from bucketHour.


Developed in collaboration with Claude Code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 00ac274, as you suggested with one change: OffsetTime returns the next bucket's start instead of keeping tm's minutes, since keeping them puts every Lord Howe bin after its 30-minute gap on :30.

  • Druid: your reading was right (03:30 EDT landed in 06:00Z on the test cluster). A gap start now moves forward to the bucket's wall-clock time, and comparing minutes instead of hours also fixes Lord Howe.
  • The DuckDB, BigQuery and Databricks spines now start from the bucket's wall-clock start; from a start after a gap they stepped from 03:00. plusGrain had the same issue as OffsetTime.
  • The fall-back mirror case: a repeated hour that starts a bucket (02:00 in Berlin at 2h, midnight in Havana) is labeled with its first occurrence by ClickHouse and Druid and its second by DuckDB. Go picked the second in eastern zones, so the right-joined spine dropped that bucket on ClickHouse and Druid; it now picks the first, and DuckDB builds its own spine.

Tests: golden spines for 2h New York, 6h Havana and 2h Berlin on DuckDB and ClickHouse and for 2h New York and 6h Havana on Druid, the legacy null fill at 2h, and timeutil cases. Go now matches ClickHouse's buckets on both DST days in eight zones for every hour multiple.

The ClickHouse cast of Date values to DateTime64 at sub-day grains now checks the base unit,
so it also covers minute and hour multiples, which toStartOfInterval rejects on Date values.
The canvas component spec from main mocks the time manager's availableGrainsMap, which this branch reads.
- TruncateTime and OffsetTime resolve an hour that starts in a DST gap to the first instant after the gap, as the engines and luxon do,
  and an hour that a fall-back change repeats to its first occurrence, as ClickHouse and Druid do.
  OffsetTime steps from the grid hour of the bucket, so Go spines and the legacy null fill stay on the local grid after a gap.
- Druid moves a bucket that starts in a DST gap forward to the first instant after it,
  and compares minutes of the day, which also fixes the 30-minute DST change of Australia/Lord_Howe.
- The DuckDB, BigQuery and Databricks spines start from the bucket's wall-clock start, so a spine that starts after a gap stays on the grid.
- plusGrain steps between bucket starts the same way in the frontend.
@nishantmonu51

Copy link
Copy Markdown
Collaborator Author

Merged main. The ClickHouse Date cast from #10100 now checks the grain's base unit, so minute and hour multiples on Date columns no longer fail in toStartOfInterval.

@pjain1 pjain1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The DST gap fix looks good, and everything I ran passes on 00ac274: timeutil, queries, the dialect tests, the DST resolver tests on ClickHouse and Druid, and the frontend time specs. I left one documentation note inline; it doesn't need a code change.


Developed in collaboration with Claude Code

// startOfLocalHour returns the first instant of the given hour on tm's date in tz; the hour may overflow into adjacent dates.
// time.Date does not specify which instant it picks around a DST change, so this pins down both cases:
// an hour that starts in a DST gap resolves to the first instant after the gap, as in DuckDB, ClickHouse, Druid and luxon,
// and an hour that a fall-back change repeats resolves to its first occurrence, as in ClickHouse and Druid, whose spines are built from it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No change needed here, but it would help to note a known limitation in this comment. DuckDB labels a bucket that starts at a repeated hour with the second occurrence, so the legacy timeseries null fill, which steps with OffsetTime, adds one empty bucket before the real one on fall-back days.

For example, in America/Havana on 2023-11-05 at 6h, MetricsViewTimeSeries on DuckDB returns 22:00Z=36, 04:00Z=null, 05:00Z=42 and 11:00Z=6. It only happens when the repeated hour falls on a bucket boundary: 2h in Central Europe, Australia and New Zealand, 3h in Eastern Europe, and every multiple in Cuba. It costs one extra empty point on one day a year, so I don't think it justifies another special case. A line here would save someone from rediscovering it.


Developed in collaboration with Claude Code

A new canvas defaults to the first metrics view with a time dimension by name,
so `AdBids_grains_metrics` took that place and the canvas setup e2e spec found a `total_records` KPI instead of "Sum of Bid Price".
Rename the time-grains fixtures to `AdBids_metrics_grains*` so they sort after `AdBids_metrics`.

Also format the explore time-grains spec, which failed the Prettier check,
and wait for reconciliation in the canvas time-grains spec, which flaked while the dashboard was still building.
The Korean catalog landed on main after these keys were added to `en` and `es`,
so the i18n guard failed on the missing `time_grain_n_minutes` and `time_grain_n_hours`.
@nishantmonu51
nishantmonu51 merged commit 1760e77 into main Nov 9, 2026
16 checks passed
@nishantmonu51
nishantmonu51 deleted the nishant/custom-time-grains branch November 9, 2026 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants