Acumatica · Bigquery

BigQuery Performance Tuning — A Field Guide

BigQuery Performance Tuning — A Field Guide is the work that turns raw data into decisions. The pipeline from "we have data" to "we have a model that runs in production" is the.

John Kihiu12 min read

BigQuery performance problems almost never come from BigQuery being slow — they come from queries that scan far more data than they need to, or from a schema that never had partitioning or clustering applied. Because BigQuery bills (and throttles, if you're on flat-rate slots) by bytes scanned and slot-time consumed, the tuning story is really a cost and concurrency story wearing a performance costume. Fix the scan pattern and the "performance problem" usually disappears.

Partitioning first

Partitioning splits a table into segments — most commonly by a DATE or TIMESTAMP column, or by an integer range — so that a query with a filter on the partitioning column only reads the partitions it needs instead of the whole table. A table partitioned by DATE(created_at) queried with WHERE created_at >= '2026-07-01' will prune everything before that date at the metadata level, before any bytes are scanned. The most common mistake is filtering on a derived expression that hides the partition column from the query planner, such as wrapping it in a function that BigQuery can't use for pruning — always filter on the raw partitioning column directly where possible.

Clustering on top of partitions

Clustering sorts data within each partition by up to four columns you specify, and BigQuery uses that ordering to skip blocks that can't match your filter — similar in spirit to an index, though it works by physically co-locating similar rows rather than maintaining a separate structure. Clustering pairs well with high-cardinality filter or join columns, like customer_id or order_id, that you can't partition on directly because partitioning has a limit on the number of partitions a table can have. A table partitioned by day and clustered by customer_id gives you fast date-range scans and fast per-customer lookups within those ranges.

SQL · PARTITIONED AND CLUSTERED TABLE
CREATE TABLE analytics.orders
PARTITION BY DATE(created_at)
CLUSTER BY customer_id, status
AS
SELECT * FROM staging.orders_raw;

-- Prunes to one partition, then skips clustered blocks
-- that don't match customer_id.
SELECT order_id, total
FROM analytics.orders
WHERE DATE(created_at) = '2026-07-20'
  AND customer_id = 48213;

Reading the query plan and bytes scanned

Before optimizing anything, look at the bytes-processed estimate the BigQuery console shows before you run a query, and check the execution details after — specifically slot-time consumed and the per-stage breakdown of records read versus records written. A query that reads billions of rows to return a hundred is almost always missing a partition filter or is doing a SELECT * where a handful of columns would do. BigQuery is columnar, so every column you select is bytes you pay for regardless of whether you filter rows down afterward.

SELECT * is the single biggest cost lever

Because storage is columnar, SELECT * reads every column in every scanned partition. Naming only the columns a query actually needs is frequently a bigger win than adding clustering, and it costs nothing to try.

Slots, concurrency, and on-demand vs. flat-rate

BigQuery executes queries using slots — units of computational capacity. On the on-demand pricing model, slots scale automatically and you pay per byte scanned; under flat-rate (or the newer editions-based reservations), you purchase a pool of slots and every concurrent query competes for that fixed pool. If dashboards start queuing during business hours, that's usually a slot contention problem, not a query problem — check the INFORMATION_SCHEMA.JOBS views for queries with high wait times relative to execution time, which points to contention rather than inefficiency.

Materialized views vs. scheduled queries

For a dashboard hammering the same aggregation repeatedly, a materialized view keeps a precomputed, incrementally refreshed result and can dramatically cut both latency and scan cost. Don't reach for a scheduled query that just re-runs the same expensive aggregation on a timer when a materialized view solves it with less operational overhead.

Joins and data skew

Large joins in BigQuery can suffer from data skew — one join key with a disproportionate number of matching rows forces a single worker to handle far more data than the rest, stalling the whole query. This shows up as one stage in the query plan taking dramatically longer than others. Broadcasting a small dimension table (letting BigQuery replicate it to every worker instead of shuffling the large table) is often the fix, and BigQuery does this automatically for genuinely small tables — the problem case is a "small" table that's actually a few million skewed rows.

Wrapping up

Most BigQuery performance work is unglamorous: partition by the column you filter on, cluster by the column you join or filter on next, select only the columns you need, and watch bytes-scanned before you watch query duration. The expensive query is rarely the one with a bad algorithm — it's the one reading data it never needed to touch.

John Kihiu
Acumatica ERP Developer · Laravel Engineer

Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.