BigQuery is Google’s fully managed, serverless data warehouse that enables scalable analysis over petabytes of data. Google recently announced upcoming changes to BigQuery’s commercial model and in this article, we'll summarize and explain all you need to know about it.
On July 5th, BigQuery will change not only in terms of pricing but in terms of service. The flat-rate model is being replaced by BigQuery Editions.
Introducing BigQuery Editions
To support customers with their flexibility needs, Google introduced three editions (Standard, Enterprise, and Enterprise Plus) to support different types of workloads. Each edition provides a set of capabilities at a different price point to match the requirements of different types of customers. In addition to the three pricing tiers, they will continue to offer the on-demand pricing option that allows customers to pay for data processed.
Source: Google
Please note that if you plan to keep using the on-demand (pay-as-you-go model), there will be a 25% increase in the cost of your queries.
Announcing BigQuery Autoscaling
Autoscaling with BigQuery dynamically adjusts the capacity in response to planned or unplanned changes in demand to ensure you pay only for what you use. You can set up maximum and optional baseline compute capacity, and let BigQuery take care of provisioning and optimizing compute capacity based on usage without any manual intervention on your part. This ensures you get sufficient capacity while reducing management overhead and underutilized capacity.
Source: Google
You can start using editions immediately by navigating to “Capacity Management” in the BigQuery console. Starting on July 5, 2023, BigQuery customers will no longer be able to purchase flat-rate annual, flat-rate monthly, and flex slot commitments
Adding BigQuery Compressed Storage
As data volumes grow exponentially, customers find it increasingly complex and expensive to store and manage data at scale. Unlike with flat-rate pricing, editions will allow you to use the physical storage option which in many cases may be several times smaller.
Physical storage cost is double the logical storage ($0.04 instead of $0.02 per GB) but with compression rates up to even 12:1 it can reduce your storage bill significantly
Timeline and what to expect
Source: Google
Frequently Asked Questions
Do all workloads in the same project have to be on the same edition?
All queries run in a project are assigned to the same reservation. A reservation only has one edition.
Can we assign a project to two reservations with different editions?
No. An edition can have multiple reservations created in it, but a reservation can only exist in a single edition.
Can customers share slots across two editions?
No. Idle slot sharing is only possible within a given edition.
Can customers upgrade or change between editions?
A reservation can be upgraded to any higher edition. Users cannot change from a higher edition to a lower edition or from no edition to BigQuery Standard edition (only from no edition to Enterprise or Enterprise Plus).
Can customers mix different pricing models?
Customers can mix workloads across on-demand analysis and editions reservations, but they do not share the same capacity pool.
Are discounts available for longer term slot commitments?
BigQuery Enterprise and Enterprise Plus editions offer optional committed use discounts. 1-year and 3-year capacity commitments are available.
Do capacity commitments of BigQuery editions offer any special pricing for add-on products, like BigQueryOmni, BigQuery ML, BigQuery Log Analytics, or others?
Capacity commitments secure a discount for BigQuery editions compute only and do not affect the prices of other products.
How does this change affect storage pricing?
BigQuery editions do not affect storage pricing.
How is BigQuery Autoscaler different from flex slots?
Flex slot reservation requests will only succeed if the requested capacity is available, otherwise the reservation will fail and you have to try again with a request for a smaller number of slots. And because this capacity is not guaranteed at the time of request, customers will pre-purchase flex slots in advance of expected usage surges. This creates operational overhead for customers and additional cost—in the form of engineers and unused capacity. In contrast, autoscaling is managed by BigQuery, so slots are allocated as they are needed and based on workload demand. Customers can set an (optional) baseline and maximum slot capacity needed, and the Autoscaler will optimize without additional manual intervention needed. There is less operational overhead and no wasted capacity.
What happens if my query needs more than the set max slots? Will the query error out?
No, the query will just take longer to complete.
Can new customers still use on-demand pricing?
Yes, new customers can opt to use on-demand pricing (byte scanned model), but autoscaling is not available for these workloads.
Can new customers purchase and use flex slots?
Flex slots will be available for purchase until July 5, 2023, at which time flex slot reservations will auto-migrate to BigQuery Enterprise edition baseline.
Still have questions regarding BigQuery updates or you want to evaluate the actions to be prepared for this change?
Do not hesitate to contact us and our team of experts will be happy to assist!
