Announcing Server-Side Scan Planning with Starburst and Databricks

We are excited to announce that the Starburst Enterprise 481-e STS now includes support for server-side scan planning, which has been added to the Iceberg connector through the REST Catalog API. This means that any Iceberg REST catalog that implements the /plan endpoint can now plan scans on behalf of Starburst, returning the exact data files to read without Starburst needing to process Iceberg metadata files. The engine executes the plan as-is, and Starburst respects vended credentials included in the plan response.

While Starburst supports server-side scan planning with any REST Catalog, the most compelling initial integration is with Databricks Unity Catalog. Unity Catalog can now enforce fine-grained access control, including row filters and column masks with governed tags, for managed Iceberg tables on Starburst using server-side scan planning. We’ve worked closely with Databricks to ensure that Unity Catalog policies are respected when Starburst queries a Unity-governed Iceberg table, allowing governance to be centralized on Unity Catalog with no bespoke logic in the query engine.

What is Server-Side Scan Planning?

The Iceberg REST spec added POST …/tables/{table}/plan in Iceberg version 1.7.0. Instead of having the Starburst query engine read metadata files and plan the scan itself, the client sends selected columns and filter expressions to the catalog server through this endpoint. The catalog returns a list of file scan tasks and scoped storage credentials, which the engine executes as usual.

This small protocol change gave the catalog a hook into the read path of a query. Now, the catalog can decide what files an engine sees, apply transformations, vend short-lived credentials scoped to the specific files in the response, and audit file access. None of this requires our engine to understand Databricks’ policy model, but policies are still respected when data is returned to the caller.

For Starburst, the value of supporting this protocol is that it works across the entire Iceberg REST ecosystem, and will work out of the box as other catalogs implement this endpoint.

Unity Catalog’s Fine-Grained Access Control

Databricks Unity Catalog uses /plan to implement “ScanAPI”: an endpoint to enforce fine-grained access control for external engines. With ScanAPI, when a Starburst query reaches a Unity-managed Iceberg table that carries active FGAC policies, Unity:

• Evaluates the caller’s ABAC policies against the user’s identity.
• Materializes sanitized data into temporary data files with row filters and column masks already applied.
• Returns file scan tasks pointing at those sanitized files, plus scoped credentials.

The engine only sees data authorized for that user. The policy was defined once, in Unity Catalog, and applies whether the query runs in Databricks or Starburst.

This addresses the governance gap many lakehouse customers encounter. Unity Catalog already provides table-level access control across engines via credential vending, but credential vending is coarse: you have access to the whole table, or you don’t. Real-world governance needs more than that: PII column masks, regional row filters, role-driven views of the same data, etc. Replicating that policy in every engine doesn’t scale. Server-side scan planning lets Unity govern data centrally, and engines like Starburst can honor those decisions using ScanAPI.

A Real-World Example

Want to know more about how this works in reality? Check out this video by Databricks.

What’s Next?

Server-side scan planning support ships in Starburst Enterprise Platform 481-e STS. While the Unity Catalog integration is the most fully developed example today, Starburst will support any other REST catalogs that implement this endpoint.

Overall, this represents a stepping stone to a shared policy language. This mirrors other initiatives in the industry toward similar goals, notably Snowflake’s approach towards an open-sourced way of governing tables by certain trusted compute engines.

For more on the protocol and the joint Databricks/Starburst work, see Databricks’ cross-engine ABAC announcement and our earlier post on Starburst integration with Unity Catalog.