Teams use RudderStack, an agentic, warehouse-native customer data platform, to collect, govern, unify, and activate their customer data. RudderStack’s core application gives data teams all the tools and pipelines they need to build a solid data foundation around their data warehouse. Rudder Lookout makes all the data from that foundation available to marketers via an AI native application for analyzing, building, and activating audiences. Today I’m going to focus on the foundation, explaining our approach to Infrastructure as code and why we support both Terraform and our own CLI.

Two paths for IaC with RudderStack

Customer data pipelines often start simple, connecting a web or mobile application to a warehouse like BigQuery or Snowflake, but they rarely stay that way. Transformations get added to enrich, filter, or apply custom processing to events before delivery. Tracking plans, powered by extensive data catalogs, are used to validate and reject events at the source. Connection-level settings get tuned to control how each pipeline interacts with its downstream systems.

Configuration at this scale becomes production infrastructure. Changes need review and approval before they’re applied. Pipelines have to be tested in a staging environment. When a configuration bug is discovered in production, pipelines need to be reverted to a known good version. The solution to this is well understood by the industry and involves an infrastructure as code (IaC) approach: define configurations in files, commit those files to a version control system like Git, and use a tool to apply changes to each environment.

Terraform is a widely adopted tool for this workflow and RudderStack has had an official Terraform provider for years. We remain committed to maintaining it and expanding its coverage.

At the same time, we decided to build Rudder CLI, our own IaC tool alongside our Terraform provider. We believe that a specialized tool can provide a better user experience compared to a generic tool like Terraform. It can encode information, rules and workflows that would be difficult to model in Terraform.

In this post, we present three examples of how we leverage this to provide an improved experience: no dependency on state files and atomic resource associations, validation across the entire pipeline configuration, and support for a development workflow that goes beyond applying configuration.

CLI advantage one: The backend is the state

Terraform requires a state file to keep track of applied changes. The most important part of this state is the association between resources defined in Terraform configuration files and resources instantiated in remote systems. While state includes more information than that, such as their latest known configuration, this association is what allows Terraform to know which resource to update when a configuration changes and which one to delete when a corresponding configuration block is removed.

Teams are responsible for that storage, typically an S3 bucket. And because this state file is so critical to Terraform operations, it needs backups, protection against corruption, and encryption. Access to it must be restricted. This adds a maintenance overhead to teams but there is also one more danger: Changes to resources can never happen atomically with changes to their state. If Terraform creates a resource but fails before recording it in the state file, that resource becomes an orphan.

A state file makes sense for a general purpose tool. Terraform supports a wide variety of systems and cannot assume that every system can store the association between a remote source and its configuration declaration.

Rudder CLI, being a tool specialized to manage RudderStack resources, avoids this problem. RudderStack controls both the CLI and the backend APIs and can store this association natively.

Each resource in our backend gets an external id property which Rudder CLI populates with the identifier a user chose in their configuration files. APIs that create resources receive this external id in the same request body as the rest of the resource’s properties. Our backend persists both the resource and its association to the Rudder CLI identifier atomically.

Rudder CLI gets an inventory of all managed resources by simply asking the API for those that have a populated external id, and uses those responses to reconcile. There is no need for an external state file because the API provides everything state related that Rudder CLI needs. With no state file there is no danger of a state deletion or corruption.

CLI advantage two: Validations across the entire resource graph

Terraform maintains a resource graph when one resource uses another resource’s property, usually its remote identifier, and uses it to order the plan. This resource graph is not validated as a whole. Providers validate each resource independently, so they don’t check if a resource is associated with an incompatible one. They can’t validate constraints such as a name property being unique for the same resource type. Users are responsible for writing configurations that satisfy those broader constraints, and failing to do so surfaces as an error during apply. This can leave an environment partially synced until the configuration is fixed and applied again.

Rudder CLI validates the whole project against RudderStack’s own rules, either as part of planning or as a standalone command. Having access to the entire resource graph, it can apply rules that span resources. For example, consider a Google Ads destination, which can be connected only to web sources in device mode. Validating this rule needs to consider the source, destination and connection resources. A configuration that defines a connection to an Android source will report an error against the offending line in the project files:

error[event-stream/connection/semantic-valid]: destination 'google-ads' (type 'googleads') does not support source 'android-app': source type 'android' is not among supported source types: web

--> connections/event-stream.yaml:9:7

9 | destination: "#destination:google-ads"

With Terraform this rule cannot be expressed: a connection resource has only identifiers of the connected source and destination, not their types and properties. The same mistake will cause apply to fail because RudderStack’s API will reject that configuration, requiring a fix and a retry.

CLI advantage three: Beyond applying configurations

Managing a data pipeline is not only a matter of creating the right resources. Consider a fairly ordinary setup: an Android application sends events through an event stream source, a tracking plan governs them, a transformation enriches them, and everything is delivered to Snowflake. Setting up this pipeline in a RudderStack workspace is what IaC tools are good at but it’s only part of the work. Instrumenting the application properly and validating the transformation logic are examples of tasks required to have a healthy pipeline. We are building Rudder CLI to help with more of that workflow than syncing alone. Rudder CLI projects capture the desired state of a pipeline and can be used for much more than that.

For example, Rudder CLI exposes list commands to browse resources currently present in a workspace. Its typer command can generate typed clients from the same tracking plan configurations that are used to instantiate a pipeline, making a CLI project the source of truth for both the state of the pipeline in a RudderStack workspace and the instrumentation clients embedded in an application. It supports transformation tests, linked to the same test configurations, that can be used to verify transformation logic before it is applied to production pipelines.

These are capabilities that are outside the scope of a Terraform provider, yet they leverage the same configuration files. This becomes even more important in agentic workflows where an agent can reason about a pipeline setup by looking at project files and using a single tool to work with that information. It can write configurations for sources, destinations, and connections, write transformation code along with its tests and execute them, compose tracking plans and instrument applications accordingly, all by working on one CLI project and using one tool.

Where each tool fits

All three examples come from the same fact: Rudder CLI only has to understand one API, so it can encode things a generic tool has nowhere to put. That is not an argument against Terraform. If your team already manages its infrastructure as code with Terraform and wants a single workflow across all of it, our provider remains a first-class part of our IaC toolkit and we are continuing to maintain and extend it.

Rudder CLI is the better fit when you want your pipeline configuration checked as a whole before it is applied, and the same project to carry your instrumentation and transformation logic alongside the resources it defines.