“A journey of a thousand miles begins with a single step.”— Lao Tzu, Tao Te Ching, chapter 64
Last month we introduced CAS – a new S3-storage for ClickHouseMergeTree tables. Since then we have been working on improvements and released three updates. Altinity customers are also testing CAS with the real data. In this article we will benchmark CAS and regular S3-backed MergeTree head by head in several typical scenarios. We will show that CAS is a drop-in storage replacement for S3-backed MergeTree that significantly reduces the storage cost.
Setting up the stage
For this test, we will set up a two-replica cluster in AWS running c8g.2xlarge nodes and configure both S3 and CAS disks. The table below shows both configurations side by side. The configurations are almost identical, the only difference is metadata_type and cas_server_root_id tags required to turn on the CAS.
<s3_cache> <type>cache</type> <disk>s3</disk> <path>s3_cache/</path> <max_size>50Gi</max_size> </s3_cache> </disks> </storage_configuration></clickhouse>
We can check system.disks and system.storage_policies to confirm that disks and policies are in place:
As you can see, setting up a CAS disk is as simple as a regular S3 one.
Once disks are configured, we can start testing data loading and query performance. We will using two Antalya builds for this purpose:
- 26.6.8.20002.altinityantalya – this is the latest available public build at the time of writing
- 26.8.34.20001.altinityantalya – this is a development version but it should be released in a two weeks
Single-tier Inserts
First, we will create tables that are fully stored on the object storage using default ClickHouse settings. This is an extreme case. The primary use case for S3-backed MergeTree (both the regular and CAS implementations) is a tier where data is moved using TTL jobs. But the extreme test allows us to highlight different characteristics of the approaches that otherwise could be overlooked.
For example, we create tables and load data from a public “ontime” dataset as follows:
Similarly for a CAS table:
Here are the results, showing time in seconds for various load scenarios:
As you can see, CAS is significantly slower on inserts. We will explain that when analyzing the S3 workload below.
Data size as reported by ClickHouse is identical with a very tiny improvement for CAS:
It gets dramatically different when looking into S3 bucket though:
- clickhouse/cas-demo2 (regular S3): 29.8GB and 108140 objects
- cas/cas-demo2 (CAS): 14.9GB and 52768 objects.
So CAS stores a single data copy and also reduces a number of stored objects!
Multi-tier Inserts
The current primary use case for S3-backed MergeTree (both the regular and CAS) is a tiered configuration, where hot data is inserted to local storage, and later on moved to the cold tier using TTL jobs. So let’s create additional storage policies first:
Note that we disabled perform_ttl_move_on_insert. We want to make sure the data lands on default local storage volume first, even if the corresponding partition is subject to TTL.
Now we will create a new set of tables using tiering for S3 and CAS tables. Here is an example for CAS, S3 is identical:
The insert will go to a default disk, but since the data is older than one year, it will start moving to S3 or CAS disk in the background. So this test is a bit of an extreme case as well, in real applications background moves will trigger much later.
Here are insert timings for all 3 tests so far:
As you can see, inserting into a hybrid table is much faster. Here CAS is ramping up but still slower. It almost reaches parity with a regular S3 on the latest build.
The reason for slower performance in tiered table insert is metadata global locking when moving parts in the background that affects inserts to non CAS disks as well. In the 26.6.8 build, PartsLockWaitMicroseconds measured 150s for CAS! It has been reduced to 7 seconds in 26.8, which explains the improvement. There is a bug report #2772, so we expect that it can be improved further.
S3 workload
In order to understand why CAS disk is slower on inserts, let’s extract relevant statistics from the query log. We used special log_comment markers in the queries, which make it easier to find them:
CAS writes data slightly faster (S3WriteMicroseconds) and generates a smaller number of objects (S3PutObject) due to merging small files into manifests. It also reduces cost, since PUT requests are the most expensive ones.
However, CAS also does a lot of GET (S3GetObject) and HEAD (S3HeadObject) operations that slow down insertion significantly.
We are already working on improvements here that should reduce the number of S3 operations. CAS is expected to be slower on writes since it keeps and synchronizes state into S3 bucket rather than Keeper, but the difference is not going to be significant.
Query performance
Since the underlying implementation is MergeTree in all cases, the query performance is pretty much the same on hot data when data is cached on a local disk. S3 and CAS disks show almost identical numbers, and match MergeTree on a local disk as well. The main difference is cold data performance. We benchmarked queries using different approaches, including popular ClickBench. Here is a summary of ClickBench results showing total execution time for cold and warm runs:
In a nutshell, you may expect the same query performance and apply the same caching techniques to CAS-backed tables as for regular S3 ones. CAS may have a slight edge on a cold run because of packing several small files together in a manifest. However, 26.8 has introduced a new ‘Packed’ part storage type that may leverage the difference. We have not tested it yet.
The big improvement will come from a distributed cache, reused by replicas. This is in the CAS development roadmap for 2027.
Summary
In this article we compared regular S3 and CAS performance in typical end-users scenarios – from configuration, to data loading and queries. CAS is a drop-in replacement for regular S3 disks in ClickHouse. There is a lot to be improved in CAS direct writes. In the tiered storage, however, performance is almost on par with regular S3 in the latest Antalya builds, because files merge in the hot tier before upload to S3. Query performance is also identical to regular ClickHouse S3 tables.
The main advantage of CAS is cost. Since it stores a single copy of the data it is 2-3 times cheaper than regular S3 for typical ClickHouse installations. Moreover, it does not have the problem of orphaned objects left on S3 buckets, which significantly increase the cost of operating production ClickHouse clusters with regular S3 object storage.
It is important to note that migration from local storage or regular S3 to CAS storage is totally transparent. It only requires a policy change and ALTER commands. We will discuss it separately.
In the next article, we plan to talk about the cost in more detail. Stay tuned!
Join our Slack
ClickHouse is a registered trademark of ClickHouse, Inc.; Altinity is not affiliated with or associated with ClickHouse, Inc.
Get the latest ClickHouse news straight to your inbox every month.












