TL;DR: In the ClickHouse Native protocol, clients must wait for an EndOfStream packet to complete a query. Between the result Data packets and EndOfStream, ClickHouse sends several metadata packets (Logs, Progress, ProfileEvents). When combined with TCP_NODELAY and a reset TCP congestion window (cwnd) after an idle period, these unbatched packets can exceed the window, forcing an additional round trip. We reproduced this deterministically using Linux network namespaces and reduced tail latency by batching those metadata packets at the application layer.

One of our customers observed an additional ~32 ms of latency on simple SELECT queries between the client and the server.

In this customer's setup, the client and server are ~32 ms apart (egress). When a connection sat idle for 2 seconds or longer, the very next query took ~64 ms end to end: 2× the round-trip time (RTT) instead of the expected ~32 ms (1× RTT).

We had to work out what exactly was causing this additional round trip.

The investigation led to some interesting lessons about both the Linux TCP stack and the ClickHouse Native protocol.

ClickHouse server supports a wide range of protocols. HTTP and Native TCP are the two primary ways to talk to ClickHouse server. The customer's problem involves the Native TCP protocol, so that is our focus. It's important to know a little bit about how the ClickHouse Native protocol works to understand the customer's problem.

The ClickHouse Native protocol is a stateful protocol built on top of the TCP transport protocol. Client and server exchange different protocol messages in different phases (Handshake, INSERT or SELECT). The official specification for the TCP native protocol describes how it works in detail.

Here is a simplified version of how SELECT queries work at the protocol level.

Client sends Query (contains SQL, settings, flags, etc).
Server streams Data packets containing actual result rows.
Server also sends intermediate metadata packets (ProfileEvents, Progress, Logs).
Server sends EndOfStream (EOS) to mark that particular query is complete.

What this means for the client is that it has to keep reading from the connection until it receives the EndOfStream packet for that particular query, before that query is considered done.

TCP is the well-known transport-level protocol, and how it works is out of scope for this article. Here we focus primarily on two characteristics of the TCP protocol that are needed to understand the customer problem precisely.

Congestion window
TCP_NODELAY (Nagle's algorithm)

The congestion window (unlike the receiver window) is neither configured on the client side nor on the server side. It's a value that usually starts small (based on initcwnd, the initial congestion window value in Linux) and keeps increasing as long as the network between client and server doesn't face any congestion problems (for example, the server can send that number of TCP packets before waiting for an ACK from the client). This is commonly known as TCP slow start.

Basically, TCP limits the amount of unacknowledged data in flight using this congestion window (cwnd).

The initcwnd value is usually 10 MSS (Maximum Segment Size), but sometimes differs. In the customer's case it was 8 MSS.

One crucial thing to notice about the congestion window is that it keeps growing, and if the connection is idle for some time (e.g. 2 seconds in the customer's case), the Linux TCP stack resets the current congestion window size back to the initcwnd value. This idle time is called the Retransmission Timeout (RTO), and it is calculated dynamically based on the RTT of the network, with lower (200 milliseconds) and upper (120 seconds) bounds in the Linux TCP stack.

Given its age, one of TCP's default behaviors is to batch socket writes before flushing them to the network, to reduce network overhead. This is often called Nagle's algorithm, and it happens inside Linux's TCP stack, where the application has almost no control.

Modern low-latency systems (like ClickHouse) disable this by setting TCP_NODELAY=1, instructing the kernel to send socket writes immediately.

The catch here is that if TCP_NODELAY is enabled and an application issues multiple socket writes for those metadata packets (ProfileEvents, Log, Progress), each of those writes becomes an individual TCP segment on the wire.

If the server issues more small writes than the reduced cwnd can hold, the kernel must stop and wait for a client ACK before sending the remaining packets, incurring an extra round-trip penalty.

Reproducing the customer problem requires simulating the same network RTT between the client and the server (~32 ms) to make the problem visible. For simplicity, we run all the queries (with an idle period of 2 seconds) on a single TCP connection over the Native protocol between the client and the server.

Two pieces are involved here:

A Go program that runs actual queries to observe the problem the customer faced (additional ~32 ms tail latency)
A ClickHouse server within a network namespace with the customer's network conditions applied to it (simulating ~32 ms network egress delay).

We could have used Docker or Podman to create the network namespace. However, we deliberately set it up manually using netns to make the network simulation easier to understand.

We then used tc (traffic control) to simulate a 32 ms WAN delay between the client and the server. We also configured the initial TCP congestion window to 8 MSS (initcwnd 8).

Have a ClickHouse binary (< 26.9) (the one we use to reproduce the issue)
Create a network to simulate the customer infrastructure (client and server ~32 ms apart).
Verify the simulated network (you should see ~32 ms)

This Go program runs simple queries over an established connection with a 2-second idle period between the queries. We also separate overall query latency and tail latency (the time between the last Data packet and the EndOfStream packet) to observe the tail latency (both before and after the patch) more clearly.

The Go program runs N identical SELECT max(id) queries on one reused connection and splits each query's latency into:

first_row (the query result itself) and
tail (first row → end of stream).

tail is the key here: healthy ≈ 0 ms; stalled = exactly 1× RTT.

Run the server inside the simulated network
Create a table and rows for the experiments
Run the Go program to see the issue in action

You should see results similar to the following:

Observation: The first row arrived in ~33 ms (1× RTT), but the connection stalled for another ~32 ms (tail latency) waiting for the EndOfStream packet train.

The additional RTT (~32 ms) comes from the fact that once the congestion window is reset (from a large value to initcwnd, which was 8 in the customer's case) after a 2-second idle period, the server cannot send all the remaining packets (Data + those metadata packets) in a single round trip.

The server sent 8 TCP segments, ran out of cwnd space and paused until the client ACKed the first batch, costing an additional 32 ms full round trip.

There are some workarounds to make this problem less likely to happen:

Disable ProfileEvents on the server side (ProfileEvents packets are one of the causes of the large number of TCP packets on the wire)
Disable resetting the connection's congestion window (cwnd) to its initial value (initcwnd) after an idle period (tcp_slow_start_after_idle=0) on the server node (GKE in the customer's case; it's also recommended for high-performance TCP by Google Cloud)
Increase initcwnd (default is 10, the customer's case is 8) on the server node
Improve the ClickHouse Native protocol to create fewer TCP packets with application-level batching.
Use the TCP_CORK option instead of TCP_NODELAY

Unfortunately, in this case the customer had little control over workarounds (1), (2) and (3), so the only viable options were (4) and (5). Option 5 (TCP_CORK) sounded interesting, but it has a few drawbacks:

The kernel TCP stack still has full control over filling those TCP segments
It is specific to the Linux kernel

Option (4) also sounded like a nice improvement on the server side anyway. So we decided to make the ClickHouse Native protocol create fewer TCP packets with application-level batching at the end of the query.

The patch is simple: we group these metadata packets at the application level into a single socket write (hence potentially fewer TCP packets), which can then be sent in the minimum number of round trips even when the congestion window is reset.

There is one catch: we cannot blindly group all the metadata packets, as some packets like Progress are time-sensitive and have to be delivered to the client immediately (imagine the client wants to know the real progress of the query, and server-side batching delays it, communicating the wrong progress state).

We can validate that our batching is working by looking at the captured TCP packets: the total number of TCP packets the server creates for the same query is almost halved. That removes the ~32 ms of additional tail latency we originally observed.

Assuming ClickHouse server runs on port 9000.

If you run the same Go script while capturing the TCP PUSH packets (PSH, the ones where actual application data gets transferred) via tshark:

Even before looking at the captured pcap files, we can see it captured a total of 50 such TCP packets from server to client.

Also notice the many tiny TCP segments:

The same Go script, capturing on the patched server with batching:

You can clearly see the number of TCP packets transferred is half what it was before.

Looking at the captured packets clearly shows the PSH packets are much more "filled" compared to the many tiny packets before.

Running the same script with exactly the same setup as described in reproducing the exact customer problem shows how the ~32 ms tail latency disappeared:

Note that no additional round trip is needed for the server to send EndOfStream (eos) to the client.

High tail latency isn't always caused by slow database engines or long-running queries; sometimes the culprit is hidden in the interaction between application-level flushes and kernel-level TCP behavior.

Idle TCP connections reset: After an RTO idle period, the Linux TCP stack shrinks cwnd back to initcwnd, restricting how many unacknowledged packets can fit into a single round trip.
TCP_NODELAY has a trade-off: Disabling Nagle's algorithm makes individual writes fast, but issuing multiple small write() calls can create an unexpected number of TCP segments that can quickly exhaust a small congestion window.
Batch at the application layer: Grouping metadata packets (ProfileEvents, Logs, Progress) into a single buffer write before flushing drastically reduces the TCP segment count, keeping tail latency predictable even on idle, high-RTT connections.