Why Transparent Huge Pages got a bad name
The CPU caches virtual-to-physical address mappings in the Translation Lookaside Buffer (TLB). When a mapping is not in the cache, the CPU takes a TLB miss and spends cycles looking it up. With normal pages, one TLB entry covers 4 KB. With huge pages, one entry covers 2 MB, or 512 times as much memory. A JVM with a large, hot heap benefits the most from huge pages because it takes far fewer TLB misses and loses fewer CPU cycles.
THP earned its bad reputation from two defaults on older systems. Linux kernels before 4.6, released in 2016, set defrag to always, and several enterprise distributions shipped with enabled=always:
- enabled=always gave 2 MB pages to every anonymous mapping on the host, whether the process asked for them or not. A process that touched one byte of a large region could be charged a full 2 MB page. That is where the memory bloat stories came from.
- defrag=always made a process that wanted a huge page wait in direct reclaim and compaction until the kernel found one. Those waits became latency spikes, and “disable THP” went into runbooks (kernel docs).
Here is how the three common configurations compare:
What the customer was worried about
The customer had run THP in always mode years earlier, hit exactly those problems, and moved away from it. Pushback came from three groups: the compute team that owns kernel settings, the platform team that owns the host groups, and the database team running the benchmarks. Every concern was reasonable, and every one was shaped by the old defaults.
“We tried this before and it hurt.” They had, with always. The opt-in setting, madvise, gives a process 2 MB pages only for the regions where it explicitly asks. Only mappings that ask for huge pages get them.
“This is a host-wide setting. What about everything else on the box?” The platform team noted that the change would land on whole host groups, not just the nodes under test. That is true of where the setting applies and false of how it behaves. With madvise, only a process that calls madvise(MADV_HUGEPAGE) on a mapping gets huge pages for it. Go services do not opt in by default, and neither does every JVM. On this fleet, the process that asked was Azul Prime.
“Won’t memory go up?” For Azul Prime services, slightly. The heap is aligned to 2 MB boundaries instead of 4 KB, so expect a modest increase in resident memory, in the range of 2 to 5%, at the edges of heap regions. The payoff is CPU and tail latency. Services that do not opt in cannot grow from this setting, and in the data we reviewed they shrank.
“Our ARM fleet runs a 64 KB-page kernel. Huge pages there mean much bigger pages.” Correct. With 64 KB base pages, a huge page is 512 MB, so the request was split by architecture. Azul Prime does not use the shared-memory THP path on ARM64. Instead, it uses tagged heap addresses (-XX:+UseTaggedAddressForJavaHeap) with ordinary anonymous THP. The option has been available as an opt-in since Azul Prime 25.02.0.0 and has been on by default on ARM64 since 26.08.0.0. It supports heaps up to 512 GB; above that size, Azul Prime turns it off automatically and logs a warning.
The ARM request was narrower: madvise for anonymous memory and defer for defrag, with shmem_enabled left alone (Azul docs).
“Kernel rollouts are slow, and we can’t do this per service.” Also true. Fleet-wide kernel settings went out with a 48-hour soak between zones; platform owners kept overrides in their own host groups, and the change needed sign-off from teams that had never heard of such an optimization possibility. From the infrastructure side, the setting is easy to roll out because nothing visible changes on the hosts. The benefit shows up inside the applications that use it.
What happened when they rolled it out
The Azul Prime heap was never the fragmentation risk. Fragmentation is a problem when a process maps a large region, touches a sliver of it, and gets charged for 2 MB. The Azul Prime heap on x86_64 is the opposite shape. It is committed up front at startup and, with -Xms equal to -Xmx, is never handed back to the OS. Azul Prime already carves it into 2 MB blocks, so backing it with 2 MB pages adds no internal fragmentation that was not already there. On x86_64, the C4 collector also multi-maps that heap through Linux shared memory, which is why the separate shmem_enabled setting matters: ordinary anonymous THP never reaches a shared-memory mapping (Azul docs). Azul Prime does the asking itself. When it detects THP on the host, it calls madvise on the Java heap and some off-heap structures, with no JVM flag needed. The host makes huge pages available, and the runtime that can use them opts in.
The surprise was memory. On hosts that had still been running always, a JVM service that was not running Azul Prime, and so had never opted in, stepped down about 5% in resident memory within one to three days of the rollout, roughly 3 GiB per container at the 99th percentile, from about 57 GiB to about 54 GiB, and stayed there. Under always, every mapping on the box had been padded out to 2 MB pages. Once madvise removed the padding, the services that had never asked for huge pages shrank the most. The Azul Prime services were expected to hold steady, since their heaps keep their huge pages. One caveat for anyone checking this on their own fleet: on x86_64, top and ps over-report resident set size (RSS, the memory a process holds in RAM) for Azul Prime because the heap is multi-mapped. Compare proportional set size (PSS, which divides shared pages among the processes sharing them; smem reports it) or /proc/meminfo rather than per-process RSS (Azul known issues).
Same JVM service, three datacenters: p99 per-container RSS stepped down 4.7–5.4% within one to three days of hosts moving THP from always to madvise. Customer production metrics, anonymized.
The savings took patience to show up. The drop appeared one to three days after the rollout, datacenter by datacenter. Changing enabled affects only future allocations. The kernel does not break up huge pages that already exist, so the savings arrive as old memory is recycled through restarts and autoscaling. The clean check: services that restarted before the change came back at the old memory level, and services that restarted after it came back at the new, lower level.
What was measured, and what was not. The measured outcome of this rollout is memory. The customer’s CPU and tail-latency benchmarks on the Azul Prime services were still in progress when this was written, so this post does not report them. For scale, Azul’s documentation puts the gain from 2 MB pages on the Azul Prime heap at 0 to 15% in median latency and CPU, with about 10% common across real customer workloads and little or none on microbenchmarks (Azul docs).
How to enable it
This applies to Azul Prime running without Azul Zing System Tools (ZST), optional since Azul Prime 19.07 and needed only for heaps over 2.5TB and older operation systems. In ZST mode, Azul Prime brings its own large-page implementation, and none of this applies. Details are in the Azul docs and the kernel admin guide.
1. Check that the kernel supports shared-memory THP (x86_64). The file should list all six policies. If it prints fewer, the kernel is too old for this path.
2. On x86_64, set four values. Opt in for anonymous memory, opt in for shared memory, compact in the background, and let khugepaged keep working. Azul’s docs list defer+madvise as an alternative when other tuning on the host requires it, but it brings synchronous compaction back for the regions that asked for huge pages, which includes the Azul Prime heap, so prefer defer (kernel docs). Do not set defrag to always.
3. On ARM64, skip the shared-memory line. Use the same commands without shmem_enabled. Do not pass -XX:+UseLargePages on ARM64, because it takes priority and disables the tagged-address path.
4. Make it survive a reboot. Use sysfsutils on Ubuntu, a tuned profile on RHEL, CentOS, Oracle Linux, and Amazon Linux 2, and a systemd oneshot unit on Amazon Linux 2023. Record the host’s current values first. If some already match and others do not, something else tuned that box, so merge rather than overwrite. Most distributions, and the kernel itself, default defrag to madvise, which can pause a Java process in direct compaction, so set it to defer on every host.
5. In Kubernetes, change nothing in the image. THP is a node-level setting. Pods inherit it and cannot set it. If you cannot change the host configuration directly, a privileged DaemonSet that runs the commands on each node before application pods start is the reliable route.
6. Verify with a probe. The GC log header shows whether the heap got huge pages, and /proc/meminfo shows how much. On a production x86_64 host, the ShmemHugePages total should match the combined -Xmx of the Azul Prime heaps running on it. The sample below, from Azul’s documentation, is from a small test JVM:
7. Measure where the benefit lives. Not in host-level dashboards, and not in microbenchmarks, whose hot heap is too small to miss the TLB. Look at CPU and p95/p99 latency on the services that opted in, using the same hardware and heap size as before the change.
What the customer’s experience shows
Every “disable THP” line in a runbook was written for specific conditions: a kernel that shipped with always, compaction that ran in the page-fault path, and runtimes that could not ask for huge pages. All three conditions have changed.
The customer separated what had burned them (always, synchronous defrag, one configuration for every architecture) from what they wanted, which was fewer TLB misses on the JVMs that could use them. They rolled out the setting that matches the current kernel and the current runtime. The result was a leaner footprint on services that never opted in and a 2 MB-backed heap on Azul Prime services that did.
FAQ
Should I enable Transparent Huge Pages for Java applications?
THP is not a prerequisite; JVMs run fine with it on or off. It is worth enabling for JVMs with large heaps, as long as it is opt-in: set enabled to madvise and defrag to defer, so only runtimes that ask for 2 MB pages get them and allocation does not wait for compaction in the page-fault path. Azul recommends THP for Azul Prime heaps of 100 GB or more, or whenever throughput or start time is lower than expected without it (Azul docs). Azul Prime opts its Java heap in automatically when the host allows it.
What is the difference between THP always, madvise, and never?
always backs every anonymous mapping with 2 MB pages, whether the process asked or not, which inflates memory for processes that touch only part of a region.
never turns THP off, so every process pays the 4 KB-page TLB cost.
madvise gives huge pages only to mappings that a process marks with madvise(MADV_HUGEPAGE).
Does enabling THP affect every process on the host?
The setting applies host-wide, but with madvise allocation stays opt-in per mapping. Processes that never call madvise(MADV_HUGEPAGE) keep using 4 KB pages. Azul Prime opts in for its Java heap with no JVM flags, and -XX:-UseTransparentHugePages turns that off for a single process.
Why doesn’t my Java heap use huge pages even though THP is enabled?
The enabled setting covers only anonymous memory. On x86_64, the Azul Prime heap is mapped through shared memory, so shmem_enabled must also be set to advise. It defaults to never on most distributions. On ARM64, Azul Prime uses tagged heap addresses with ordinary anonymous THP, so shmem_enabled is not needed.
