NVMe Memory Tiering in VMware Cloud Foundation 9.1 – What’s New and How to Enable It

Posted by on in | 3 Views

Memory has always been the first resource to run out in most vSphere clusters. CPUs sit at 30–40% while hosts are already full on RAM, and adding DIMMs is one of the most expensive line items on any server order. With DRAM prices going up and lead times getting longer, this is a real problem for anyone running a private cloud.

VMware introduced NVMe Memory Tiering in VCF 9.0 to address exactly this. It worked well, but the setup was clunky – multiple ESXCLI commands, a host reboot and a hardware RAID controller if you wanted redundancy. VMware Cloud Foundation 9.1 (GA May 2026) fixes most of those pain points. In this post, I will walk through what Memory Tiering is, what changed in 9.1, and how to enable it on your ESX hosts.

What is NVMe Memory Tiering?

Memory Tiering lets ESX use a fast NVMe device as a second tier of memory alongside DRAM:

  • Tier 0 – DRAM, which holds the hot (actively used) memory pages.
  • Tier 1 – NVMe, which holds cold pages that VMs have allocated but are not actively touching.

The hypervisor continuously classifies VM memory pages as hot or cold and moves cold pages to NVMe. This happens beneath the guest OS, so applications don’t need any changes. A few things worth knowing:

  • Only VM memory pages are tiered. ESX kernel pages always stay in DRAM.
  • Tiering only kicks in under memory pressure – roughly at 80% memory consumption on the host. This threshold is fixed.
  • Tiering works at 4KB page granularity, so large pages are disabled on VMs when tiering is enabled.
  • The best candidates are workloads that reserve a lot more memory than they actively use – which, in my experience, is most enterprise VMs.

What’s New in VCF 9.1

1. Better performance

Broadcom reports up to 16% better performance in HammerDB database tests and around 12% lower CPU usage in VMmark tests compared to VCF 9.0. The gains come from smarter cold page detection and more efficient page movement between tiers.

2. Software NVMe mirroring – no RAID controller needed

In VCF 9.0, if you wanted redundancy for the NVMe tier, you needed a hardware RAID controller (Tri-Mode controller or Intel VROC). In 9.1, mirroring is built into vSphere itself. Reads are distributed round-robin across both devices in the mirror. This matters because without a mirror, an NVMe failure behaves like a DIMM failure – the VM is restarted by HA when it touches a page on the failed device. With a mirror, the VM keeps running.

3. No reboot and simpler configuration

Configuration now lives in a single pane in vSphere Configuration Profiles, and the NVMe partition is created automatically. Host reboot is no longer required – only maintenance mode. On the CLI side, three commands become one (more on that below).

4. Proper observability

  • vCenter – host and cluster summary cards showing per-tier breakdown, consumed vs. active memory, and NVMe device health.
  • Performance charts – bandwidth and latency per tier, down to per-VM tier bandwidth.
  • VCF Operations – a dedicated Memory Tiering dashboard plus a What-If analysis to model savings before you enable anything.

5. More VM types can power on

In 9.0, some VM profiles simply couldn’t power on when tiering was enabled on the host. In 9.1, security VMs, low-latency VMs and Fault Tolerance VMs can all power on (some of them just don’t participate in tiering). Nested virtualization is also fully supported now – good news for homelabbers.

Understanding the Tiering Ratio

The ratio defines how much NVMe capacity is used for Tier 1, expressed as a percentage of host DRAM. A few examples:

Host DRAMRatioNVMe used for Tier 1Total memory presented
512 GB25% (4:1)128 GB640 GB
512 GB100% (1:1)512 GB1 TB
512 GB400%2 TB2.5 TB

The ratio can range from 1 to 400%, and the maximum Tier 1 partition size is 4 TB in 9.1. Any unused NVMe capacity is cycled for endurance. Don’t jump straight to the maximum – start with 1:1 and let the Memory Tiering dashboard tell you whether you can go further.

How to Enable Memory Tiering in VCF 9.1

Prerequisites

  • ESX hosts running VCF / vSphere 9.1.
  • A supported NVMe device – check the vSAN SSD section of the Broadcom Compatibility Guide. Use a high-endurance device.
  • Two NVMe devices per host if you want software mirroring.
  • Host in maintenance mode (let DRS evacuate the VMs first).

Option 1 – vSphere Configuration Profiles (UI)

  1. In the vSphere Client, go to the cluster → Configure → Desired State → Configuration and create a draft.
  2. Find the Memory Tiering section and set enable to TRUE.
  3. Click View host-specific details and select the NVMe device to use as Tier 1 for each host.
  4. Set the tiering ratio, and optionally enable mirroring and encryption (for data-at-rest compliance).
  5. Run the pre-check and apply the draft. Hosts will need maintenance mode, but no reboot.

Option 2 – ESXCLI

This is the part that trips people up. In VCF 9.0 we used three separate commands:

# VCF 9.0 and earlier – DO NOT use on 9.1
esxcli system settings kernel set -s MemoryTiering -v TRUE
esxcli system settings advanced set -o /Mem/TierNvmePct -i 100
esxcli system tierdevice create -d /vmfs/devices/disks/<NVMe_device>

On 9.1 these old commands may look like they succeeded, but tiering never actually activates. Use the new memtier namespace instead – one command selects the device and sets the ratio:

# VCF 9.1 and later
esxcli memtier enable -d /vmfs/devices/disks/<NVMe_device> -r 100

# Optional – enable encryption for the Tier 1 device
esxcli memtier config set --encryption true

Replace <NVMe_device> with your device identifier (you can find it with esxcli storage core device list) and adjust -r to the ratio you want.

Verify

Exit maintenance mode and check the host in vCenter. Total memory capacity should reflect DRAM plus the configured ratio, and the Memory Tiering widget should show a non-zero Tier 1 capacity. If Tier 1 shows 0, tiering wasn’t enabled correctly – most likely the old 9.0 commands were used.

Design Considerations from the Field

  • Assess first. Use the Memory Tiering Assessment Tool (MTAT) or the VCF Operations What-If analysis before you buy hardware. Look at active vs. consumed memory – that gap is your opportunity.
  • Mirror in production. Now that it no longer needs a RAID controller, there’s little reason not to. Without it, an NVMe failure means HA restarts.
  • vMotion is slower for tiered VMs. Pages on NVMe must be read back into DRAM before transfer. On the destination host, all pages start in DRAM and get tiered again over time. Factor this into maintenance windows for large clusters.
  • Keep latency-sensitive workloads in mind. They can now power on, but they won’t benefit from tiering. Keep enough DRAM headroom for them.
  • Don’t forget upgrades. If you have automation or host profiles from 9.0 that use the old ESXCLI commands, update them before moving to 9.1.

Wrapping Up

Memory Tiering in VCF 9.0 proved the concept. VCF 9.1 makes it practical: no reboot, no RAID controller, a single command and real visibility into what’s happening on each tier. Broadcom claims up to 40% lower server TCO – as always, “up to” depends on your workloads, so run the What-If analysis against your own data. For memory-bound clusters where CPU is sitting idle, this is one of the most useful features in the 9.1 release, and well worth testing in your lab.

I hope this is informative for you. Thanks for reading! Be social and share it on social media if you feel worth sharing it.

References