Supercharge Memory Tiering in VCF 9.1 / vSphere 9.1

Updates in Memory Tiering in VCF 9.1 vSphere 9.1

Memory Tiering has been around the corner for a while now in VCF 9.0. The demand for DRAM is still high. Memory Tiering helps to reduce the DRAM demand by storing cold memory pages on NVME based SSDs to extent ESXi-hosts DRAM. This post covers the configuration of this feature in 9.1 including advanced parameters and guidance for usage in Omnissa Horizon VDI environments.

If you are on VCF 9.0 you can check out my previous blog post:

Agenda:

1. VCF 9.1 Memory Tiering Improvements
2. Enable Memory Tiering NVME SSD Device
3. Size Memory Tiering parameters
4. Configure Memory Tiering parameters
5. Advanced Memory Tiering parameters
6. Verify Memory Tiering
7. Memory Tiering for VDI (Omnissa Horizon)

VCF 9.1 Memory Tiering Improvements

Performance Improvements

VMware reports up to a 16% performance improvement while also reducing the CPU overhead of Memory Tiering.

NVMe Software RAID

Possibility to configure NVME SSDs for RAID1 mirroring

Configuration Improvements

Easier configuration by using vSphere configuration profiles and rebootless configuration.

Monitoring Improvements

Improved views in vCenter and VCF Operations

Enable Memory Tiering NVME SSD Device

Use an SSD which typically is represented as Mixed Used to allow up to 3 Drive writes per Day. Random I/O performance is crucial. So use SSDs with modern SSD controllers of the PCI-E Gen4 or 5 generation.

For homelab builds use a TLC based SSD with an integrated DRAM cache. Modern PCI-E Gen5 SSDs like SanDisk WD SN8100 or Samsung 9100 pro will result in the best performance.

CLI - GUI option below

In VCF 9.1 / vSphere 9.1 all commands use the esxcli memtier command:

At first put the host in maintenande Mode and connect via SSH or ESH shell.

esxcli system maintenanceMode set -e true

List NVME Devices
esxcli memtier device list

This will select all NVME SSDs without any partitions. This will also include vSAN capable devices.

Select Device which should be used:
esxcli memtier enable -d <Device eui. Number from step above>

You can also add a second NVME SSD by separating the second device using a comma seperation

esxcli memtier enable -d <device_1,device_2>

A secondary device (RAID 1 mirror) can be added later.

GUI Option in vCenter

vCenter GUI -> Cluster -> Configure -> Desired State -> Configuration -> Settings -> Draft

Size Memory Tiering parameters

The following configurations are host specific. They can also be configured at scale using vSphere Configuration profiles.

Configure the percentage of Tiered Memory on the SSD device. You can use the following examples as reference. The default value from VMware is 100% which doubles the memory. You can also use values like 50% in case you run RAM intensive workloads like SQL servers.

Based on Broadcom techdocs there is a sizing guideline for Memory Tiering based on VMs active DRAM memory. Active Memory can be displayed as column in vCenter or is included in a RVTools export. VMs below 50% Memory Tiering are perfect candidates for the default configuration of 100% Memory Tiering percentage where half of it’s memory is in DRAM and the other half can be stored on the SSD. VMs with active memory above 50% like in memory databases require a lower percentage or maybe even candidates to be excluded from Memory Tiering. The following image from Broadcom illustrates the situation:

Source: techdocs.broadcom.com

Configure Memory Tiering parameters

Configure the desired percentage on the host:

esxcli memtier config set --tier-size-pct 50

Optionally configure encryption of the NVME Tier, which slightly decreases performance:

esxcli memtier config set --encryption true

These options are also available in the vCenter Configuration profiles GUI

Advanced Memory Tiering parameters

You can configure Memory Tiering overrides per VM to allow Memory being used or not used for specific VMs:

Disable Memory Tiering:

Advanced Attribute sched.mem.enableTiering: FALSE

Encrypt Memory Tiering for specific VM

sched.mem.EncryptTierNVMe: TRUE

Enable Memory Tiering for Nested Virtualization or Windows Virtualization based Security VMs

sched.mem.enableNestedTiering: TRUE

Verify Memory Tiering

Configuration Verification can be either performed in the UI or via CLI

UI

vCenter -> ESXi Host

CLI

esxcli hardware memory get

vsish -e get /memory/tiers/0/info

vsish -e get /memory/tiers/1/info

Memory Tiering Usage of each VMs can be checked with the memstats command via ESX CLI:

memstats -r hw-stats

Memstats -r vmtier-stats

Typacally Memory Tiering gets used at 60-70% host DRAM Tier0 usage. On lower levels of usage all VM memory will be stored in Tier0 to optimize for best performance and reduce SSD wear level.

Memory Tiering for VDI (Omnissa Horizon)

VDI is an ideal UseCase for Memory Tiering. There is often a low percentage of really active memory as Windows, Linux and the apps consume a certain amount of RAM but often do not use it all the time.

A Tier_Size_Pct (%) of 100% is BestPractice as its double the memory. For low performance VDI 200% could also be an option.

Please note that Memory Tiering cannot be used with physical GPU activated VMs like NVIDIA vGPU.