Understanding Host and Guest Memory Usage and Other Memory Management Concepts

Understanding “Host” and “Guest” Memory Usage and Related Memory Management Concepts Kit Colbert Sr. Staff Engineer, VMware

Disclaimer This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. “These features are representative of feature areas under development. Feature commitments commitments are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery.”

Agenda Motivation Define host and guest memory usage and ask some questions Memory management concepts Answer our questions Best practices Summary Q&A

Motivation

Host and Guest Memory Usage

Using Host and Guest Memory Usage Useful for quickly analyzing a VM’s status  –  Coarse-grained information  –  Important for prompting further investigation

Requires an understanding of memory management concepts  –  Many aspects of host/guest memory interaction are not obvious

Define host and guest memory and ask some questions

Host Memory Usage Defined Host Memory Usage  –  The amount of physical host memory in megabytes allocated to a guest • Includes virtualization overhead (e.g. hypervisor data)

Guest Memory Usage Defined Guest Memory Usage  –  The amount of memory in megabytes actively used by a guest operating system and it’s applications  –  The percent of total guest memory actively used by a guest

Memory Terminology memory size total amount of memory presented to a guest allocated memory memory assigned to applications

active memory allocated memory recently accessed or used by applications

unallocated memory memory not assigned

inactive memory allocated memory not recently accessed or used

Host memory usage measures this, sorta…

Guest memory usage measures this

Some commonly asked questions Why is host memory usage so high? Why is host memory usage greater than guest memory usage? Why is host/guest memory usage different than what I see inside the guest?

Memory Management Concepts

Discussion of Concepts Focus on high-level concepts, not implementation details  –  Simpler and easier to understand  –  Keep the fundamental ideas and drop the unimportant ones

Other presentations may present this information differently  –  May discuss more implementation than concept  –  That’s ok – different viewpoints for same thing!

Virtual Memory Creates uniform memory address space  –  Operating system maps application virtual addresses to physical addresses  –  Gives the operating system memory management abilities that are transparent to the application

“virtual” memory guest 

“physical” memory hypervisor 

Hypervisor adds extra level of indirection  –  Maps guest’s physical addresses to machine addresses  –  Gives the hypervisor memory management abilities that are transparent to the guest

“machine” memory

Virtual Memory VM

“virtual” memory

Application

“virtual” App guest  memory

guest 

“physical” memory

Operating System

“physical” OS memory hypervisor 

hypervisor 

“machine” memory

Hypervisor

“machine” Hypervisor memory

Application Memory Management Starts with no memory Allocates memory through syscall to operating system

App

Often frees memory voluntarily through syscall Explicit memory allocation interface with operating system

OS

Hyper visor

Operating System Memory Management Assumes it owns all physical memory No memory allocation interface with hardware

App

 –  Does not explicitly allocate or free physical memory

Defines semantics of “allocated” and “free” memory  –  Maintains “free” list and “allocated” lists of physical memory  –  Memory is “free” or “allocated” depending on which list it resides

OS

Hyper visor

How “Allocated” and “Free” Lists Work Allocated List

Free List App

OS

Memory becomes “allocated” or “free” based on which list (piece of memory) has a pointer to it

Hypervisor Memory Management Very similar to operating system memory management

App

 –  Assumes it owns all machine memory  –  No memory allocation interface with hardware

OS

 –  Maintains lists of “free” and “allocated” memory Hyper visor

VM Memory Allocation VM starts with no physical memory allocated to it

App

Physical memory allocated on demand  –  Guest OS will not explicitly allocate

OS

 –  Allocate on first VM access to memory (read or write) Hyper visor

VM Memory Reclamation Guest physical memory not “freed” in typical sense

App

 –  Guest OS moves memory to its “free” list  –  Data in “freed” memory may not have been modified

Guest Free List

OS

Hypervisor isn’t aware when guest frees memory  –  Freed memory state unchanged  –  No access to guest’s “free” list  –  Unsure when to reclaim “freed” guest memory

Hyper visor

VM Memory Reclamation Cont’d Guest OS (inside the VM)  –  Allocates and frees…  –  And allocates and frees…

Inside the VM Guest free list

App VM

 –  And allocates and frees…

VM

OS

 –  Allocates…  –  And allocates…  –  And allocates…

Hypervisor can’t reclaim memory through guest frees!

Hyper visor

What to do about VM memory reclamation?

Why not just watch the free list? Idea  –  Hypervisor could identify memory used by guest for it’s free list  –  Watch that memory area for changes and free physical memory that backs guest memory added to free list

Problem  –  Free list may be hard to identify • Depends on guest type and may change from release to release

 –  “Free” is not always free • “Free memory” may be used in buffer cache or elsewhere by guest

 –  Guest may not free memory when hypervisor wants it to

What can the hypervisor do to reclaim VM memory? Nothing  –  Route taken by pretty much everyone except VMware  –  No memory overcommitment! (Inefficient use of memory)  –  May try to convince you memory overcommitment isn’t important

Something  –  VMware-innovated techniques  –  Supports memory overcommitment • Efficient use of memory • Strict resource control guarantees

Memory Overcommitment VM 1

VM 2

VM 3

Hyper visor

Aggregate guest memory greater than host memory  –  Not enough physical memory to satisfy all VM memory needs  –  Eventually VM may want physical memory when none is left

Why is memory overcommitment important? Ensures physical memory is actively used as much as possible  –  Guest VMs may have lots of unused or inactive memory  –  No reason to back one VM’s unused/inactive memory with physical memory if other VMs can use it  –  Hypervisor can reclaim unused/inactive memory and redistribute to other VMs who will actively use it

Increases VM-to-host consolidation ratio  –  Each VM has a smaller physical memory footprint since only its active memory is resident  –  You can fit more VMs in the same amount of physical memory

VM Memory Reclamation Techniques

Transparent Page Sharing (TPS) Simple idea: why maintain many copies of the same thing?

VM 1

 –  If 4 Windows VMs are running, there are 4 copies of Windows code

VM 2

VM 3

Hyper visor

 –  Only one copy is needed

Share memory between VMs when possible

VM 1

VM 2

 –  Background hypervisor thread identifies identical sets of memory  –  Points all VMs at one set of memory, frees the others  –  VMs are unaware of the change

Hyper visor

VM 3

Ballooning Hypervisor wants to reclaim memory Guest OS is not aware of this

VM 1

VM 2 App

App

OS

OS

 –  Thinks it owns all physical memory  –  Sits inside its own “box”, unaware it’s running in a VM or that other VMs are running

Goal: make the guest aware so it frees up some of its memory Solution: artificially create memory pressure inside the VM  –  “Push” memory pressure from the hypervisor into the VM  –  Use “balloon” driver inside the VM to create memory pressure

memory pressure Hyper visor

Inflating Balloon

3 pieces of Guest: 6 memory are allocated

1. Balloon driver allocates memory

Guest App

2. Balloon driver pins allocated memory 3. Guest may reclaim other memory 4. Balloon driver tells hypervisor what memory it allocated 5. Hypervisor frees machine memory backing memory allocated by balloon driver 6. Hypervisor now has more free physical memory

Balloon Driver

Guest free list

OS List of memory

Hyper visor

Hypervisor: 5 8 pieces of memory are allocated

Inflating Balloon Cont’d Why can the hypervisor reclaim ballooned memory? What in the VM assumes the memory has a specific value? Balloon Driver

1. Balloon driver doesn’t Because of a contract with the hypervisor

2. OS doesn’t Because the memory allocated to an app

3. Nothing in the VM relies on the memory’s value for correctness

OS List of memory

Hyper visor

Hypervisor can safely reclaim ballooned memory because the VM does not rely on a particular value for that memory.

Inflating Balloon Cont’d Guest OS swapping, a possible side effect of ballooning Two possibilities for guest free memory: Guest free list

Guest free list

2. Needs to swap!

1. No swapping necessary!

1. VM has lots of free memory

2. VM doesn’t have much free memory

Guest App

Balloon Driver

OS

Hyper visor

Guest OS chooses whether to swap or not!

Swapping Hypervisor can swap VM memory

App

 –  Swaps out to a per-VM swap file  –  Transparent to the VM

Hypervisor swapping is a last resort

OS

 –  Both TPS and ballooning preferred • Low overhead for TPS • Guest swapping due to ballooning performs better than hypervisor swapping • But both take time

 –  Hypervisor swapping quickly reclaims memory, but is more expensive overall

Hyper visor

When to reclaim memory

Quick Review Physical memory is not reclaimed on a guest that is “free”  –  VM may accrue lots of physical memory

Transparent page sharing is always running  –  May or may not help reduce a VM’s physical memory consumption

Ballooning or swapping is the only other way to reclaim memory  –  Both have performance overhead  –  Want to use only when necessary

Question: when to invoke ballooning or swapping?

When To Reclaim Memory Memory overcommitment  –  Aggregate guest memory is greater than host memory

Not memory overcommitted  –  Never reclaim memory from guest through ballooning or swapping • No need to! • (Transparent page sharing is always running)

Memory overcommitted  –  Only reclaim once free physical memory drops below threshold • Start ballooning when memory starts to fall toward threshold • Start swapping as memory nears/passes threshold * Assumes no VM or resource pool memory limits are set!

Answer our questions

High Host Memory Usage Why is host memory usage so high?  –  VM may have used lots of memory in the past • On boot, Windows zeros all guest memory • Starting several guest applications together, e.g., at boot time or when building a big project, can lead to high memory usage • Guest workload spike

 –  Host memory usage represents “high water mark” of guest’s memory usage on non-memory overcommitted hosts • No reason to reclaim memory from VM when not memory overcommitted

Host Memory Greater Than Guest Memory Why can host memory be greater than guest memory?  –  At some point in the past, the guest consumed a larger amount of memory than it is actively using now  –  With little to no memory pressure on the physical host, there is no reason for hypervisor to reclaim memory from the guest  –  The guest maintains a high host memory usage even though its guest memory usage is small

Expected behavior  –  Nothing to worry about!

Host/Guest Memory Usage Different Inside Guest Why is host/guest memory usage different than what I see inside the guest OS?  –  Guest memory • Guest has better visibility while estimating “active” memory • ESX active memory estimate technique can take time to converge

 –  Host memory • Host memory usage doesn’t correspond to any memory metric within the guest • Host memory usage size is based on a VM’s relative priority on the physical host and memory usage by the guest

Again, this is expected!

Best Practices

Host Memory Usage Large host memory usage OK  –  Expected behavior much of the time!

Host memory usage shouldn’t limit guest memory usage  –  Host memory usage should be large enough to accommodate guest workload  –  Don’t over commit host memory forcing the guest to continuously swap

Use shares to adjust relative priorities among VMs when memory is overcommitted  –  Host memory usage is based on shares when overcommitted

Guest Memory Usage VM’s memory size should be slightly larger than the average guest memory usage  –  Accommodates workload spikes without guest swapping  –  Hypervisor will handle excess host memory usage  –  Larger VM memory size means more overhead memory

Summary

Summary Host/guest memory usage is not as simple as it looks!  –  Represents complex set of data  –  Requires understanding of memory management concepts  –  Many aspects that are not obvious!

Memory Management Concepts  –  Guest doesn’t “free” memory in a typical sense s ense  –  Hypervisor isn’t aware of memory a guest has “freed”  –  Hypervisor uses transparent page sharing, ballooning, and swapping to reclaim memory  –  Ballooning and swapping used only when host is memory overcommitted!

Summary Cont’d Host/guest memory can be very useful with proper knowledge  –  Quick way to determine the status of VM  –  Important tool to prompt further investigation  –  Requires good mental model of memory management concepts • Hopefully we’ve helped you with that today!

Q&A Kit Colbert Sr. Staff Engineer VMware

Thank you for coming. Rate your session and watch for the highest scores!

Backup Slides

Measuring Active Guest Memory Guest self-measurement won’t work  –  Each guest uses different method to estimate active memory  –  Can’t compare activeness data from Windows guest with Linux guest  –  Comparable estimate important for making allocation decisions decisi ons

Statistical Sampling  –  Select a subset of memory at random  –  Compute the percentage of pages accessed in a minute  –  Average percentage gives an estimate of “active” guest memory

Ballooning and Swapping Ballooning  –  Provide adequate swap space in guest OS • Ballooning can lead to high guest memory swapping • Ballooning not effective if a guest can’t swap

 –  Limit maximum balloon size if lots of guest memory pinned • Pinned memory cannot be swapped • Use sched.mem.memctl.max to set maximum balloon size

Swapping  –  Ensure adequate swap space on VMFS datastore • Swap size = VM memory size – VM memory reservation