3 months ago
- Dom0's memory placement was skewed because Xen's E820 trim allocated from lowest-address nodes first; fixed by a proportional trim that distributes memory across all host NUMA nodes.
- Three pieces of NUMA topology (SRAT, SLIT, x2APIC IDs) are now synthesized for dom0 under specific conditions (PVH, pinned vCPUs, equal vCPU/pCPU count, multiple nodes), making dom0 fully NUMA-aware.
- Bugs discovered during implementation included: node-numbering using internal Xen IDs instead of host proximity domains, incorrect SRAT memory range splitting, and a proportional trim issue on sub-NUMA hardware.
- A new hypercall (XENMEM_get_mfn_pxms) exposes the host proximity domain of foreign frames to dom0, allowing per-node memory pools and enabling backend drivers to know the ring's node.
- Backend drivers (blkback, netback) now affinitize kthreads, IRQs, and event channels to the ring's host node, with fixes for IRQ affinity overwrites by irqbalance and teardown WARN messages.
- Frontend drivers (blkfront, netfront) distribute multi-queue rings across guest nodes using explicit node rotation, and XPS steering is added to netfront to keep transmit paths NUMA-local.
- The entire paravirtual I/O path (guest to ring to backend to physical device) is now NUMA-local except for one deliberate interconnect traversal for shared physical NICs.
- Device locality for NIC/HBA interrupts now works because dom0 sees real topology, enabling proper interrupt allocation and steering maps.
- A latent vNUMA memory placement bug in the toolstack (vnode indices used as physical node IDs) was discovered; it only surfaced under memory-bandwidth benchmarks on non-contiguous placements.
- The code is available in Edera's Xen branch and Linux kernel patches, with the toolstack-specific orchestration (placement heuristics) not upstreamed.