Red Hat OpenShift Virtualization in nested VMware vSphere Cluster

In this post, I'll go thru the process of running Virtual Machines on OpenShift Virtualization in a nested setup inside VMware vSphere. This requires both ESXi hosts and a VCenter, both on 6.7U3 or up.

Nested virtualization is a configuration where the Virtual Machine running the OpenShift Node on ESXi exposes the bare metal capabilities so new Virtual Machines can be created inside it. This could be used as a showcase for the technology, lab environment or POCs. OpenShift VMs are based on KVM, the virtualization layer from the Linux Kernel and are implemented based on the KubeVirt project.

Disclaimer: Nested virtualization is not officially supported on any platform. Officially OpenShift Virtualization is meant to run on Bare Metal nodes. Please do not use it in Production.

Deploy and configure OpenShift 4 on vSphere in IPI

First create an OpenShift cluster using VMware IPI (Installer Provisioned Infrastructure) to leverage the easy and quick deploy. Going thru this is a little outside the scope of the post but the documentation details the process. At the end, the installer will provision and configure all virtual machines on your VCenter and the cluster will be answering in both API and Console.

Enable Nested Virtualization

Now, enable nested virtualization in the OpenShift template VM. Go to your vCenter or ESXi console and edit the template VM, usually named clustername-clusterid-rhcos, in my case it’s “ocp-9s46k-rhcos”.

You need to expand the CPU section and check “Expose hardware assisted virtualization to the guest OS”: