Over the years, VMware ESXi has allowed installation on multiple boot device types across various hardware platforms (blade chassis and/or rackmount servers). The boot device types included internal hard drives, and best practice was to have redundant drives in RAID 1 which increased the deployment cost. Another option was to boot from SAN, using either FC or iSCSI; but this added complexity to some installations and additional costs. The most common and cost-effective solution was to use SD cards, and some server vendors even offered redundant SD card options.
I have used SD cards as a boot device with many customers for years and in my lab environments too.
Booting off USB has allowed me to make better use of the internal drives for other use cases, like vSAN or NetApp ONTAP Select (OTS), as seen in several of my blog posts.
If you’ve upgraded to ESXi 7 and you use USB or SD card boot devices, may have likely experienced some host failures found in VMware KB83963:
~Bootbank cannot be found at path ‘/bootbank’ errors being seen after upgrading to ESXi 7.0 U2 (83963)
This VMware KB article recommends you upgrade to ESXi 7.0U2C to resolve the bootbank errors. Unfortunately, ESXi 7.0U3 has been taking down entire clusters as seen in VMware KB86100.
Before sharing my recommendation on how to get ESXi 7.0U3 running in your environment, for those who are currently using USB or SD cards, let me explain why USB and SD cards are now an issue with the latest version of ESXi.
With VMware ESXi 7, VMware made changes to the layout for storage system partitions on its boot device. This was partly done to be more flexible, and to add support for additional VMware solutions, including third party plugins. You may see more detailed information in this VMware blog post.
Premature SD card failures have occurred, in part, because of the growing reads/writes on the ESX-OSDATA partition and the excessive reads from using VMware tools, as seen in this VMware KB2149257.
Due to the impact of this change with ESXi 7, VMware recommends moving VMware Tools to a RamDisk. While this workaround is easy to implement, it must be applied to every host in your environment using an SD card as a boot device.
Those with VDI environments should address this immediately
- VDI environments frequently issue read operations to the SD card to access VMware Tools. The frequency of these read operations can cause SD cards to burn out prematurely resulting in SD card corruption.
VMware is deprecating the use of SD cards with vSphere ESXi 7.0U3 and highly recommends using one of the specified options in VMware KB85685.
- Plan to use a local persistent disk as the boot device for all partitions.
- If a local persistent device is not already available, add another persistent device. This persistent local boot device can be industrial grade M.2 flash (SLC and MLC), SAS, SATA (including SATADOM), or PCIe NVMe disk (32GB minimum, 128GB recommended).
- If a persistent local device is not available as a boot device, SD cards can be used for boot bank partitions. However a separate persistent local device to store the OSDATA partition (32GB minimum, 128GB recommended) must be provided.
- Provide SAN/NAS Boot as an option for the entire System Storage (Boot banks + OSDATA partitions), when ESXi is installed on a SAN/NAS LUN.
- Existing customers planning to upgrade to future revisions of ESXi and who cannot use any of the above options will not be in a supported configuration.
VMware vSphere 6.5 and 6.7 will be End of Support on October 15th 2022. This is less one year from now and due to the current supply chain issues, finding replacement boot media may take months. If you are booting ESXi from an SD card, or USB drive, I/we strongly recommend looking into solving this issue for your environment as soon as possible.
End of Support notice for ESXi 6.5 and 6.7:
https://kb.vmware.com/s/article/66977#:~:text=The%20End%20of%20General%20support%20for%20vSphere%206.7%20will%20be,vSphere%20Upgrade%20Service%20is%20available
I see booting from SD card or USB drive in most environments and, because of this, most of my customers are impacted with the removal of SD cards and USB drives as boot devices. If you also fall into the category of using an SD card or USB drive as your boot device, you have several options.
A most common solution currently is to boot from SAN. Stay tuned for my next blog post where I show how easy it is reconfigure your hosts from SD > boot from SAN (BFS).