Category Archives: ESXi

Updating ESXi Vendor Images with Latest Patches

This post will walk through updating a vendor specific ESXi image with updated VIBs. In this instance we are applying patch ESXi650-201803001 which bundles the esx-base, esx-tboot, vsan, and vsan health VIBs (ESXi650-201803401-BG) with the updated CPU microcode (ESXi650-201803402-BG), to provide part of the hypervisor-assisted guest mitigation for operating systems of the Branch Target Injection issue (CVE-2017-5715) commonly known as Spectre. The appropriate patches for ESXi 6.0 and 5.5 can be found in VMware Security Announcement VMSA-2018-0004.3 here.

For more information on Meltdown and Spectre see this blog post, VMwares responses can be found here, on the VMware Security & Compliance Blog here, as well as VMware Security Announcement VMSA-2018-0004 here. Ensure your vCenter Server is also patched accordingly by following the guidance in this post.

There are a number of ways to push out ESXi patches to hosts, such as CLI, Update Manager, Auto Deploy. The latest images can be downloaded from the patch repository here. As we are using vendor specific images, which are typically slow to be updated from the main VMware image, there is no vendor image available that mitigates against Spectre at the time of writing. Therefore the steps below will cover replacing VIBs in the HPE ESXi 6.5 image with the updated VIBs released by VMware. The same process can be used for other vendor images and ESXi versions by downloading the appropriate images, however the custom image we create may not be supported, and therefore may not be appropriate for production environments.

meltdown-spectre-vmware

The steps below assume Auto Deploy and Image Builder are already setup. You don’t need to use Auto Deploy to be able to use the Image Builder, but the services do need to be started, if they’re not then see the Auto Deploy Guide. Download the latest vendor image, in my case I am using HPE, and the latest ESXi build from the patch repository here.

Log into the vSphere web client and click the Auto Deploy icon from the home page.

Auto_Deploy

Click the Software Depots tab. Software depots contain images or software packages. If you don’t already have a custom software depot click the Add Software Depot icon to add a new custom depot where images will be stored. Use the Import Sofware Depot to upload a zip file, in this case we need to add the vendor image (in my case VMware-ESXi-6.5.0-Update1-7388607-HPE-650.U1.10.2.0.23-Feb2018-depot.zip) and the updated VMware image (ESXi650-201803001.zip).

Select the software depot containing the vendor image, in my case VMware-ESXi-6.5.0-Update1-7388607-HPE-650.U1.10.2.0.23-Feb2018-depot. Under Image Profiles select the vendor image and click Clone.

Auto_Deploy_2

We are cloning the vendor image to replace the updated VIBs. Enter a name and vendor for the image, select the software depot.

Image_Builder_2

On the next page the software packages are listed, those already included in the build are ticked. Ensure the Software depot is set to All depots in the drop-down.

Review the updated VIBs in the appropriate ESXi patch release.

ESXi650-201803401-BG:

  • VMware_bootbank_esx-base_6.5.0-1.41.7967591
  • VMware_bootbank_esx-tboot_6.5.0-1.41.7967591
  • VMware_bootbank_vsan_6.5.0-1.41.7547709
  • VMware_bootbank_vsanhealth_6.5.0-1.41.7547710

ESXi650-201803402-BG:

  • VMware_bootbank_cpu-microcode_6.5.0-1.41.7967591

Use the search function to find each of the updated VIBs. Un-select the existing version and select the new version to add it to the build.

Image_Builder_3

For the Spectre patches remember to include the CPU microcode.

Image_Builder_4

Once complete click Next and Finish. Select the custom software depot where the image has been created. The image is now ready to use with an Auto Deploy rule, or can be exported in ISO or ZIP format by right clicking and selecting Export Image Profile.

Image_Builder_5

For the Spectre updates after the new image has been installed/applied to an ESXi host we can perform some verification of the hypervisor-assisted guest mitigation. This blog post from virtuallyGhetto provides PowerCLI functions and instructions for validating the correct microcode and patches are present. In the example below I have updated host 1 but not host 2:

Verify_1

The virtual machines can also be validated to confirm they are seeing the new CPU features, a power cycle is required for each VM. Before power cycling:

Verify_2

After power cycling:

Verify_3

ESXi 6.5 FCoE Adapters Missing

After installing or upgrading to ESXi 6.5 FCoE adapters and datastores are missing. In this case the hardware in use is a HP ProLiant BL460c Gen9 server with HP FlexFabric 10Gb 2-port 536FLB adapters, although this seems to have been a problem for other vendors (see here) and versions too.

This issue should be resolved with a driver provided by the vendor which has the FCoE auto discovery on boot parameter enabled. Cross reference your hardware against the VMware Hardware Compatibility Guide here, and confirm you are using the correct version of the bnx2fc driver and firmware. If no updated driver is available from the vendor then review the workarounds outlined below.

Stateful Installs

Credit to this article, SSH onto the host and run the following commands.

esxcli fcoe adapter list lists the discovered FCoE adapters, at this stage there will be no results.

esxcli fcoe nic list lists the adapters available as potential FCoE candidates. Locate the name of the adapter.

esxcli fcoe nic enable -n vmnicX enables the adapter, replace vmnicX with the adapter name, for example vmnic2.

esxcli fcoe nic discover -n vmnicX enables discovery on the adapter, replace vmnicX with the adapter name.

esxcli fcoe adapter list lists the discovered FCoE adapters, you should now see the FCoE adapters listed.

The storage adapters should now be showing in the vSphere web client, however if you are using stateless installs with Auto Deploy, then this workaround is not persistent and is lost at reboot.

storageadapters2

Stateless Installs

Credit to this article, we were able to create a custom script bundle to enable discovery on the FCoE adapters as part of the deploy rule with the steps below. Custom script bundles open up a lot of possibilities with Auto Deploy, but at this stage they are CLI only. I also noticed that if you create a deploy rule with a script bundle from the CLI, although it shows in the GUI if you then edit that rule in the GUI (for something unrelated, e.g. updated host profile) then it removes the script bundle without warning. So this is something you would need to weigh up against your environment, if you are already using CLI to configure deploy rules it shouldn’t be a problem.

PowerCLI can now be installed directly through PowerShell, if you don’t already have PowerCLI installed see here.

  • First up we’ll need to create the script on a Linux / Unix system. I just used a test ESXi host we had kicking about over SSH. Type vi scriptname.sh replacing with an appropriate name for your script.
  • The file will open, type i to begin editing.
  • On the first line enter #!/bin/ash followed by the relevant enable and discover commands from the section above. You can see in the example below the commands for enabling vmnic2 and vmnic3 as FCoE adapters.

ssh1

  • Press escape to leave the text editor and type :wq to save changes to the file and close.
  • Next we need to create the script bundle that will be imported into Auto Deploy. Type tar -cvzf bundlename.tgz scriptname.sh

ssh2

  • Copy the script bundle with the .tgz extension to your local machine, or the computer from where you will be using PowerCLI to create the deploy rule. In my case I copied the file over with WinSCP.
  • You should also have an ESXi image in zip format, make a note of the location. Add the script bundle and the ESXi software depot by running the following commands Add-ScriptBundle location\bundlename.tgz and Add-EsxSoftwareDepot location\file.zip. If you need further assistance with building custom images or using PowerCLI to manage Auto Deploy see the VMware Auto Deploy 6.x Guide and How to Create Custom ESXi Images posts.

ps1

  • Build the deploy rule using your own variables, again if you’re already using Auto Deploy I’m assuming you know this bit, we’re just adding an additional item in for the script bundle. See the guide referenced above if you need assistance creating deploy rules. I have used:
    • New-DeployRule -Name "Test Rule" -Item "autodeploy-script","HPE-ESXi-6.5.0-Build-5146846", LAB_Cluster, -Pattern "ipv4=192.168.0.101" | Add-DeployRule

ps2

  • The deploy rule is created and activated, I can now see it in the Auto Deploy GUI in the vSphere web client, with the associated script bundle. When the host boots from the deploy rule the script is extracted and executed, and the FCoE adapters are automatically enabled and discovered on boot.

autodeployGUI

  • If you don’t use the | Add-DeployRule parameter then the deploy rule will be created but show inactive. You can activate using the GUI but do not edit the rule using the GUI or the script bundle will break.
  • If you are updating an existing image then don’t forget to remove cached rules by remediating host associations, under the Deployed Hosts tab.

ESXi Command Line Upgrades

Upgrading and patching of ESXi hosts can be done using the esxcli software commands, with either the online depot, or an offline bundle. For managing multiple hosts Update Manager is generally the best way to go. Update Manager is now built into VCSA 6.5 (vCenter Server Appliance 6.5 Install Guide) or can be installed on a Windows server (VMware Update Manager 6.0 Install Guide / VMware Update Manager 6.5 Install Guide).

In both the methods outlined below we will be connecting to the ESXi host via SSH. For assistance with enabling SSH review this KB article, remember to disable SSH when you’re done. Before beginning you should ensure any powered on virtual machines are shut down or migrated off the host. The host should be placed into maintenance mode and requires a reboot after patches are applied. You may find the following commands of use:

Lists the installed ESXi build version: vmware -v

Lists installed vibs: esxcli software vib list

List VMs present on the host: vim-cmd vmsvc/getallvms

Gracefully shut down a VM, replacing number with the VMID obtained from the above command: vim-cmd vmsvc/power.shutdown number

Power off a VM, replacing number with the VMID obtained from the above command: vim-cmd vmsvc/power.off number

Power on a VM, replacing number with the VMID obtained from the above command: vim-cmd vmsvc/power.on number

Enter maintenance mode: vim-cmd /hostsvc/maintenance_mode_enter

Exit maintenance mode: vim-cmd /hostsvc/maintenance_mode_exit

When installing individual vibs replace -d with -v, for example: esxcli software vib install -v viburl

The esxcli software commands below all use the update tag, this ensures that only newer contents of a patch are applied. If a system contains newer revisions of the selected patches then these will not be applied. The install tag can potentially overwrite existing drivers, and therefore the update method is recommended for upgrading ESXi and installing patches to prevent an unbootable state.

Online Depot

Useful for patching or upgrading individual hosts which have an internet connection and sufficient boot drive capacity. Open an SSH connection to the ESXi host using a client such as Putty, and log in with the root account. First enter the following command to open the firewall for outgoing http requests:

esxcli network firewall ruleset -e true -r httpClient

Find the image profile to upgrade to by reviewing the ESXi patch tracker here. To upgrade the ESXi host run the following command, replacing Imageprofile with the desired image profile name.

esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p Imageprofile

For example:

esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-6.0.0-20161104001-standard

When the upgrade is complete use reboot to restart the host. Finally close the outgoing http port:

esxcli network firewall ruleset -e false -r httpClient

Offline Bundle

First download the relevant offline bundle from VMware, for upgrades ESXi ISO images can be found here, patches in zip format can be found here.

Next we need to upload the downloaded file to a datastore the ESXi host or hosts have access to. Log into the vSphere web client or the ESXi host UI. Navigate to the Storage view, right click the datastore and select Browse Files. Click the upload file icon and select the zip file downloaded earlier. With the patches now accessible from the host we can start the update process.

Open an SSH connection to the ESXi host using a client such as Putty. Install the downloaded updates using the following command, replacing datastore with the name or UUID of the datastore, and zip with the file name of the downloaded patches:

esxcli software vib update -d /vmfs/volumes/datastore/zip

For example:

esxcli software vib update -d /vmfs/volumes/Datastore01/ESXi600-201611001.zip

Check the installation result, a reboot is required. The content listed below this is a breakdown of the VIBs installed, removed, and skipped. Restart the host using the reboot command.

result

Following on from upgrading or patching an ESXi host you should also ensure VMware Tools is updated on any guest virtual machines.

For more information on ESXi command line tools see the Troubleshooting with ESXi Shell and vSphere Management Assistant Guide posts.