Category Archives: Auto Deploy

ESXi 6.5 FCoE Adapters Missing

After installing or upgrading to ESXi 6.5 FCoE adapters and datastores are missing. In this case the hardware in use is a HP ProLiant BL460c Gen9 server with HP FlexFabric 10Gb 2-port 536FLB adapters, although this seems to have been a problem for other vendors (see here) and versions too.

This issue should be resolved with a driver provided by the vendor which has the FCoE auto discovery on boot parameter enabled. Cross reference your hardware against the VMware Hardware Compatibility Guide here, and confirm you are using the correct version of the bnx2fc driver and firmware. If no updated driver is available from the vendor then review the workarounds outlined below.

Stateful Installs

Credit to this article, SSH onto the host and run the following commands.

esxcli fcoe adapter list lists the discovered FCoE adapters, at this stage there will be no results.

esxcli fcoe nic list lists the adapters available as potential FCoE candidates. Locate the name of the adapter.

esxcli fcoe nic enable -n vmnicX enables the adapter, replace vmnicX with the adapter name, for example vmnic2.

esxcli fcoe nic discover -n vmnicX enables discovery on the adapter, replace vmnicX with the adapter name.

esxcli fcoe adapter list lists the discovered FCoE adapters, you should now see the FCoE adapters listed.

The storage adapters should now be showing in the vSphere web client, however if you are using stateless installs with Auto Deploy, then this workaround is not persistent and is lost at reboot.


Stateless Installs

Credit to this article, we were able to create a custom script bundle to enable discovery on the FCoE adapters as part of the deploy rule with the steps below. Custom script bundles open up a lot of possibilities with Auto Deploy, but at this stage they are CLI only. I also noticed that if you create a deploy rule with a script bundle from the CLI, although it shows in the GUI if you then edit that rule in the GUI (for something unrelated, e.g. updated host profile) then it removes the script bundle without warning. So this is something you would need to weigh up against your environment, if you are already using CLI to configure deploy rules it shouldn’t be a problem.

PowerCLI can now be installed directly through PowerShell, if you don’t already have PowerCLI installed see here.

  • First up we’ll need to create the script on a Linux / Unix system. I just used a test ESXi host we had kicking about over SSH. Type vi replacing with an appropriate name for your script.
  • The file will open, type i to begin editing.
  • On the first line enter #!/bin/ash followed by the relevant enable and discover commands from the section above. You can see in the example below the commands for enabling vmnic2 and vmnic3 as FCoE adapters.


  • Press escape to leave the text editor and type :wq to save changes to the file and close.
  • Next we need to create the script bundle that will be imported into Auto Deploy. Type tar -cvzf bundlename.tgz


  • Copy the script bundle with the .tgz extension to your local machine, or the computer from where you will be using PowerCLI to create the deploy rule. In my case I copied the file over with WinSCP.
  • You should also have an ESXi image in zip format, make a note of the location. Add the script bundle and the ESXi software depot by running the following commands Add-ScriptBundle location\bundlename.tgz and Add-EsxSoftwareDepot location\ If you need further assistance with building custom images or using PowerCLI to manage Auto Deploy see the VMware Auto Deploy 6.x Guide and How to Create Custom ESXi Images posts.


  • Build the deploy rule using your own variables, again if you’re already using Auto Deploy I’m assuming you know this bit, we’re just adding an additional item in for the script bundle. See the guide referenced above if you need assistance creating deploy rules. I have used:
    • New-DeployRule -Name "Test Rule" -Item "autodeploy-script","HPE-ESXi-6.5.0-Build-5146846", LAB_Cluster, -Pattern "ipv4=" | Add-DeployRule


  • The deploy rule is created and activated, I can now see it in the Auto Deploy GUI in the vSphere web client, with the associated script bundle. When the host boots from the deploy rule the script is extracted and executed, and the FCoE adapters are automatically enabled and discovered on boot.


  • If you don’t use the | Add-DeployRule parameter then the deploy rule will be created but show inactive. You can activate using the GUI but do not edit the rule using the GUI or the script bundle will break.
  • If you are updating an existing image then don’t forget to remove cached rules by remediating host associations, under the Deployed Hosts tab.

vCenter Server Appliance Integrated TFTP Server

This post covers the steps required to use the vCenter Server Appliance for Auto Deploy, with the built in TFTP server in vSphere 6.5. For more information on Auto Deploy, and to see the process for creating ESXi images and deploy rules to boot hosts, see the VMware Auto Deploy 6.x Guide. This post assumes that you have a working vCenter Server Appliance, and may be of use if you have recently migrated from Windows vCenter to VCSA.

Enable Auto Deploy

Open the vSphere web client and click System Configuration, Nodes. Select the vCenter Server and open the Related Objects tab. The Auto Deploy, ImageBuilder Service, and VMware vSphere ESXi Dump Collector services should all be set to Automatic and Running.

To start a service right click and select Start, then select Edit Startup Type and choose Automatic.

servicesLog out of the web client and log back in. You should now see the Auto Deploy icon on the home page.


Enable TFTP

Now that Auto Deploy is enabled we can configure the TFTP server. Enable SSH on the VCSA by browsing to the Appliance Management page: https://VCSA:5480 where VCSA is the IP or FQDN of your appliance.

Log in as the root account. From the Access page enable SSH Login and Bash Shell.


SSH onto the vCenter Appliance, using a client such as Putty, and log in with the root account. First type shell and hit enter to launch Bash.

To start the TFTP service enter service atftpd start. Check the service is started using service atftpd status.


To allow TFTP traffic through the firewall on port 69; we must run iptables -A port_filter -p udp -m udp –dport 69 -j ACCEPT. Validate traffic is being accepted over port 69 using iptables -nL | grep 69.


The TFTP server will now work, however we need to make a couple of additional changes to make the configuration persistent after the VCSA is rebooted. There isn’t an official VMware way of doing this, and as it’s done in Linux there may be more than one way of achieving what we want. Basically I am going to backup iptables and create a script to restore iptables and start the TFTP service when the appliance boots. The steps are outlined below and this worked for me, however as a reminder this is not supported by VMware, and if you are a Linux expert you’ll probably find a better way round it.

The following commands are all run in Bash on the vCenter Server Appliance, you can stay in the existing session we were using above.

First make a copy of the existing iptables config by running iptables-save > /etc/iptables.rules.

Next change the directory by running cd /etc/init.d, and create a new script: vi, for example: vi

Press I to begin typing. I used the following, which was copied from the Image Builder Service startup script, and modified for TFTP.

#! /bin/sh
# TFTP Start/Stop the TFTP service and allow port 69
# chkconfig: 345 80 05
# description: atftpd

# Provides: atftpd
# Required-Start: $local_fs $remote_fs $network
# Required-Stop:
# Default-Start: 3 5
# Default-Stop: 0 1 2 6
# Description: TFTP

service atftpd start
iptables-restore -c < /etc/iptables.rules

The file must be in the above format to be compatible with chkconfig which runs the script at startup. I left the defaults in from the Image Builder Service as it made sense they started at the same time and had the same dependencies. If you wish to modify further see the following sources: Bash commands, Script Options, Startup, and for the iptables commands.

Press escape to leave the editor and :wq to save the file and quit.

Next set execute permissions on the script by running chmod +x, for example: chmod +x

To set the script to run at startup use chkconfig –add, for example: chkconfig –add

Reboot the vCenter appliance to test the script is running. If successful the atftpd service will be started and port 69 allowed, you can check these with service atftpd status and iptables -nL | grep 69.

Close the session and disable SSH if required.

Configure DHCP

In this example I will be using PXE boot to boot the ESXi hosts using a DHCP reservation. On the DHCP scope that will be serving the hosts I have configured reservations and enabled options 066 and 067. In the value for option 066 (Boot Server Host Name) goes the IP address or FQDN of the vCenter Server where TFTP is running. In the value for option 067 (Bootfile Name) I have entered the BIOS DHCP File Name (undionly.kpxe.vmw-hardwired).


Now that Auto Deploy is up and running using the built-in components of VCSA 6.5 you can begin creating ESXi images and deploy rules to boot hosts; using the Auto Deploy GUI. See the VMware Auto Deploy 6.x Guide.

VMware Auto Deploy 6.x Guide

Auto Deploy has some great new features in vSphere 6.5 including a hugely improved and feature rich GUI built into the web client. The Auto Deploy Guide has been updated for vSphere 6.5 including a walk through of the Auto Deploy GUI and step by step creation of a custom ESXi image with deploy rules to boot ESXi hosts.

What is Auto Deploy?

VMware Auto Deploy works with Host Profiles to provision and customise your ESXi estate. When a host is deployed using Auto Deploy the state information is loaded to memory upon boot, the state is not permanently stored on the physical host by default. In theory this means there is no requirement for local storage, however we can configure stateless caching or stateful installs for use with a local drive if required.

What are the benefits?

After the initial setup you’ll benefit from quick and easy ESXi deployments, patches, driver updates, etc. provisioning of hosts on a mass scale all running the same version of code, huge savings in configuration times for new hosts and a more reliable and stable environment.

What do I need to make it work?

  • An Active Directory DNS, DHCP infrastructure already in place, you’ll need to enable DHCP options 66 and 67…

View original post 2,568 more words

VMware Auto Deploy 6.x Guide

What is Auto Deploy?

VMware Auto Deploy works with Host Profiles to provision and customise your ESXi estate. When a host is deployed using Auto Deploy the state information is loaded to memory upon boot, the state is not permanently stored on the physical host by default. In theory this means there is no requirement for local storage, however we can configure stateless caching or stateful installs for use with a local drive if required.

What are the benefits?

After the initial setup you’ll benefit from quick and easy ESXi deployments, patches, driver updates, etc. provisioning of hosts on a mass scale all running the same version of code, huge savings in configuration times for new hosts and a more reliable and stable environment.


What do I need to make it work?

  • An Active Directory DNS, DHCP infrastructure already in place, you’ll need to enable DHCP options 66 and 67 so make sure your DHCP server supports this if it isn’t Windows.
  • A TFTP server for the host to load the gPXE bootloader from.
  • An Auto Deploy server to stream the ESXi image.
  • A syslog collector and ESXi dump collector are required to gather your ESXi logs and crash dumps.
  • A vCenter server and Enterprise Plus licensing for your hosts.
  • ESXi images and deploy rules can be managed using Power CLI or the vSphere web client.

What happens if my Auto Deploy server goes down?

Initially nothing, don’t forget the Auto Deploy server is only for booting your hosts, so if it goes down it won’t take the hosts with it. However if one of your ESXi hosts reboots while the Auto Deploy server is down then it won’t be able to boot back up. With this in mind VMware introduced stateless caching mode, when enabled this works by caching the ESXi image to a local drive (which can simply be a USB drive or SD card). The host will continue to boot from the Auto Deploy server, however should the Auto Deploy server be unavailable then the host will fall back to boot from the cached image. Stateful install is an installation method for deploying ESXi to a local drive on the server to boot from, so has no dependency on the Auto Deploy server post install.

Any design considerations?

If you are running a large environment and are likely to be simultaneously booting a high number of hosts at once then you may want to consider using reverse proxy servers. This distributes the load and reduces the strain on the Auto Deploy server. You’ll also need to decide if your hosts should use stateless caching mode, as outlined above. Read through the Auto Deploy Best Practises Guide for vSphere 6.0 or vSphere 6.5.


Install Process

Previously, in vSphere 5.5, Auto Deploy was a manual install from the vCenter Server installation media. From vSphere 6.x onwards Auto Deploy is included in the vCenter management node, it just needs enabling.

The steps below outline the process for versions 6.0 and 6.5, if you are configuring Auto Deploy on vSphere 6.5 and/or want to use the embedded TFTP server in the vCenter Server Appliance then refer to the vCenter Server Appliance Integrated TFTP Server post, which runs through the process in more detail.

  • Open the vSphere web portal and log in with an account that is a member of the System Configuration Administrators group, you may need to use the SSO admin account.
  • Browse to Administration > System Configuration > Services.
  • Select Auto Deploy from the list of services.
  • From the dropdown Actions menu click Edit Startup Type and change to Automatic. Start the service.


  • Additional step for vSphere 6.5 only: to allow us to manage Auto Deploy using the GUI we also need to enable the Image Builder service; select the Image Builder Service from the list of services.
  • From the dropdown Actions menu click Edit Startup Type and change to Automatic. Start the service.


  • Auto Deploy requires a syslog and ESXi dump collector, these are built into the vCenter installation. The VMware Syslog Collector is enabled by default with versions of vSphere 6.x.

Locate and enable the VMware vSphere ESXi Dump Collector service.


  • Log out of the vSphere web client and log back in.
  • Click the Home icon, additional Auto Deploy options have now appeared.


  • Still within the vSphere web portal go back to the inventory and highlight your vCenter.
  • Click Configure/Manage > Settings > Auto Deploy.
  • Select Download TFTP Boot Zip and keep the download safe; you’ll need this shortly.
  • Make a note of the BIOS DHCP File Name (it should be undionly.kpxe.vmw-hardwired).
  • Next install your TFTP server; I have used Solarwinds TFTP which is free, there is also an embedded TFTP server in the VCSA. Make a note of the TFTP server root directory.
  • Extract the TFTP boot zip we downloaded earlier to the same location as your TFTP server root directory.


  • Next configure your DHCP scope; add reservations for your hosts and enable options 066 and 067.
  • In the value for option 066 (next-server) enter the FQDN of the TFTP boot server.
  • In the value for option 067 (boot-filename) enter the BIOS DHCP File Name (undionly.kpxe.vmw-hardwired).

The initial installation and setup is now complete and we can begin creating and customising ESXi images. Don’t forget to set the boot order on your ESXi hosts to use network boot first (if you plan on using stateless image caching add your local drive as a secondary boot device).

Auto Deploy uses rulesets to determine which image and configuration a host should boot from. Traditionally Auto Deploy has been configured using Power CLI, the Power CLI commands are still applicable and can be used if preferred. The Auto Deploy plugin for vSphere 6.0 introduced something of a GUI to the Windows client, however we will not be covering that since the Windows client is now depreciated from v6.5 onwards. In v6.5 the vSphere web client includes a hugely improved, feature rich, Auto Deploy interface. In this post we will create an image using ESXi Image Builder, touch on Host Profiles; as these go hand in hand with Auto Deploy, and setup some Deploy Rules to ensure the hosts boot correctly. Follow either the CLI configuration or the GUI configuration.

Auto Deploy 6.x CLI Configuration

ESXi Image Builder

We start with ESXi Image Builder; a subset of powerful command line tools used to manage ESXi images. We will upload a vanilla ESXi image and then add an additional driver. If you have an image that you do not want to edit, such as a vendor specific image, then skip the additional software packages stage.

  • First we need a copy of ESXi to work with, navigate to the ESXi download page at and select the ESXi Offline Bundle download of the version you want to install.
  • Make a note of where you have put the download, do not unzip the contents. Create a software repository folder where you can keep your images.
  • Right click vSphere PowerCLI and select run as administrator. Type Connect-VIserver where is the name of your vCenter server.
  • Now we need to add the software depot for the ESXi image , type Add-EsxSoftwareDepot “location\file” specifying the location and file name of your offline bundle. Repeat this for any software packages you want to add to the image, you can add multiple software depots.
  • Next type Get-EsxImageProfile to display the available profiles from your software depot. If the names are truncated as they are in the screenshot below use Get-EsxImageProfile | select name.


If you do not need to add any additional drivers or software packages to your image, e.g. if you have a ready to use image from your hardware vendor, then skip this section and move on to creating a host profile and setting up a deploy rule.

  • We will now clone an ESXi image and add the software package to this image, so that we still have a copy of the original. In the example below the ESXi-6.0.0-20160302001-standard will be cloned to create a new image called Test Image. Change the names and vendor as appropriate.

New-EsxImageProfile -CloneProfile “ESXi-6.0.0-20160302001-standard” -Name “Test Image” -Vendor “VMware, Inc.”

  • Now run Get-EsxSoftwarePackage and make a note of the name of the package you want to add, providing you added the software depot earlier you will see it in this list. In this example we are adding the nimble-psp software package.
  • Add the software package to your ESXi image by running the following command; change the name of your image profile and software package as appropriate.

Add-EsxSoftwarePackage –ImageProfile “Test Image” –SoftwarePackage nimble-psp


  • To check your software package has been successfully applied run the following command which lists the vibs present, replace the image profile name.

Get-EsxImageProfile “Test Image” | select -ExpandProperty viblist

  • Finally export the ESXi image profile to a ZIP folder and place it in your software repository so we have a hard copy of the image.

Export-ESXImageProfile –ImageProfile “Test Image” –ExportToBundle –File “C:\Test”

  • Replace the name and software repository location, if you’d prefer an ISO replace –ExportToBundle with –ExportToIso and .zip with .iso.

Host Profiles

Host profiles allow you to create a customised template of settings to be applied to your ESXi host. You can configure an existing ESXi host and take a copy of the settings as a base for the host profile. If you do not have any existing ESXi hosts in your inventory consider configuring Auto Deploy without a host profile attached, boot the ESXi host and configure as appropriate, then extract the settings to a host profile and update Auto Deploy. When configuring your reference host make sure the syslog and ESXi dump collector are set, for more information see Setting Up a vSphere Auto Deploy Reference Host for vSphere 6.0 or vSphere 6.5.

  • Using the vSphere client or the web client navigate to Home > Host Profiles
  • Click Extract profile from a host.
  • Select the host to export settings from and click Next.
  • Enter a name and description for the host profile, click Next and Finish.

Select the newly created host profile, we have a number of options. Click Actions, Edit Settings to add any further configuration. If you want to use stateless caching it can be configured by expanding ‘System Image Cache Configuration’ and changing the relevant option.


Once you have a host profile setup you can attach it to a host or cluster. You will see in the GUI that the host or cluster shows non-compliant, this is because the profile hasn’t been applied yet. The profile can be applied manually but we’re going to add some deploy rules to do that for us when the host boots. If you require further assistance with Host Profiles see the vSphere Host Profiles Guide.

Deploy Rules

Now we have an ESXi image and host profile we can setup a deploy rule to apply this configuration to our hosts.

  • To setup our deploy rule we’re going to use the command below. Update the rule name and change the item value to the name of the ESXi image you added earlier. Replace Cluster_01 with the name of the cluster to join in vCenter and replace HostProfile with the name of the host profile to use for customisations. The pattern value relates to the IP addresses of the hosts you want this rule to apply to.

New-DeployRule –Name “Test Rule” –Item “Test Image”, Cluster_01, HostProfile –Pattern “ipv4=”

  • If you get the error New-DeployRule could not find a trusted signer, then run $DeployNoSignatureCheck=$true and try again.
  • The rule is created in the working-set and you will see the files getting uploaded to Auto Deploy. We need to activate the rule using Add-DeployRule “Test Rule” with the name of the deploy rule we just setup.
  • Now when you boot a host where your deploy rule is applicable you will see it load ESXi and the customisations you specified in your host profile.
  • If you are updating an existing image then to remove any cached rules before rebooting the host you can also run Get-VMHost hostname | Test-DeployRuleSetCompliance | Repair-DeployRuleSetCompliance replacing hostname with your host.


To remove a deploy rule from the active-set to the working-set use the command below, to delete a deploy rule all together add -Delete at the end of the command.

Get-DeployRule | Remove-DeployRule

To check a host has the correct image use the command below.

Get-VMHost | Test-DeployRuleSetCompliance

If the host lists an incorrect image, e.g. if the image has recently been updated, then run the following command and then reboot your host to boot from the updated / new image.

Get-VMHost | Test-DeployRuleSetCompliance | Repair-DeployRuleSetCompliance

If you want to dive deeper into the options around creating deploy rules use Get-DeployCommand to see the available cmdlets.

Auto Deploy 6.5 GUI Configuration

We will now walk through the new Auto Deploy GUI and create a custom ESXi image with deploy rules to boot ESXi hosts. Log into the vSphere web client and select Auto Deploy from the home page. We’ll go through the tabs one by one, starting with Software Depots. Software depots contain images or software packages. Click the Add Software Depot icon to add an online software depot or a new custom depot which you can add images to later. Click the Import Sofware Depot to upload a zip file such as an ESXi image. Unlike the command line utility, any software depots you add to the GUI are saved there when you return.

Let’s go ahead and make a start. First we need a copy of ESXi to work with, this can be a vendor optimised image or a vanilla copy of ESXi. In our example we have added a software depot containing the HPE optimised ESXi image. We are going to add the Nimble Connection Manager to the image, so I’ve also added the Nimble-NCM zip. You can add any third party drivers or software that vibs are provided for. I’ve created a custom software depot where my new image will be saved. You can add as many different software depots as you like.


Next we want to take the base ESXi image and clone it so we can add some additional software packages. Select the software depot containing the base ESXi image. Highlight the image and click Clone. Enter a name, description and vendor for the new image, and select a software depot where the new image will be stored. In this case I have chosen the new custom software depot I created above.


The software packages from all software depots that were added to Auto Deploy will be listed. Select the package(s) that you want to apply. Set the acceptance level of the image and click Next. The acceptance level determines which level vibs are accepted for installation; VMware Certified being the most strict and Community Supported the most open. For further information on vib signature levels see this page.


Review the details on the summary page and click Finish.


Now when I click the custom software repository I can see my new ESXi image. The bottom right task pane will show a list of software packages included in the image when selected.


Once all the required software packages have been added to the ESXi image we need to create a deploy rule to boot the hosts. Open the Deploy Rules tab, click New Deploy Rule. Enter an name for the deploy rule and specify the variables for identifiying which hosts the deploy rule should apply to, then click Next.

In this example I’m applying the deploy rule to hosts in an IP range.


Select the ESXi image to deploy to the hosts, change the software depot from the drop down menu if needed, then click Next. If you have any issues with vib signatures you can skip the signature checks using the tick box.

Select a host profile to apply to the hosts, and click Next. If you don’t have a host profile see the Host Profiles section above.

Select the location of the hosts and click Next. Review the summary page and click Finish.


The deploy rule is created but in an inactive state. Select the deploy rule and note the options; Activate / Deactivate, Clone, Edit, Delete. Click Activate / Deactivate, a new window will open. Select the newly created deploy rule and click Activate, Next, and Finish.

Now the deploy rule is activated; when you boot a host where the deploy rule is applicable you will see it load ESXi and the customization specified in the host profile. Deploy rules need to be deactivated before they can be edited.


You can setup multiple deploy rules using different images for different clusters or host variables. Hosts using an Auto Deploy ruleset are listed in the Deployed Hosts tab, hosts that didn’t match any deploy rules are listed under Discovered Hosts.