VMware Auto Deploy 6.x Guide

Auto Deploy has some great new features in vSphere 6.5 including a hugely improved and feature rich GUI built into the web client. The Auto Deploy Guide has been updated for vSphere 6.5 including a walk through of the Auto Deploy GUI and step by step creation of a custom ESXi image with deploy rules to boot ESXi hosts.

ESXsi

What is Auto Deploy?

VMware Auto Deploy works with Host Profiles to provision and customise your ESXi estate. When a host is deployed using Auto Deploy the state information is loaded to memory upon boot, the state is not permanently stored on the physical host by default. In theory this means there is no requirement for local storage, however we can configure stateless caching or stateful installs for use with a local drive if required.

What are the benefits?

After the initial setup you’ll benefit from quick and easy ESXi deployments, patches, driver updates, etc. provisioning of hosts on a mass scale all running the same version of code, huge savings in configuration times for new hosts and a more reliable and stable environment.

What do I need to make it work?

  • An Active Directory DNS, DHCP infrastructure already in place, you’ll need to enable DHCP options 66 and 67…

View original post 2,568 more words

VMware Update Manager 6.5 Install Guide

VMware Update Manager provides centralised patch and version management for ESXi hosts, virtual machines, and virtual appliances. Update Manager can be used to upgrade and patch ESXi hosts, install and update third-party software on hosts, upgrade virtual machine hardware, VMware Tools, and virtual appliances. This guide will cover the installation of VMware Update Manager 6.5 on Windows Server 2016. For further information on Update Manager performance and best practises, or for implementing an advanced topology in a large environment, see also the vSphere 6.5 Update Manager Performance and Best Practices post.

Architectural Considerations

  • Update Manager 6.5 is installed on a Windows server, this can be the same server as vCenter. The Update Manager Download Service can be used for secure environments with no internet access.
  • For the vCenter Server Appliance a separate installation is not required since Update Manager is now bundled into the VCSA.
  • Update Manager must be registered with a vCenter Server instance.
  • The following Windows deployment models are recommended by VMware:
    • All-in-one model; embedded database for small environments.
    • Medium deployment mode; Update Manager and vCenter Server installed on the same Windows server with separate external databases. Recommended for most environments.
    • Large deployment model; dedicated Windows server for Update Manager, separate to vCenter Server, with external database. Recommended for deployments where datacentre instances contain over 100 hosts and 1000 virtual machines each.

Software Considerations

  • Update Manager 6.5 requires vCenter Server v6.5, can be used to upgrade and patch hosts running ESXi 5.5 and above, and install VMware Tools for Windows and Linux guests.
  • Update Manager 6.5 requires a 64-bit version of Windows; Windows Server 2008 SP2 64-bit or later.
  • Microsoft .NET Framework 3.5 SP1 should be installed on the Windows server.
  • An external database should be Microsoft SQL Server 2008 R2 SP2 or above, or Oracle 11g or 12c. You can review a full list of compatible versions at the Database Interoperability Matrix.
  • The user requires Oracle DBA role, or SQL sysadmin server role, or db_owner fixed database role on the Update Manager and MSDB database.
  • For environments with up to 5 hosts and 50 VMs the bundled internal installation of SQL Server 2012 Express can be used.

Other Considerations

  • A dedicated Windows server requires a minimum of 2 GB RAM for Update Manager, where Update Manager is installed on the same Windows server as vCenter there should be a minimum of 8 GB RAM.
  • VMware recommend at least 120 GB of free space on the drive where the patching repository will be stored. Use the VMware Sizing Estimator to work out how the capacity required for your environment.
  • Update Manager connects to vCenter Server on TCP port 80. ESXi hosts connect to Update Manager using ports 9084 and 902. A full list of network port requirements can be viewed in this kb.
  • FA reboot is not required post-installation of Update Manager so there is no down time.
  • Official resources – Update Manager 6.5 Documentation Centre, vSphere 6.5 Release Notes

Create Data Source

Before beginning if you intend to use Update Manager with an external SQL database be sure to install the SQL Server Native Client (if vCenter Server is installed on the same Windows server using an external database then this is likely to be already installed).

You must also configure a 64-bit ODBC data source. Data sources can be configured in Windows Server via Control Panel > Administrative Tools > ODBC Data Sources (64-bit).

vumodbc

Click System DNS, Add and input the details for the external database, test the data source before continuing.

Installation

Download the VMware vCenter Server and modules for Windows VMware Downloads; this includes VMware Update Manager 6.5. Mount the ISO and right click autorun.exe, click Run as administrator. Select Server under the vSphere Update Manager menu, tick the Embedded Database Option to include the Microsoft SQL Server 2012 Express installation. Do not select this box if you intend on connecting to an external database. Click Install.

vum1

Select the installation language and click Ok, then click Next to begin the install wizard.

vum3

Accept the license agreement and click Next.

vum5

Review the support information, if the Update Manager instance will not have access to the internet untick Download updates from default sources immediately after installation, otherwise click Next.

vum6

Enter the information for the vCenter Server and click Next.

vum7

If you are using an external database select the Data Source Name for the database and click Next. Select how Update Manager should be identified, host name or IP, accept the default ports and configure a proxy server if required, then click Next.

vum8

Review the default installation locations, change if necessary, and click Next.

vum9

If the installation destination has less than 120 GB of available space you will receive the warning below, click Ok to continue.

vum10

Click Install to begin the installation.

vum11

The installation progress status bar will be displayed.

vum12

Once installation is complete click Finish.

vum13

Post-Installation

After the installation has completed an Update Manager option will be added to the vSphere web client home page in the navigation panel.

Select the newly installed Update Manager. Here you can configure basic settings such as a download schedule, the default settings are pre-configured to get you up and running. Let’s have a quick look through the other tabs.

um1

Hosts Baselines and VM Baselines tabs list Baselines which are attached to hosts or virtual machines. A baseline contains patches, an upgrade image, etc.

um4

The Patch Repository contains downloaded patches, these can be attached to a baseline.

um3

ESXi images can be imported via the ESXi Images tab, these are uploaded as an ISO and can be vanilla ESXi images, vendor provided images, or custom created images. For assistance with creating custom ESXi image using the ESXi Image Builder see the VMware Auto Deploy Guide.

um2

The Update Manager settings are all fairly intuitive; download updates or upload an ESXi image, and create a baseline. Once you have a baseline go to the hosts and clusters view, select the host or virtual machine to update and browse to Manage > Update Manager.

Here you can attach baselines, check compliance, scan for updates, stage patches (downloads but doesn’t install), remediate patches (installs, host will be rebooted), or return to the Update Manager admin view.

um5

The Windows client cannot be used to administer Update Manager 6.5 since it is not supported in vSphere 6.5 onwards.

VMware ESXi 6.x Install Guide

This guide has been validated for the installation of both ESXi 6.0 and ESXi 6.5.

VMware ESXi is a bare-metal hypervisor deployed as a stateful or stateless install. ESXi optimises the use of a physical servers resources to be logically shared out for virtual machines. By virtualising compute we can benefit from a smaller datacentre footprint, less use of power and other hardware such as networking equipment, cost savings from server consolidation, better utilisation and management of existing resources, improved disaster recovery and availability. The graphic on the left below represents traditional workloads running on dedicated servers, the graphic on the right represents a virtualised environment with consolidated workloads.

workloads

 Licensing

  • ESXi is available as a standalone free hypervisor, you will need to sign up for an account with VMware.
  • ESXi is available as part of the vSphere product suite with a rich feature set ranging from Standard to Enterprise Plus with Operations Management; compare editions.
  • The default license installed with ESXi provides Enterprise Plus trial capabilities for a maximum of 60 days. Before the trail expires a valid license key must be installed.

Hardware Considerations

  • ESXi 6.x supports a wide range of hardware across multiple vendors, however before proceeding check the hardware compatibility guide.
  • A host machine with at least 2 CPU cores is required.
  • ESXi 6.x supports 64-bit x86 processors released after September 2006.
  • To provide virtualisation functionality the NX/XD bit must be enabled for the CPU in the BIOS, x64 CPUs should also have the support for hardware virtualisation (Intel VT-x or AMD RVI) feature enabled.
  • ESXi 6.x requires at least 4 GB of physical RAM, however 8 GB is recommended to run virtual machines.
  • Sufficient capacity and redundancy should be provided for network and storage resources.
  • ESXi requires a boot device that is a minimum of 1 GB in size, although additional capacity is recommend:
    • USB or SD card: minimum of 4 GB to allow for extended coredump partition, VMware recommend 16 GB for additional flash cells to prolong the life of the boot media.
    • Local disk, SAN, or iSCSI LUN: a 5.2 GB disk is required for the VMFS volume and a 4 GB scratch partition. If a smaller disk is used then the /scratch partition will be located on the ESXi host ramdisk, which can hamper performance.

Other Considerations

Install

ESXi can be installed manually, which we will do in this post, scripted, or through Auto Deploy.

Boot from the ISO either by using local media or the server remote management software (iLO, DRAC, IMM).

esxi2

  • Press Enter to begin the installation wizard.
  • Press F11 to accept the license agreement.
  • The installer will now scan for available installation targets.
  • Select the destination for the ESXi installation.
  • Select the keyboard language and press Enter to continue.
  • Configure a root password and press Enter to continue.
  • Confirm the installation by pressing F11. The selected disk will be repartitioned.
  • The ESXi installation will now commence.
  • Once the installation is complete press Enter to reboot.
  • The ESXi instance will now boot.

Press F2 to enter the system menu, log in using the root password configured during installation.

Use the Configure Management Network menu to configure an IP address and DNS settings. If applicable you can also select the network adapter(s) to use for the management network, and configure the use of a VLAN.

esxi15

Press escape to return to the home screen, you will be prompted to restart the management network, press Y.

Access your host by opening a web browser and entering https:// followed by the IP or FQDN of your host with a trailing /ui. Log in as root.

web2

The ESXi host is now ready to start provisioning virtual machines.

web3

If you have installed ESXi 6.0 you have the option to access the host using the vSphere Windows client. To download and install the client open a web browser to the IP address of your ESXi host. It is recommended that you use the web interface since the Windows client is no longer supported from version 6.5 onwards.

ESXi can also be managed using the command line, for more information see Troubleshooting with ESXi Shell and ESXi Command Line Upgrades.

As a standalone host ESXi features are limited, to get the full benefits of virtualisation you should look to pool resources together by deploying a Windows vCenter Server or vCenter Server Appliance. Update Manager works alongside vCenter to patch and upgrade your ESXi hosts.

Configuring Nimble Storage

Once your Nimble storage array is setup and zoned with the appropriate servers, you can start presenting volumes. The first step is to create an initiator group; a volume or LUN is mapped to an initiator group which contains the initiator World Wide Port Names (WWPN).

Open a browser and navigate to the IP address of the Nimble device. Select Manage and Initiator Groups, click Create. Enter a name and add the aliases and WWPN of the servers you want to present storage to. You can create multiple initiator groups and a server can be a member of multiple initiator groups.

nimble1

When all your initiators are added and grouped you can proceed to create and map a volume. Click Manage and Volumes, select New Volume.

Enter a volume name and chose a performance policy based on the application or platform you are using, or create your own. From the drop down list of initiator groups you created earlier select the initiator group to present the new volume to. The LUN number is auto populated but can be changed, click Next.

nimble-2

Enter the size of the volume and any reserves or quotas you wish to apply, click Next. Volumes are thin provisioned by default.

If you want to configure snapshots to protect your volume, or volume caching you can do so here by selecting the relevant options, otherwise click No volume collection, Next and Finish.

The volume is now mapped to the initiator. There are a couple of further steps you should take on your server before using the newly presented storage.

nimble4

Windows Initiator

If you are presenting Nimble storage direct to a Windows host you should install the Nimble Windows Toolkit. The toolkit includes the Nimble Device Specific Module (DSM) and you will need the Multipath I/O (MPIO) feature installed. For the purpose of this post I will be installing version 2.3.2 of the Nimble Windows Toolkit which requires Windows 2008 R2 or later and .NET Framework 4.5.2.

Log in to Infosight and select Software Downloads from the Resources drop down. Click Windows Toolkit and download the appropriate version. Check for any required Windows hot-fixes in the release notes.

nwt

On your Windows server run the setup as an administrator and follow the onscreen prompts. Once the Nimble Windows Toolkit is installed the server will require a reboot. You will then see Nimble Connection Manager listed in programs, and the Nimble DSM in use on the properties of the disk in Disk Management.

ESXi Initiator

If you are presenting Nimble storage to an ESXi host you can install Nimble Connection Manager which includes the Nimble Path Selection Policy (PSP) for VMware. You will need ESXi 5.x or above.

Log in to Infosight.nimblestorage.com and select Software Downloads from the Resources drop down. Click Connection Manager (NCM) for VMware and download the appropriate version.

Within the zip file that is downloaded you will find the Nimble vibs; you can either deploy the zip file using VMware Update Manager, inject the vibs to your hosts manually or build them into your image if you are using VMware Auto Deploy.

nmp

VMware Auto Deploy 6.x Guide

What is Auto Deploy?

VMware Auto Deploy works with Host Profiles to provision and customise your ESXi estate. When a host is deployed using Auto Deploy the state information is loaded to memory upon boot, the state is not permanently stored on the physical host by default. In theory this means there is no requirement for local storage, however we can configure stateless caching or stateful installs for use with a local drive if required.

What are the benefits?

After the initial setup you’ll benefit from quick and easy ESXi deployments, patches, driver updates, etc. provisioning of hosts on a mass scale all running the same version of code, huge savings in configuration times for new hosts and a more reliable and stable environment.

What do I need to make it work?

  • An Active Directory DNS, DHCP infrastructure already in place, you’ll need to enable DHCP options 66 and 67 so make sure your DHCP server supports this if it isn’t Windows.
  • A TFTP server for the host to load the gPXE bootloader from.
  • An Auto Deploy server to stream the ESXi image.
  • A syslog collector and ESXi dump collector are required to gather your ESXi logs and crash dumps.
  • A vCenter server and Enterprise Plus licensing for your hosts.
  • ESXi images and deploy rules can be managed using Power CLI or the vSphere web client.

What happens if my Auto Deploy server goes down?

Initially nothing, don’t forget the Auto Deploy server is only for booting your hosts, so if it goes down it won’t take the hosts with it. However if one of your ESXi hosts reboots while the Auto Deploy server is down then it won’t be able to boot back up. With this in mind VMware introduced stateless caching mode, when enabled this works by caching the ESXi image to a local drive (which can simply be a USB drive or SD card). The host will continue to boot from the Auto Deploy server, however should the Auto Deploy server be unavailable then the host will fall back to boot from the cached image. Stateful install is an installation method for deploying ESXi to a local drive on the server to boot from, so has no dependency on the Auto Deploy server post install.

Any design considerations?

If you are running a large environment and are likely to be simultaneously booting a high number of hosts at once then you may want to consider using reverse proxy servers. This distributes the load and reduces the strain on the Auto Deploy server. You’ll also need to decide if your hosts should use stateless caching mode, as outlined above. Read through the Auto Deploy Best Practises Guide for vSphere 6.0 or vSphere 6.5.

autodeploy

Install Process

Previously, in vSphere 5.5, Auto Deploy was a manual install from the vCenter Server installation media. From vSphere 6.x onwards Auto Deploy is included in the vCenter management node, it just needs enabling.

The steps below outline the process for versions 6.0 and 6.5, if you are configuring Auto Deploy on vSphere 6.5 and/or want to use the embedded TFTP server in the vCenter Server Appliance then refer to the vCenter Server Appliance Integrated TFTP Server post, which runs through the process in more detail.

  • Open the vSphere web portal and log in with an account that is a member of the System Configuration Administrators group, you may need to use the SSO admin account.
  • Browse to Administration > System Configuration > Services.
  • Select Auto Deploy from the list of services.
  • From the dropdown Actions menu click Edit Startup Type and change to Automatic. Start the service.

autodeploy1

  • Additional step for vSphere 6.5 only: to allow us to manage Auto Deploy using the GUI we also need to enable the Image Builder service; select the Image Builder Service from the list of services.
  • From the dropdown Actions menu click Edit Startup Type and change to Automatic. Start the service.

autodeploy2

  • Auto Deploy requires a syslog and ESXi dump collector, these are built into the vCenter installation. The VMware Syslog Collector is enabled by default with versions of vSphere 6.x.

Locate and enable the VMware vSphere ESXi Dump Collector service.

syslog

  • Log out of the vSphere web client and log back in.
  • Click the Home icon, additional Auto Deploy options have now appeared.

autodeploy3

  • Still within the vSphere web portal go back to the inventory and highlight your vCenter.
  • Click Configure/Manage > Settings > Auto Deploy.
  • Select Download TFTP Boot Zip and keep the download safe; you’ll need this shortly.
  • Make a note of the BIOS DHCP File Name (it should be undionly.kpxe.vmw-hardwired).
  • Next install your TFTP server; I have used Solarwinds TFTP which is free, there is also an embedded TFTP server in the VCSA. Make a note of the TFTP server root directory.
  • Extract the TFTP boot zip we downloaded earlier to the same location as your TFTP server root directory.

tftp

  • Next configure your DHCP scope; add reservations for your hosts and enable options 066 and 067.
  • In the value for option 066 (next-server) enter the FQDN of the TFTP boot server.
  • In the value for option 067 (boot-filename) enter the BIOS DHCP File Name (undionly.kpxe.vmw-hardwired).

The initial installation and setup is now complete and we can begin creating and customising ESXi images. Don’t forget to set the boot order on your ESXi hosts to use network boot first (if you plan on using stateless image caching add your local drive as a secondary boot device).

Auto Deploy uses rulesets to determine which image and configuration a host should boot from. Traditionally Auto Deploy has been configured using Power CLI, the Power CLI commands are still applicable and can be used if preferred. The Auto Deploy plugin for vSphere 6.0 introduced something of a GUI to the Windows client, however we will not be covering that since the Windows client is now depreciated from v6.5 onwards. In v6.5 the vSphere web client includes a hugely improved, feature rich, Auto Deploy interface. In this post we will create an image using ESXi Image Builder, touch on Host Profiles; as these go hand in hand with Auto Deploy, and setup some Deploy Rules to ensure the hosts boot correctly. Follow either the CLI configuration or the GUI configuration.

Auto Deploy 6.x CLI Configuration

ESXi Image Builder

We start with ESXi Image Builder; a subset of powerful command line tools used to manage ESXi images. We will upload a vanilla ESXi image and then add an additional driver. If you have an image that you do not want to edit, such as a vendor specific image, then skip the additional software packages stage.

  • First we need a copy of ESXi to work with, navigate to the ESXi download page at vmware.com and select the ESXi Offline Bundle download of the version you want to install.
  • Make a note of where you have put the download, do not unzip the contents. Create a software repository folder where you can keep your images.
  • Right click vSphere PowerCLI and select run as administrator. Type Connect-VIserver where is the name of your vCenter server.
  • Now we need to add the software depot for the ESXi image , type Add-EsxSoftwareDepot “location\file” specifying the location and file name of your offline bundle. Repeat this for any software packages you want to add to the image, you can add multiple software depots.
  • Next type Get-EsxImageProfile to display the available profiles from your software depot. If the names are truncated as they are in the screenshot below use Get-EsxImageProfile | select name.

powercli1

If you do not need to add any additional drivers or software packages to your image, e.g. if you have a ready to use image from your hardware vendor, then skip this section and move on to creating a host profile and setting up a deploy rule.

  • We will now clone an ESXi image and add the software package to this image, so that we still have a copy of the original. In the example below the ESXi-6.0.0-20160302001-standard will be cloned to create a new image called Test Image. Change the names and vendor as appropriate.

New-EsxImageProfile -CloneProfile “ESXi-6.0.0-20160302001-standard” -Name “Test Image” -Vendor “VMware, Inc.”

  • Now run Get-EsxSoftwarePackage and make a note of the name of the package you want to add, providing you added the software depot earlier you will see it in this list. In this example we are adding the nimble-psp software package.
  • Add the software package to your ESXi image by running the following command; change the name of your image profile and software package as appropriate.

Add-EsxSoftwarePackage –ImageProfile “Test Image” –SoftwarePackage nimble-psp

powercli2

  • To check your software package has been successfully applied run the following command which lists the vibs present, replace the image profile name.

Get-EsxImageProfile “Test Image” | select -ExpandProperty viblist

  • Finally export the ESXi image profile to a ZIP folder and place it in your software repository so we have a hard copy of the image.

Export-ESXImageProfile –ImageProfile “Test Image” –ExportToBundle –File “C:\Test Image.zip”

  • Replace the name and software repository location, if you’d prefer an ISO replace –ExportToBundle with –ExportToIso and .zip with .iso.

Host Profiles

Host profiles allow you to create a customised template of settings to be applied to your ESXi host. You can configure an existing ESXi host and take a copy of the settings as a base for the host profile. If you do not have any existing ESXi hosts in your inventory consider configuring Auto Deploy without a host profile attached, boot the ESXi host and configure as appropriate, then extract the settings to a host profile and update Auto Deploy. When configuring your reference host make sure the syslog and ESXi dump collector are set, for more information see Setting Up a vSphere Auto Deploy Reference Host for vSphere 6.0 or vSphere 6.5.

  • Using the vSphere client or the web client navigate to Home > Host Profiles
  • Click Extract profile from a host.
  • Select the host to export settings from and click Next.
  • Enter a name and description for the host profile, click Next and Finish.

Select the newly created host profile, we have a number of options. Click Actions, Edit Settings to add any further configuration. If you want to use stateless caching it can be configured by expanding ‘System Image Cache Configuration’ and changing the relevant option.

hostprofile

Once you have a host profile setup you can attach it to a host or cluster. You will see in the GUI that the host or cluster shows non-compliant, this is because the profile hasn’t been applied yet. The profile can be applied manually but we’re going to add some deploy rules to do that for us when the host boots. If you require further assistance with Host Profiles see the vSphere Host Profiles Guide.

Deploy Rules

Now we have an ESXi image and host profile we can setup a deploy rule to apply this configuration to our hosts.

  • To setup our deploy rule we’re going to use the command below. Update the rule name and change the item value to the name of the ESXi image you added earlier. Replace Cluster_01 with the name of the cluster to join in vCenter and replace HostProfile with the name of the host profile to use for customisations. The pattern value relates to the IP addresses of the hosts you want this rule to apply to.

New-DeployRule –Name “Test Rule” –Item “Test Image”, Cluster_01, HostProfile –Pattern “ipv4=192.168.0.10-192.168.0.15”

  • If you get the error New-DeployRule could not find a trusted signer, then run $DeployNoSignatureCheck=$true and try again.
  • The rule is created in the working-set and you will see the files getting uploaded to Auto Deploy. We need to activate the rule using Add-DeployRule “Test Rule” with the name of the deploy rule we just setup.
  • Now when you boot a host where your deploy rule is applicable you will see it load ESXi and the customisations you specified in your host profile.
  • If you are updating an existing image then to remove any cached rules before rebooting the host you can also run Get-VMHost hostname | Test-DeployRuleSetCompliance | Repair-DeployRuleSetCompliance replacing hostname with your host.

autodeployboot

To remove a deploy rule from the active-set to the working-set use the command below, to delete a deploy rule all together add -Delete at the end of the command.

Get-DeployRule | Remove-DeployRule

To check a host has the correct image use the command below.

Get-VMHost | Test-DeployRuleSetCompliance

If the host lists an incorrect image, e.g. if the image has recently been updated, then run the following command and then reboot your host to boot from the updated / new image.

Get-VMHost | Test-DeployRuleSetCompliance | Repair-DeployRuleSetCompliance

If you want to dive deeper into the options around creating deploy rules use Get-DeployCommand to see the available cmdlets.

Auto Deploy 6.5 GUI Configuration

We will now walk through the new Auto Deploy GUI and create a custom ESXi image with deploy rules to boot ESXi hosts. Log into the vSphere web client and select Auto Deploy from the home page. We’ll go through the tabs one by one, starting with Software Depots. Software depots contain images or software packages. Click the Add Software Depot icon to add an online software depot or a new custom depot which you can add images to later. Click the Import Sofware Depot to upload a zip file such as an ESXi image. Unlike the command line utility, any software depots you add to the GUI are saved there when you return.

Let’s go ahead and make a start. First we need a copy of ESXi to work with, this can be a vendor optimised image or a vanilla copy of ESXi. In our example we have added a software depot containing the HPE optimised ESXi image. We are going to add the Nimble Connection Manager to the image, so I’ve also added the Nimble-NCM zip. You can add any third party drivers or software that vibs are provided for. I’ve created a custom software depot where my new image will be saved. You can add as many different software depots as you like.

GUI1

Next we want to take the base ESXi image and clone it so we can add some additional software packages. Select the software depot containing the base ESXi image. Highlight the image and click Clone. Enter a name, description and vendor for the new image, and select a software depot where the new image will be stored. In this case I have chosen the new custom software depot I created above.

GUI2

The software packages from all software depots that were added to Auto Deploy will be listed. Select the package(s) that you want to apply. Set the acceptance level of the image and click Next. The acceptance level determines which level vibs are accepted for installation; VMware Certified being the most strict and Community Supported the most open. For further information on vib signature levels see this page.

GUI3

Review the details on the summary page and click Finish.

GUI4

Now when I click the custom software repository I can see my new ESXi image. The bottom right task pane will show a list of software packages included in the image when selected.

GUI5

Once all the required software packages have been added to the ESXi image we need to create a deploy rule to boot the hosts. Open the Deploy Rules tab, click New Deploy Rule. Enter an name for the deploy rule and specify the variables for identifiying which hosts the deploy rule should apply to, then click Next.

In this example I’m applying the deploy rule to hosts in an IP range.

GUI6

Select the ESXi image to deploy to the hosts, change the software depot from the drop down menu if needed, then click Next. If you have any issues with vib signatures you can skip the signature checks using the tick box.

Select a host profile to apply to the hosts, and click Next. If you don’t have a host profile see the Host Profiles section above.

Select the location of the hosts and click Next. Review the summary page and click Finish.

GUI7

The deploy rule is created but in an inactive state. Select the deploy rule and note the options; Activate / Deactivate, Clone, Edit, Delete. Click Activate / Deactivate, a new window will open. Select the newly created deploy rule and click Activate, Next, and Finish.

Now the deploy rule is activated; when you boot a host where the deploy rule is applicable you will see it load ESXi and the customization specified in the host profile. Deploy rules need to be deactivated before they can be edited.

GUI8

You can setup multiple deploy rules using different images for different clusters or host variables. Hosts using an Auto Deploy ruleset are listed in the Deployed Hosts tab, hosts that didn’t match any deploy rules are listed under Discovered Hosts.

Troubleshooting with ESXTOP

This post will take a look over some of the useful commands for troubleshooting performance using ESXTOP. To access esxtop open a connection to your host using the ESXi shell. This can be done with an SSH client, such as putty, using port 22 and your root login details. For more information on enabling and accessing the ESXi Shell see Using the ESXi Shell.

Once you are logged into the command line interface simply type ‘esxtop’ and hit enter. Within the esxtop screen we can use different keys to change the view and examine certain metrics such as CPU ‘c’, memory ‘m’, network ‘n’, disk ‘d’. We will look at the main parameters below for troubleshooting performance issues on an ESXi host.

At any time you can press space to update the display (although this refreshes automatically every 5 seconds) ‘q’ to quit and ‘h’ for help. Add or remove new fields using ‘f’ and if you want to save this configuration for next time type an upper-case ‘W‘.

CPU

In order to understand the information presented by esxtop we must first understand the role of the CPU scheduler. ESXi pools resources together, such as logical processors, which can then be used independently by the CPU scheduler. A virtual machine can have access to virtual processors running on logical processors within the same core, or on different physical cores. It is the job of the CPU scheduler to examine processor topology between sockets, cores and logical processors to optimise the placement of virtual CPUs across the system.

The second point we need to understand is that esxtop uses worlds to show CPU usage. A world is a VMkernel schedulable entity, similar to a processes or thread in other operating systems. You will see references to both the CPU scheduler and VM worlds in the explanations below.

Once in the esxtop screen press ‘c’ to display CPU statistics.

esxtopc

CPU load average relates to the average CPU usage for the ESXi host over the last 1, 5 and 15 minutes. A load average of 0.5 suggests CPU is half utilised, 1.0 suggests CPU is fully utilised and a value above this would mean the ESXi host is using more physical CPUs than currently available.

%RDY column shows the percentage of time spent waiting for the CPU scheduler. Consider that 1% is roughly 200 milliseconds and 100% is roughly 20,000 milliseconds. Therefore a value between 5% and 10% or higher (or 1,000 to 2,000 milliseconds) could potentially be a cause for concern.

%USED amount of time spent executing CPU core cycles by the virtual machine. A substantially higher value on one virtual machine compared with others could mean it is the cause of performance issues on the host.

%SYS shows the percentage of time spent performing system activities on behalf of the world. A value of 10% to 20% or higher could be a symptom of a high IO virtual machine.

 %CSTP is a value for virtual machines with multiple vCPUs, and shows the time spent waiting for one or more of those virtual CPUs to become ready. If this is above 3% it generally means the number of vCPUs should be decreased.

%MLMTD percentage of time a ready to run vCPU was not scheduled due to a CPU limit setting. If this value is above 0 then the limit should be removed to improve performance.

%SWPWT relates to the time spent waiting for swapped pages to be read from disk. If the value exceeds 5 you could potentially have an issue with memory over-commitment.

Memory

Once in the esxtop screen press ‘m’ to display memory statistics.

At the top of the screen you will see PMEB /MB. These values are in MB and list the total memory of the host (total) memory used by the ESXi VMKernel (VMK) other memory in use (other) and free memory (free).

Another useful field is the memory state, this categorises the ESXi host memory state based on how much of the minimum free memory amount is available. A high state means there is enough free memory available to the host, clear means there is less than 100% of the minfree and ESXi begins actively calling TPS to collapse pages, soft means there is less than 64% of minfree and the host beings reclaiming memory using the balloon driver, hard means there is less than 32% of minfree and the host swaps and compresses memory, low means there is less than 16% of minfree and ESXi blocks VMs from allocating more RAM.

esxtopm

Other useful columns are listed below, mainly these relate to over-commitment of physical memory.

MEM overcommit avg shows the average memory overcommit for the last 1, 5 and 15 minutes.

MCTLSZ amount of guest physical memory in MB that the ESXi host is reclaiming by inflating the balloon driver. This occurs when the host is over committed and does not have enough available physical memory.

SWCUR amount of memory in MB swapped by the VMKernel, again any value over 0 is another symptom of memory over-commitment.

SWR/s and SWW/s shows the rate at which the host is reading or writing to swapped memory, once again any value over 0 indicates possible memory over-commitment.

CACHEUSD shows the amount of memory in MB that has been compressed by the ESXi host. Compression occurs when the host is over-committed on memory so this should not be above 0.

ZIP/s and UNZIP/s indicates the host is actively compressing memory and accessing compressed memory respectively. Values larger than 0 imply the host is over-committed on memory.

Network

Once in the esxtop screen press ‘n’ to display network statistics.

For network you can look at the %DRPTX and %DRPRX columns. These represent the dropped packets transmitted and dropped packets received respectively. Values above 0 here could signify high network utilisation.

The USED-BY and TEAM-PNIC columns display a list of the virtual machines on the host and the vmnic that it is using.

Outside of esxtop, but also useful, back at the command line use ‘esxcli network nic list’ to list the available network adapters. To see the stats such as packets/bytes transmitted/received and dropped enter ‘esxcli network nic stats get –n vmnic0’ changing the vmnic as appropriate.

esxtopn

These are just some parts of esxtop I have found useful, it goes much deeper with more commands and syntax here: https://communities.vmware.com/docs/DOC-9279.