Category: VMware Troubleshooting

VMware Troubleshooting

vSphere Management Assistant Guide

by esxsi1 Comment on vSphere Management Assistant Guide

The vSphere Management Assistant (vMA) can be used to remotely manage and troubleshoot multiple hosts from the command line. The vSphere Management Assistant is a SUSE Linux Enterprise based virtual appliance deployed within your vSphere infrastructure, it allows centralised management and troubleshooting of multiple ESXi hosts with automatic login, and scripting tools for developers. The vMA appliance includes the vSphere Command Line Interface (vCLI), vSphere SDK for Perl, and components for logging and authentication. The vCLI can also be installed separately on a machine of your choice running Windows or Linux. The standalone vCLI installation allows administrators to run all the commands available within the vMA, if you’re interested in installing vCLI standalone v6.5 can be downloaded here as a simple executable install. Review the release notes here for system requirements.

This post will cover the installation and configuration of vSphere Management Assistant 6.5; compatible with vSphere 5.0 and above. For managing individual hosts, locally or remotely, the ESXi Shell can be used, see the Troubleshooting with ESXi Shell post.

Installing vMA

vSphere Management Assistant v6.5 can be downloaded here, review the release notes here. Unzip the contents of the download and make a note of the file location.

In order to deploy the virtual appliance we need an available Network Protocol Profile. In the vSphere web client browse to the datacentre level where the appliance will reside, select the Manage tab and click Network Protocol Profiles. Click the green plus symbol to create a new profile, follow the wizard and assign the relevant network and settings to the profile.

networkprofile

The vSphere Management Assistant is a simple OVF deployment.

  • In the vSphere web client right click the host or cluster where the virtual appliance will reside. Click Deploy OVF Template.
  • Browse to the downloaded OVF file which was extracted from the .zip download and click Next.
  • Review the details of the appliance and click Next.
  • Accept the license terms and click Next.
  • Enter a name and location for the virtual appliance, click Next.
  • Select the storage to be used and click Next.
  • Select the network to use for the virtual machine and choose the IP allocation (DHCP or static). If static is selected enter the DNS servers, gateway and subnet mask. An additional page prompts for the IP address. Click Next.
  • On the summary page tick Power on after deployment and click Finish.

ovf1

If no Network Protocol profile is present and associated to the network in use then the virtual appliance is unable to power on, you will receive the error Cannot initialize propery ‘vami.netmask0.vSphere_Management_Assistant_(vMA)’. Network ‘VM Network’ has no associated protocol profile. In this case you should ensure the profile has been created and correctly configured.

Once the appliance is powered on open the console. Enter 0 to check the configuration, use the relevant numbers to configure the default gateway, hostname, DNS, and IP address allocation. Once complete enter 1 to exit the setup program.

vma

You will be prompted to change the default password for the vi-admin account, enter the old password vmware and a new password. Once loaded you can connect to the vSphere Management Assistant using an SSH client such as Putty. You can manage the virtual appliance by browsing to https://:5480 where is the IP address or FQDN of the appliance.

Configuring vMA

Open an SSH connection to the IP address or FQDN of the vSphere Management Assistant. Login as the vi-admin user and the password you changed during setup.

The vMA allows administrators to store credentials for automatic authentication when managing ESXi hosts. Using a component called vi-fastpass two accounts are created and the passwords stored in an unreadable format; vi-admin (administrator account) and vi-user (read only). These accounts prevent the user from having to log in to each host and facilitate unattended scheduled script operations.

Alternatively vMA can be configured to use Active Directory for authentication, providing more security controls. To use AD authentication the domain must be accessible from the vMA and DNS must be in place. The following commands are useful for AD tasks in vMA:

  • Join vMA to the domain: sudo domainjoin-cli join domain user where domain is the domain to join and user is the domain user with appropriate privileges.
  • Check the domain status: sudo domainjoin-cli query.
  • Remove vMA from the domain: sudo domainjoin-cli leave.

We can add ESXi hosts or vCenter Servers to vMA using the following commands:

  • To add a system to vMA using the default fastpass authentication: vifp addserver server -authpolicy fpauth -username user -password password where server is the ESXi host or vCenter Server to add, and user and password are the credentials to authenticate with.
  • To add a system to vMA using AD authentication: vifp addserver server –authpolicy adauth –username domain\\user where server is the FQDN of the server and domain\\user is the domain and user to authenticate with.
  • To list the systems added to vMA: vifp listservers.

With the systems authenticated and added to vMA we can now set a target system for executing vCLI commands or vSphere SDK for Perl scripts.

  • Use vifptarget -s server where server is the IP address or FQDN of the vCenter Server or ESXi host. The target system is shown in the command prompt.
  • You can add multiple targets and execute commands across multiple ESXi hosts using the bulkAddServers and mcli scripts, explained in this post by William Lam.

Using vMA

The same commands available to the ESXi shell, such as esxcli, esxcfg, esxtop (resxtop since we are connecting remotely), can be used with vCLI. Furthermore the vCLI includes a subset of vmware-cmd and vicfg commands. You can use more and less commands to assist with truncating information. For example esxcli –help | more and esxcli –help | less. More allows for scrolling down only, use enter to scroll one line at a time and space to scroll a page at a time. Less allows for scrolling both backwards (ctrl + b) and forward (ctrl +f), use q to return back to the command line. The following VMware documentation will get you started with the command line interface.

Let’s take a look at some of the most popular commands. The vmware-cmd command can be used for virtual machine operations, vicfg is primarily used for host operations and is intended to replace esxcfg long term. The main set of commands for managing the vSphere environment you will see is esxcli. The command set is broken down into namespaces, to view the available namespaces just enter esxcli.

namespaces

This propogates down the chain, for example use esxcli storage to view the options within the storage namespace. You can use –help at any level of esxcli for assistance.

storagenamespaces

You can view a full list of esxcli commands by entering esxcli esxcli command list. The screenshot below has been cropped and isn’t a full list, it may be beneficial to drill down through the relevant individual sections using the method outlined above.

list

As you can see the range of esxcli commands is vast, let’s take a look at a few examples.

  • esxcli hardware allows us to view and change the physical server hardware information and configuration. Use esxcli hardware cpu global set to enable or disable hyperthreading.

hardware

  • esxcli system allows us to view and change the ESXi system configuration. To enable or disable maintenance mode use esxcli system maintenanceMode set.

maintenance-mode

  • esxcli storage can be used for storage related tasks, use esxcli storage core path list to view attached LUNs, or esxcli storage vmfs upgrade to upgrade VMFS.

vmfs.PNG

  • esxcli network allows us to perform network related tasks, use esxcli network vswitch standard to create a new standard virtual switch.

switch

For details on patching or upgrading ESXi from the command line see the ESXi Command Line Upgrades post. I also found this great blog post by Chanaka Ekanayake who has put together some of the most useful commands and examples for use with vMA and vCLI.

VMware Troubleshooting

Troubleshooting with ESXi Shell

by esxsiLeave a Comment on Troubleshooting with ESXi Shell

The ESXi Shell gives us a subset of commands for troubleshooting and managing individual ESXi hosts. ESXi Shell can be useful to quickly investigate and resolve issues with single hosts, for example if management agents are unresponsive. This section will cover how to enable ESXi Shell, how to access the ESXi Shell, and how to use the ESXi Shell. It is important to remember that when the ESXi Shell and SSH services are enabled you are potentially opening up vulnerabilities to which attackers may be able to use maliciously. For this reason you will see a warning on any hosts in the web client when the ESXi Shell and / or SSH service is enabled. You can suppress the ESXi Shell warning by following this kb. Remember to disable the ESXi Shell when you have finished, it is also possible to configure time-outs when enabling the ESXi Shell; availability time-out to determine how long ESXi Shell is enabled for, and idle time-out to determine how long idle sessions are kept connected.

You can remotely manage multiple hosts using the vSphere Management Assistant, for more information see the vSphere Management Assistant Guide.

Enabling ESXi Shell

By default the ESXi Shell is disabled, it can be enabled using the DCUI or web client (local or vSphere).

  • DCUI (Direct Console User Interface)
    • Access the console of the ESXi host by plugging in a monitor and keyboard, or establishing a remote console session using remote server tools such as ILO, IMM, etc.
    • Press F2 and enter the root password. Browse to Troubleshooting Options.
    • Select ESXi Shell and press Enter to toggle between enabled and disabled. If you are going to access the Shell locally this is sufficient, for remote connections you must also enable SSH.
    • Press Esc twice to exit out of the menus.

esxishell

  • ESXi host web client (standalone hosts v6.5 and above)
    • Browse to the IP address of FQDN of the host and log in with the root password.
    • From the Navigation menu select Manage, and open the Services tab.
    • Locate and Start TSM for the ESXi Shell, and TSM-SSH for SSH if required.

esxiweb

  •  vSphere web client (hosts connected to vCenter Server)
    • Browse to the IP address or FQDN of the vCenter Server and log in with an administrator account.
    • Locate the host in the inventory and select the Configure tab.
    • Scroll down to the Security Profile menu under System.
    • Click Edit and start the Direct Console UI, ESXi Shell, and SSH services.

vsphereweb

Access ESXi Shell

Once enabled, the ESXi Shell can be accessed locally using the DCUI or remotely over SSH.

  • For DCUI access to the ESXi Shell press ALT + F1 from the ESXi console screen. Log in with the root password.

dcui

  • For remote access open a connection over port 22 using an SSH client such as Putty, and log in with the root password.

putty

Using ESXi Shell

The ESXi Shell contains the full range of esxcli and esxtop commands, as well as esxcfg for legacy purposes (although be aware that esxcfg is depreciated and may be phased out in future releases). The ESXi Shell is useful for performing maintenance and troubleshooting individual hosts, it cannot be used for scheduling scripting jobs. For managing multiple hosts and scripting use vSphere CLI (vCLI) either as a local installation or with the vSphere Management Assistant (vMA).

Have a look in /usr/sbin to view the available commands for the ESXi Shell; enter cd /usr/sbin and then ls. Note that commands are case sensitive.

commands

esxtop is a powerful utility for examining ESXi host performance metrics and investigating performance issues. In the ESXi Shell enter esxtop with variables such as c for CPU, m for memory, n for network, and d for disk, read more in the Troubleshooting with ESXTOP post.

esxcli is a comprehensive set of commands for managing the vSphere environment. The command set is broken down into namespaces, to view the available namespaces use the esxcli command.

namespaces

This propogates down the chain, for example use esxcli storage to view the options within the storage namespace. You can use –help at any level of esxcli for assistance.

storagenamespaces

You can view a full list of esxcli commands by entering esxcli esxcli command list. The screenshot below has been cropped and isn’t a full list, it may be beneficial to drill down through the relevant individual sections using the method outlined above.

list

As you can see the range of esxcli commands is vast, let’s take a look at a few examples.

  • esxcli hardware allows us to view and change the physical server hardware information and configuration. Use esxcli hardware cpu global set to enable or disable hyperthreading.

hardware

  • esxcli system allows us to view and change the ESXi system configuration. To enable or disable maintenance mode use esxcli system maintenanceMode set.

maintenance-mode

  • esxcli storage can be used for storage related tasks, use esxcli storage core path list to view attached LUNs, or esxcli storage vmfs upgrade to upgrade VMFS.

vmfs

  • esxcli network allows us to perform network related tasks, use esxcli network vswitch standard to create a new standard virtual switch.

switch

To exit the ESXi Shell use the exit command. Hopefully this post provides enough to get you started, if you are using ESXi Shell on a regular basis and want to view previously executed commands see this post by William Lam. For details on patching or upgrading ESXi from the command line see the ESXi Command Line Upgrades post.

VMware Troubleshooting

Troubleshooting with ESXTOP

by esxsi1 Comment on Troubleshooting with ESXTOP

This post will take a look over some of the useful commands for troubleshooting performance using ESXTOP. To access esxtop open a connection to your host using the ESXi shell. This can be done with an SSH client, such as putty, using port 22 and your root login details. For more information on enabling and accessing the ESXi Shell see Using the ESXi Shell.

Once you are logged into the command line interface simply type ‘esxtop’ and hit enter. Within the esxtop screen we can use different keys to change the view and examine certain metrics such as CPU ‘c’, memory ‘m’, network ‘n’, disk ‘d’. We will look at the main parameters below for troubleshooting performance issues on an ESXi host.

At any time you can press space to update the display (although this refreshes automatically every 5 seconds) ‘q’ to quit and ‘h’ for help. Add or remove new fields using ‘f’ and if you want to save this configuration for next time type an upper-case ‘W‘.

CPU

In order to understand the information presented by esxtop we must first understand the role of the CPU scheduler. ESXi pools resources together, such as logical processors, which can then be used independently by the CPU scheduler. A virtual machine can have access to virtual processors running on logical processors within the same core, or on different physical cores. It is the job of the CPU scheduler to examine processor topology between sockets, cores and logical processors to optimise the placement of virtual CPUs across the system.

The second point we need to understand is that esxtop uses worlds to show CPU usage. A world is a VMkernel schedulable entity, similar to a processes or thread in other operating systems. You will see references to both the CPU scheduler and VM worlds in the explanations below.

Once in the esxtop screen press ‘c’ to display CPU statistics.

esxtopc

CPU load average relates to the average CPU usage for the ESXi host over the last 1, 5 and 15 minutes. A load average of 0.5 suggests CPU is half utilised, 1.0 suggests CPU is fully utilised and a value above this would mean the ESXi host is using more physical CPUs than currently available.

%RDY column shows the percentage of time spent waiting for the CPU scheduler. Consider that 1% is roughly 200 milliseconds and 100% is roughly 20,000 milliseconds. Therefore a value between 5% and 10% or higher (or 1,000 to 2,000 milliseconds) could potentially be a cause for concern.

%USED amount of time spent executing CPU core cycles by the virtual machine. A substantially higher value on one virtual machine compared with others could mean it is the cause of performance issues on the host.

%SYS shows the percentage of time spent performing system activities on behalf of the world. A value of 10% to 20% or higher could be a symptom of a high IO virtual machine.

 %CSTP is a value for virtual machines with multiple vCPUs, and shows the time spent waiting for one or more of those virtual CPUs to become ready. If this is above 3% it generally means the number of vCPUs should be decreased.

%MLMTD percentage of time a ready to run vCPU was not scheduled due to a CPU limit setting. If this value is above 0 then the limit should be removed to improve performance.

%SWPWT relates to the time spent waiting for swapped pages to be read from disk. If the value exceeds 5 you could potentially have an issue with memory over-commitment.

Memory

Once in the esxtop screen press ‘m’ to display memory statistics.

At the top of the screen you will see PMEB /MB. These values are in MB and list the total memory of the host (total) memory used by the ESXi VMKernel (VMK) other memory in use (other) and free memory (free).

Another useful field is the memory state, this categorises the ESXi host memory state based on how much of the minimum free memory amount is available. A high state means there is enough free memory available to the host, clear means there is less than 100% of the minfree and ESXi begins actively calling TPS to collapse pages, soft means there is less than 64% of minfree and the host beings reclaiming memory using the balloon driver, hard means there is less than 32% of minfree and the host swaps and compresses memory, low means there is less than 16% of minfree and ESXi blocks VMs from allocating more RAM.

esxtopm

Other useful columns are listed below, mainly these relate to over-commitment of physical memory.

MEM overcommit avg shows the average memory overcommit for the last 1, 5 and 15 minutes.

MCTLSZ amount of guest physical memory in MB that the ESXi host is reclaiming by inflating the balloon driver. This occurs when the host is over committed and does not have enough available physical memory.

SWCUR amount of memory in MB swapped by the VMKernel, again any value over 0 is another symptom of memory over-commitment.

SWR/s and SWW/s shows the rate at which the host is reading or writing to swapped memory, once again any value over 0 indicates possible memory over-commitment.

CACHEUSD shows the amount of memory in MB that has been compressed by the ESXi host. Compression occurs when the host is over-committed on memory so this should not be above 0.

ZIP/s and UNZIP/s indicates the host is actively compressing memory and accessing compressed memory respectively. Values larger than 0 imply the host is over-committed on memory.

Network

Once in the esxtop screen press ‘n’ to display network statistics.

For network you can look at the %DRPTX and %DRPRX columns. These represent the dropped packets transmitted and dropped packets received respectively. Values above 0 here could signify high network utilisation.

The USED-BY and TEAM-PNIC columns display a list of the virtual machines on the host and the vmnic that it is using.

Outside of esxtop, but also useful, back at the command line use ‘esxcli network nic list’ to list the available network adapters. To see the stats such as packets/bytes transmitted/received and dropped enter ‘esxcli network nic stats get –n vmnic0’ changing the vmnic as appropriate.

esxtopn

These are just some parts of esxtop I have found useful, it goes much deeper with more commands and syntax here: https://communities.vmware.com/docs/DOC-9279.