Tag Archives: Guest Introspection

Guest Introspection Lost Communication

Quick post to reference an issue with Guest Introspection ESX Agents briefly but consistently losing connectivity. There are a number of causes to Guest Introspection communication errors; such as network and certificate issues, administrator@vsphere.local or service account lock outs, and firewall or network misconfiguration.

In our case the lost communication errors would occur and then reconnect, with the status changing back to green, mostly straight away but within a maximum of one second. Thus not causing any workflow issues, but enough to generate an alert. The following errors were found in the host and ESX Agent event logs.

[VMware vCenter – Alarm Guest Introspection Host status] Lost communication with ESX module.

Alarm Definition: ([Event alarm expression: ESX module enabled.  Supporting versions {0} of the ESX module protocol and {1} of the VFile protocol.; Status = Green] OR [Event alarm expression: The ESX module was uninstalled.; Status = Green] OR [Event alarm expression: Lost communication with ESX module.; Status = Red])

Event details: Lost communication with ESX module.

[VMware vCenter – Alarm Guest Introspection SVM status] A connection between the ESX module and the Guest Introspection solution, 1.5.3, failed.

Alarm Definition: ([Event alarm expression: Guest Introspection solution, {1}, enabled. Supporting version {0} of the VFile protocol.; Status = Green] OR [Event alarm expression: A connection between the ESX module and the Guest Introspection solution, {0}, failed.; Status = Red] OR [Event alarm expression: The Guest Introspection solution, {2}, was contacted by an incompatible version of the ESX module. The solution version is {0} and the ESX module version is {1}.; Status = Red] OR [Event alarm expression: Failed to receive status from Guest Introspection solution {0}.; Status = Red])

Event details: A connection between the ESX module and the Guest Introspection solution, 1.5.3, failed.

events

This is a known issue with versions 6.2.3 and 6.2.4 of NSX. There is a work around which I have copied below from the release notes of both versions, however this is potentially only feasible if you have a low number of hosts.

Issue 1662842: Guest Introspection: Connectivity lost between MUX and USVM when trying to resolve unresolvable Windows SIDs

Guest Introspection service will go into a warning state, with each Guest Introspection going in and out of a warning state. Until the Guest Introspection VM reconnects, network events will not be delivered to the NSX Manager. This will affect both Activity Monitoring and ID Firewall in the case that logon events are detected through the Guest Introspection path.

Workaround: To return Guest Introspection to a stable state, Guest Introspection VMs must be configured to ignore lookups for these well-known SIDs. This is achieved by updating a configuration file on each Guest Introspection VM and then restarting the service. In addition, Active Directory log scraping can be used as a workaround for detecting logon events for ID Firewall.

Steps to ignore SID lookups for unresolvable SIDs:

  1. Login to Guest Introspection VM.
  2. Edit the file at /usr/local/usvmmgmt/config/ignore-sids.lst.
  3. Append the following 2 lines: S-1-18-1, S-1-18-2.
  4. Save and close the file.
  5. Restart the Guest Introspection service with command: rcusvm restart.

Update – issue is fixed in NSX 6.3.0. Download NSX 6.3 here, NSX 6.3 release notes here, NSX 6.3 what’s new here. For assistance with upgrading NSX Manager see the VMware NSX Manager Upgrade post.

Fixed Issue 1662842: Guest Introspection: Connectivity lost between MUX and USVM when trying to resolve unresolvable Windows SIDs

Guest Introspection service will go into a warning state, with each Guest Introspection going in and out of a warning state. Until the Guest Introspection VM reconnects, network events will not be delivered to the NSX Manager. This will affect both Activity Monitoring and ID Firewall in the case that logon events are detected through the Guest Introspection path. Fixed in 6.3.0.

Service Deployment Failed: vibNotInstalled

When installing service deployments such as Guest Introspection through NSX Manager the installation fails. The host task displays error Cannot complete the operation. See the event log for details.

vum3

The event log displays error event.com.vmware.vim.eam.issue.vibNotInstalled.fullFormat.

vum2

The service deployment shows the error Agent VIB module cannot be detected on the host. This may be due to a pending reboot on the host due to an earlier operation. Please check for any errors on the vCenter console of host that may be causing this. Resolving this error will attempt to install the VIB on the host again.

vum4

If an instance of Update Manager is registered with the same vCenter Server as NSX Manager then Update Manager must be fully functional. When Update Manager is present it is relied upon by NSX Manager to approve the install or uninstall or vibs. Resolve the issue with Update Manager, or uninstall Update Manager.

vum1

Once Update Manager is removed or repaired to a working state the service deployments will install successfully.  This VMware KB confirms the issue occurs when Update Manager is unavailable and advises that modules are manually installed instead, however it is not clear in stating that the service deployments succeed without Update Manager, and therefore the UM instance can be removed as a work around.

up

NSX Manager Guest Introspection

Guest introspection is a service that is deployed from NSX Manager to offload security functions to a dedicated security appliance on each host; thus removing the need for an AV agent within the guest operating system.

Using the Guest Introspection driver baked into VMware Tools and a third party service virtual machine, such as McAfee MOVE, all virtual machines are protected by real-time inspection as soon as they are powered on. This reduces administrative and guest memory overheads, whilst standardising deployments.

vShield Manager and Endpoint

Guest introspection functionality was previously achieved using vShield Manager with vShield Endpoint as part of the vCloud Networking and Security suite. NSX Manager v6.2.4 onwards is the replacement product for vShield Manager which has now reached end of life. Guest Introspection replaces vShield Endpoint, you may have noticed in ESXi 5.5 U2 the vShield drivers were renamed to guest introspection drivers as part of the VMware Tools install.

When upgrading from vShield Manager to NSX Manager the vShield Endpoint VIBs are already present on the hosts, these need upgrading to Guest Introspection. For assistance with upgrading from vShield Manager to NSX Manager see the post Upgrading vShield Manager to NSX Manager. This post will detail a clean installation process for the Guest Introspection service, as well as extending an IP Pool for use with Guest Introspection.

NSX Manager and Guest Introspection

Guest introspection is installed on a per cluster basis using the vSphere web client. Deploying Guest Introspection installs a new VIB and ESX Agent on each host in the cluster. You should check with your third party security vendor for compatibility and specific instructions. In most cases, such as with McAfee MOVE, an additional service virtual machine for offloaded anti-malware and AV scanning is deployed to each host.

capture

Both the Guest Introspection ESX Agent and the third party appliance will require storage and a dedicated IP address, this can be configured using either DHCP or a VMware IP pool. The IP addressing of these ESX agents should be factored in to your solution design. The network is provided by a vSphere distributed switch, if you are not using distributed switches then it is possible to set an agent network on each host as a work around under Configuration > Agent VM Settings in vSphere.

To enable Guest Introspection log into the vSphere web client and browse to Networking & Security, then click Installation. Click the green plus symbol to add a new service deployment.

deployment1

In the new service deployment screen select Guest Introspection and click Next.

deployment2

Select the cluster or clusters to deploy the service to and click Next.

Select the storage and management network for the ESX Agents, the default IP assignment is DHCP, ensure the selected network has access to a DHCP server. Alternatively click Change and select IP Pool. You can select an existing IP Pool or create a new one with the necessary network details. If your IP Pool fills up follow the steps at the bottom of this post to extend. When the storage and network settings are configured click Next.

deployment3

Review the details on the confirmation page and click Finish.

The service will now be deployed, the status will be displayed in the Installation Status column. You will also see the ESX agents being deployed in the vSphere recent tasks pane.

deployment5

Once complete the installation status should show succeeded and the service status ok. The Guest Introspection service has now been deployed to the selected clusters and you can move on to deploying and configuring your chosen third party appliance.

deployment7

If you are using stateless environments then you should update the Auto Deploy image with the NSX VIBs, otherwise the guest introspection status will change to not ready after a host is rebooted.

Browse to https://NSX/bin/vdn/nwfabric.properties (where NSX is the IP or FQDN of the NSX Manager) and find the VIB URL for your version of ESXi, open the relevant URL which will auto download vxlan.zip. For assistance with updating Auto Deploy images see the VMware Auto Deploy Guide.

  • Service deployment failed with Agent VIB module cannot be detected on the host? See this post.
  • Guest Introspection intermittently losing connectivity? See this post.

Extending NSX Manager IP Pools

When creating Service Deployments through NSX Manager a new IP Pool can be created for use with the service. During the service deployment wizard although we can create new pools, there is no option to extend an existing pool. In the event a pool requires additional capacity you can follow the steps outlined below.

From the home page of the vSphere web client select Networking & Security, click NSX Managers.

ip1

With the NSX Manager selected open the Manage tab and click Grouping Objects, IP Pools.

The existing IP Pools will be listed, here you can add, remove, and edit IP Pools. The Used / Total column will tell you how many IP addresses have been used in the pool. For this example we have an IP Pool with 22/22 used addresses, we will therefore extend the pool. Select the IP Pool to extend and click the Edit icon.

ip2

ip2

Change the relevant settings in the pop-out window, I will be altering the static IP Pool to include an additional 2 addresses. Click Ok once complete.

ip3

We can see the IP Pool has used 22/24 addresses.

ip4

ip4

Now there are available addresses we can go ahead and use the IP Pool for our new service deployment.