Guest Introspection Lost Communication

Quick post to reference an issue with Guest Introspection ESX Agents briefly but consistently losing connectivity. There are a number of causes to Guest Introspection communication errors; such as network and certificate issues, administrator@vsphere.local or service account lock outs, and firewall or network misconfiguration.

In our case the lost communication errors would occur and then reconnect, with the status changing back to green, mostly straight away but within a maximum of one second. Thus not causing any workflow issues, but enough to generate an alert. The following errors were found in the host and ESX Agent event logs.

[VMware vCenter – Alarm Guest Introspection Host status] Lost communication with ESX module.

Alarm Definition: ([Event alarm expression: ESX module enabled.  Supporting versions {0} of the ESX module protocol and {1} of the VFile protocol.; Status = Green] OR [Event alarm expression: The ESX module was uninstalled.; Status = Green] OR [Event alarm expression: Lost communication with ESX module.; Status = Red])

Event details: Lost communication with ESX module.

[VMware vCenter – Alarm Guest Introspection SVM status] A connection between the ESX module and the Guest Introspection solution, 1.5.3, failed.

Alarm Definition: ([Event alarm expression: Guest Introspection solution, {1}, enabled. Supporting version {0} of the VFile protocol.; Status = Green] OR [Event alarm expression: A connection between the ESX module and the Guest Introspection solution, {0}, failed.; Status = Red] OR [Event alarm expression: The Guest Introspection solution, {2}, was contacted by an incompatible version of the ESX module. The solution version is {0} and the ESX module version is {1}.; Status = Red] OR [Event alarm expression: Failed to receive status from Guest Introspection solution {0}.; Status = Red])

Event details: A connection between the ESX module and the Guest Introspection solution, 1.5.3, failed.

events

This is a known issue with versions 6.2.3 and 6.2.4 of NSX. There is a work around which I have copied below from the release notes of both versions, however this is potentially only feasible if you have a low number of hosts.

Issue 1662842: Guest Introspection: Connectivity lost between MUX and USVM when trying to resolve unresolvable Windows SIDs

Guest Introspection service will go into a warning state, with each Guest Introspection going in and out of a warning state. Until the Guest Introspection VM reconnects, network events will not be delivered to the NSX Manager. This will affect both Activity Monitoring and ID Firewall in the case that logon events are detected through the Guest Introspection path.

Workaround: To return Guest Introspection to a stable state, Guest Introspection VMs must be configured to ignore lookups for these well-known SIDs. This is achieved by updating a configuration file on each Guest Introspection VM and then restarting the service. In addition, Active Directory log scraping can be used as a workaround for detecting logon events for ID Firewall.

Steps to ignore SID lookups for unresolvable SIDs:

  1. Login to Guest Introspection VM.
  2. Edit the file at /usr/local/usvmmgmt/config/ignore-sids.lst.
  3. Append the following 2 lines: S-1-18-1, S-1-18-2.
  4. Save and close the file.
  5. Restart the Guest Introspection service with command: rcusvm restart.

Update – issue is fixed in NSX 6.3.0. Download NSX 6.3 here, NSX 6.3 release notes here, NSX 6.3 what’s new here. For assistance with upgrading NSX Manager see the VMware NSX Manager Upgrade post.

Fixed Issue 1662842: Guest Introspection: Connectivity lost between MUX and USVM when trying to resolve unresolvable Windows SIDs

Guest Introspection service will go into a warning state, with each Guest Introspection going in and out of a warning state. Until the Guest Introspection VM reconnects, network events will not be delivered to the NSX Manager. This will affect both Activity Monitoring and ID Firewall in the case that logon events are detected through the Guest Introspection path. Fixed in 6.3.0.

2 comments

  1. whats the password for Guest Introspection VM

    Like

    1. I’m not aware of it being documented anywhere, changing settings direct on the ESX Agent isn’t recommended, you should be redeploying the agents if there is something you want to change. If you are in a situation that requires you to log in direct then best to engage VMware support.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: