Guest Introspection Lost Communication

Quick post to reference an issue with Guest Introspection ESX Agents briefly but consistently losing connectivity. There are a number of causes to Guest Introspection communication errors; such as network and certificate issues, administrator@vsphere.local or service account lock outs, and firewall or network misconfiguration.

In our case the lost communication errors would occur and then reconnect, with the status changing back to green, mostly straight away but within a maximum of one second. Thus not causing any workflow issues, but enough to generate an alert. The following errors were found in the host and ESX Agent event logs.

[VMware vCenter – Alarm Guest Introspection Host status] Lost communication with ESX module.

Alarm Definition: ([Event alarm expression: ESX module enabled.  Supporting versions {0} of the ESX module protocol and {1} of the VFile protocol.; Status = Green] OR [Event alarm expression: The ESX module was uninstalled.; Status = Green] OR [Event alarm expression: Lost communication with ESX module.; Status = Red])

Event details: Lost communication with ESX module.

[VMware vCenter – Alarm Guest Introspection SVM status] A connection between the ESX module and the Guest Introspection solution, 1.5.3, failed.

Alarm Definition: ([Event alarm expression: Guest Introspection solution, {1}, enabled. Supporting version {0} of the VFile protocol.; Status = Green] OR [Event alarm expression: A connection between the ESX module and the Guest Introspection solution, {0}, failed.; Status = Red] OR [Event alarm expression: The Guest Introspection solution, {2}, was contacted by an incompatible version of the ESX module. The solution version is {0} and the ESX module version is {1}.; Status = Red] OR [Event alarm expression: Failed to receive status from Guest Introspection solution {0}.; Status = Red])

Event details: A connection between the ESX module and the Guest Introspection solution, 1.5.3, failed.

This is a known issue with versions 6.2.3 and 6.2.4 of NSX. There is a work around which I have copied below from the release notes of both versions, however this is potentially only feasible if you have a low number of hosts.

Issue 1662842: Guest Introspection: Connectivity lost between MUX and USVM when trying to resolve unresolvable Windows SIDs

Guest Introspection service will go into a warning state, with each Guest Introspection going in and out of a warning state. Until the Guest Introspection VM reconnects, network events will not be delivered to the NSX Manager. This will affect both Activity Monitoring and ID Firewall in the case that logon events are detected through the Guest Introspection path.

Workaround: To return Guest Introspection to a stable state, Guest Introspection VMs must be configured to ignore lookups for these well-known SIDs. This is achieved by updating a configuration file on each Guest Introspection VM and then restarting the service. In addition, Active Directory log scraping can be used as a workaround for detecting logon events for ID Firewall.

Steps to ignore SID lookups for unresolvable SIDs:

1. Login to Guest Introspection VM.
2. Edit the file at /usr/local/usvmmgmt/config/ignore-sids.lst.
3. Append the following 2 lines: S-1-18-1, S-1-18-2.
4. Save and close the file.
5. Restart the Guest Introspection service with command: rcusvm restart.

Update – issue is fixed in NSX 6.3.0. Download NSX 6.3 here, NSX 6.3 release notes here, NSX 6.3 what’s new here. For assistance with upgrading NSX Manager see the VMware NSX Manager Upgrade post.

Fixed Issue 1662842: Guest Introspection: Connectivity lost between MUX and USVM when trying to resolve unresolvable Windows SIDs

Guest Introspection service will go into a warning state, with each Guest Introspection going in and out of a warning state. Until the Guest Introspection VM reconnects, network events will not be delivered to the NSX Manager. This will affect both Activity Monitoring and ID Firewall in the case that logon events are detected through the Guest Introspection path. Fixed in 6.3.0.