Configuring vRealize Automation with vRealize Log Insight

In this post we will walk through integrating vRealize Automation with vRealize Log Insight to monitor and collect events for the vRA management stack. The following example we will be based on vRealize 7.3 with Log Insight 4.5, but the process has been validated with vRealize 7.x and Log Insight 4.x. If you do not already have Log Insight installed see the vRealize Log Insight Install Guide, if you are also using NSX see the NSX with Log Insight Integration Guide.

The configuration process consists of installing the management pack for Log Insight, installing the Log Insight agent on the Windows components, configuring the built-in Log Insight agent on the vRealize Automation appliances, and creating Log Insight templates and filters to gather the required information.

Management Pack Installation

Browse to the IP address or FQDN of the Log Insight instance and log in using the admin account. From the drop-down menu in the top right select Content Packs and Marketplace. Locate VMware – vRA 7 and click Install.

mp1

The management pack is now installed, review the Setup Instructions and click Ok.

mp2

Log Insight Agent Installation

From the drop-down menu in the top right again, select Administration. Under Management open the Agents page.

agents1

Select Download Log Insight Agent. Select the Windows MSI and copy the file to the Windows servers in the vRA management stack, in this case the IaaS Web and Manager servers, DEM, and Agent servers.

  • Right click and Install the file.
  • Accept the license terms and click Next.
  • Confirm the Log Insight server is auto-populated and click Next.
  • The install will now run. Click Finish once complete.

Go back to the Agents page in the Log Insight web interface. You should see the servers start populating the detected agents table. Next we’ll configure the built-in Log Insight agent on the vRA appliances. If you need to re-install or upgrade the built-in agent see this KB.

Open an SSH connection to the vRA appliance using an SSH client such as Putty. If SSH is not enabled on the appliance you can configure this by browsing to https://yourappliance:5480 and enabling SSH under the Admin tab.

Browse to the correct location cd /etc

View the liagent configuration file more liagent.ini

Edit the file, press the insert key to start typing vi liagent.ini

Remove the semi-colon that is commenting out the hostname, protocol, and port lines. Enter the hostname of the Log Insight instance, leave the protocol and ports with the default settings. To save and exit the file use :wq

Restart the Log Insight service service liagentd restart

edit

Log Insight Configuration

Log back in to the Log Insight web UI. Now that the agents are all installed or configured you should see the corresponding servers populating the agents table under Administration, Management, Agents.

agents

Click the drop-down arrow next to All Agents. Locate the vRealize Automation 7 – Windows template and click Copy Template. In the filter field add the Windows servers running vRA components with the Log Insight agent installed. Scroll down to the Agent Configuration and review the build settings if you want to configure specific event collections or if you need to change the default install path for vRA. Click Save New Group.

agentconfig

Locate the vRealize Automation 7 – Linux template and click Copy Template. Repeat the process this time adding the vRA appliances to the configuration. Once complete you should have templates configured to monitor all servers in the vRA management stack.

templates

Go back to Dashboards. Under VMware – vRA 7 you will start to see events being collected by Log Insight.

dashboard

NSX with Log Insight Integration

This post covers the steps required to configure NSX with Log Insight integration. The versions used are NSX 6.2.5 and Log Insight 4.0, for assistance with getting these products up and running see the NSX Install Guide and vRealize Log Insight Install Guide posts. Log Insight is available to NSX customers entitled to use v6.2.4 and above, at no extra cost. The Log Insight for NSX license allows for the collection of vSphere and NSX log data.

The first step is to install the NSX Content pack on the Log Insight instance, then we’ll configure NSX Manager, the NSX Controllers, and any NSX Edges to use Log Insight as a syslog server.

NSX Content Pack

Browse to the IP address or FQDN of the Log Insight appliance and log in as admin.

loginsight

Click the menu option in the top right hand corner of the page.

admin

If you need to configure vSphere integration click Administration and vSphere under the Integration menu on the left hand navigation pane. Enter the connection details of the vCenter Server. To configure only specific hosts to send logs to Log Insight click Advanced options. Test the connection and when you’re ready click Save.

vsphereint

To install the NSX Content Pack select Content Packs from the menu option in the right hand corner of the page. Under Marketplace locate the VMware NSX-vSphere Content Pack.

contentpacks

Select the content pack, accept the license agreement and click Install.

contentpacksinstall

The next message informs you to setup vSphere Integration, which we covered above, and log forwarding for the NSX Manager, Controllers, and Edge components, which we’ll cover next. Click Ok.

contentpacksinstall2

The NSX Content Pack gives us additional dashboards accessible by clicking the drop down menu next to General on the Dashboards page. We won’t see any data there yet, as we need to configure the NSX components to use syslog.

nsxcontent

NSX Manager

Browse to the IP address or FQDN of the NSX Manager and login as admin.

nsxmanager

Click Manage Appliance Settings.

log1

From the General tab locate Syslog Server and click Edit.

log2

Enter the syslog server name or IP address and use port 514 protocol UDP. Click Ok to save the settings.

log3

NSX Controllers

Configuration of a syslog server for NSX Controllers is done through an API call. For the initial configuration a REST client is required. In this example we’ll use Postman for Google Chrome. Download the Postman app from the Chrome Web Store. When you first open the app click skip to use without creating an account. On the Authorisation tab set the authorisation type to Basic Auth. Enter the admin username and password of the NSX Manager.

log7

Click the Headers tab, in the key field type Content-Type, in the value field type application/xml. (The Authorization key in the screenshot automatically generates after configuring authorisation).

headers

To view the configured syslog server of an NSX Controller enter the URL https://NSX/api/2.0/vdn/controller/controller-1/syslog, replacing NSX with the NSX Manager name, you can also update the controller if required (i.e. controller-2, controller-3, and so on). Ensure Get is selected and click Send, the output will list the syslog configuration and is displayed in the Response field.

log7

To configure the syslog server change Get to Post in the drop down menu. Then click the Body tab and select raw. Enter the following text, replacing LOG with the correct syslog server.

<controllerSyslogServer>
<syslogServer>LOG</syslogServer>
<port>514</port>
<protocol>UDP</protocol>
<level>INFO</level>
</controllerSyslogServer>

Click Send. The new syslog server will be set. Change the controller-1 section of the URL to controller-2 and click Send to configure the same syslog server for controller-2, and again for controller-3. It is important that each NSX Controller is configured with the IP address of the Log Insight server. You can change Post to Get to view the syslog server configuration again once complete.

NSX Edges

NSX Edge Service Gateways and Distributed Logical Routers can be configured for syslog in the vSphere web client. From the home page click Networking & Security, select NSX Edges.

log4

Double click the ESG or DLR and open the Manage tab, Settings, Configuration. In the Details pane next to Syslog servers click Change.

log5

Enter the syslog server name or IP, ensure the protocol is UDP and click Ok.

log6

The syslog configuration is now complete, after a few minutes you should see events start to appear in the Log Insight dashboards.

loginsightnsx

vRealize Log Insight 4.x Install Guide

vRealize Log Insight is a powerful log management and analytics tool, natively integrating with VMware products such as vRealize Automation, vRealize Operations, and vSphere, as well as providing a heterogeneous platform for third party products. By collecting logs at operating system, virtual machine, host, and vCenter level, as well as for third party products, Log Insight is able to compile dashboards, and perform data analysis to help administrators troubleshoot quickly and effectively. To read more see the product page here. In this post we will install a new Log Insight appliance, additional appliances can also be added to scale out the solution.

charts2

If you are using vRA and/or NSX see also the NSX with Log Insight Integration and vRealize Automation with Log Insight Integration guides.

Requirements

  • vRealize Log Insight can be licensed in packs of operating system instances, per CPU, or as part of vRealize and vCloud suites. A 60 day free trial can be obtained here.
  • The licensing editions of vRealize Log Insight can be found on the product page here. Advanced features are included with NSX, vRealize suites, and vCloud suites.
  • Version 4.0, 4.3, and 4.5 of the Log Insight appliance can be deployed to vCenter Server and ESXi versions 5.5 – 6.5. Only versions 4.3 and 4.5 are compatible with vSphere 6.5 U1.
  • For other VMware products check the Product Interoperability Matrixes here.
  • Access over the following ports is required for syslog: 514 (TCP/UDP), 1514 (TCP SSL), and the following ports for API: 9000 (TCP), 9543 (TCP SSL).
  • The virtual appliance comes pre-configured, when sizing the installation consider the following:
    • Extra small – 2 vCPU, 4 GB RAM, 132 GB disk (thick provisioned), vm hardware 7. Test or proof of concept, supports up to 20 ESXi hosts, 200 events per second, or 3 GB a day.
    • Small – 4 vCPU, 8 GB RAM, 510 GB disk (thick provisioned), vm hardware 7. Small production workloads, supports up to 200 ESXi hosts, 2000 events per second, or 30 GB a day.
    • Medium – 8 vCPU, 16 GB RAM, 510 GB disk (thick provisioned), vm hardware 7. Medium production workloads or Log Insight clusters, up to 500 ESXi hosts, 5000 events per second, or 75 GB a day.
    • Large – 16 vCPU, 32 GB RAM, 510 GB disk (thick provisioned), must be upgraded to vm hardware 8. Large production workloads or Log Insight clusters, supports up to 1500 ESXi hosts, 15000 events per second, or 225 GB a day.
  • Review the vRealize Log Insight Release Notes: v4.0 | v4.3 | v4.5
  • Download vRealize Log Insight: v4.0 | v4.3 | v4.5
  • For more information visit the vRealize Log Insight Information Center: v4.0 | v4.3 | v4.5

Installation

Download the required version of the VMware vRealize Log Insight virtual appliance. Log into the vSphere web client and right click the host or cluster where the appliance will be deployed, select Deploy OVF Template. Browse to the location of the downloaded OVA file and click Next. Review the template details and click Next.

log1

Accept the license agreement and click Next.

log2

Configure a name and location for the virtual appliance, click Next.

log3

Select the appropriate deployment configuration and click Next. See above for sizing assistance.

log4

Ideally the disk format should be changed to Thick Provisioned Eager Zeroed. Select the datastore to use and click Next. Select the network to use and click Next.

log5

Enter the network settings for the virtual appliance. Expand Other properties and configure a root password. Once complete click Next. When adding DNS servers do not specify more than 2 DNS entries.

log6

Review the summary page, tick Power on after deployment, and click Finish. The appliance console has a similar look and feel to ESXi. If you ever need to use the command line login with the root account. The password should be set during the OVA deployment, if you missed it then the root password is blank.

console

Open a web browser and connect to the IP address or FQDN of the newly deployed appliance. The setup wizard will autostart, click Next.

setup1

Click Start New Deployment.

setup2

setup3

Enter an email address and new password for the admin user, click Next.

setup4

Enter a license key and click Save and Continue.

setup5

Configure system notification settings and click Save and Continue.

setup6

Enter the NTP server(s) to use and click Test. If the test succeeds click Save and Continue.

setup7

Configure the SMTP server to use and click Save and Continue.

setup8

On the setup complete page click Finish.

setup9

The vRealize Log Insight appliance is now deployed and can begin collecting data. In this example we will be configuring vSphere Integration to automatically collect logs and events from vCenter Server and ESXi hosts. Click Configure vSphere Integration.

setup10

Enter the connection details of the vCenter Server. To configure only specific hosts to send logs to Log Insight click Advanced options. Test the connection and when you’re ready click Save.

vsphereint

Other administrative menus are located on the left hand side. The administration page can be accessed at any time by clicking the three line menu in the top right hand corner of the page.

admin

You can also access the Content Pack Marketplace from this menu. Content packs can be added to collect data from other VMware and third party products.

contentpacks

To add a content pack select it and click Install.

contentpacksinstall

For example to collect NSX logs and events we can install the NSX content pack.

contentpacksinstall2

With our Log Insight collecting data we can now flick through the various dashboards and available data. For more information on getting the most out of vRealize Log Insight, and a comprehensive user guide, see the Information Center: v4.0 | v4.3 | v4.5.

charts1