Site Recovery Manager Configuration and Failover Guide

This post will walk through the configuration of Site Recovery Manager; we’ll protect some virtual machines with a Protection Group, and then fail over to the DR site using a Recovery Plan. The pre-requisites for this post are for Site Recovery Manager (SRM) and the Storage Replication Adapter (SRA) to be installed at both sites along with the corresponding vSphere infrastructure, and replication to be configured on the storage array. It is also possible to use vSphere Replication, for more information see the previous posts referenced below.

Part 1 – Nimble Storage Integration with SRM

Part 2 – Site Recovery Manager Install Guide

Part 3 – Site Recovery Manager Configuration and Failover Guide

Before creating a Recovery Plan ensure that you have read the documentation listed in the installation guide above and have the required components for each site. You should also make further design considerations around compute, storage, and network. In this post we will be using storage based replication and stretched VLANs to ensure resources are available at both sites. If you want to assign a different VLAN at the failover site then you can use SRM to reconfigure the network settings, see this section of the documentation center.

SRM

Configuring SRM

Log into the vSphere web client for the primary site as an administrator, and click the Site Recovery Manager icon.

config

The first step is to pair the sites together. When sites are paired either site can be configured as the protected site.

  • Click Sites, both installed sites should be listed, select the primary site.
  • On the Summary tab, in the Guide to configuring SRM box, click 1. Pair sites.
  • The Pair Site Recovery Manager Servers wizard will open. Enter the IP address or FQDN of the Platform Services Controller for the recovery site, and click Next.
  • The wizard then checks the referenced PSC for a registered SRM install. Select the corresponding vCenter Server from the list and enter SSO administrator credentials.
  • Click Finish to pair the sites together.

Now the sites are paired they should both show connected. When we configure protection one will be made the protected site and the other failover.

config3

Next we will configure mappings to determine which resources, folders, and networks will be used at both sites.

  • Locate the Guide to configuring SRM box and the subheading 2. Configure inventory mappings.
  • Click 2.1 Create resource mappings.
  • Expand the vCenter servers and select the resources, then click Add mappings and Next.
  • On the next page you can choose to add reverse mappings too, using the tick box if required.
  • Click Finish to add the resource mappings.

config4

  • Click 2.2 Create folder mappings.
  • Select whether you want the system to automatically create matching folders in the failover site for storing virtual machines, or if you want to manually choose which folders at the protected site map to which folders at the failover site. Click Next.
  • Select the folders to map for both sites, including reverse mappings if required, and click Finish.

config5

  • Click 2.3 Create network mappings.
  • Select whether you want the system to automatically create networks, or if you want to manually choose which networks at the protected site map to which networks at the failover site. Click Next.
  • Select the networks to map for both sites and click Next.
  • Review the test networks, these are isolated networks used for SRM test failovers. It is best to leave these as the default settings unless you have a specific isolated test network you want to use. Click Next.
  • Include any reverse mappings if required, then click Finish.

Next we will configure a placeholder datastore. SRM creates placeholder virtual machines at the DR site, when a failover is initiated the placeholder virtual machines are replaced with the live VMs. A small datastore is required at each site for the placeholder data, placeholder VMs are generally a couple of KBs in size.

  • Click 3. Configure placeholder datastore.
  • Select the datastore to be used for placeholder information and click Ok.

The screenshot below shows the placeholder VMs in the failover site on the left, and the live VMs in the protected site on the right.

placeholder

Although we followed the wizard on the site summary page for the above tasks, it is also possible to configure, or change the settings later, by selecting the site and then the Manage tab, all the different mappings are listed.

mappings

Site Protection

The following steps will configure site protection, we’ll start by adding the storage arrays.

  • Click 4. Add array manager and enable array pair.
  • Select whether to use a single array manager, or add a pair of arrays, depending on your environment, and click Next. I’m adding two separate arrays.

array1

  • Select the site pairing and click Next.
  • Select the installed Storage Replication Adapter and click Next.

array2

  • Enter the details for the two storage arrays where volumes are replicated and click Next.
  • Select the array pair to enable and click Next.
  • Confirm the details on the review page and click Finish.

An array pair can be managed by selecting the SRM site and clicking the Related Objects tab, then Array Based Replication. If you add new datastores to the datastore group, you can check they have appeared by selecting Array Based Replication from the Site Recovery Manager home page, select the array, and click the Manage tab. Array pairs and replicated datastores will be listed, click the blue sync icon to discover new devices.

Now the storage arrays are added we can create a Protection Group.

  • Click 5. Create a Protection Group.
  • Enter a name for the protection group and select the site pairing, click Next.

protection1

  • Select the direction of protection and the type of protection group. In this example I am using datastore groups provided by array based replication so I’ll need to select the array-pair configured above, and Next.

protection2

  • Select the datastore groups to protect, the datastores and virtual machines will be listed, click Next.
  • Review the configuration and click Finish.

The final step is to group our settings together in a Recovery Plan.

  • Click 6. Create a Recovery Plan.
  • Enter a name for the recovery plan and select the site pairing, click Next.
  • From the sites detected select the recovery site and click Next.
  • Select the Protection Group we created above and click Next.
  • Review the test networks, these are isolated networks used for SRM test failovers. It is best to leave these as the default settings unless you have a specific isolated test network you want to use. Click Next.
  • Review the configuration and click Finish.

Now we have green ticks against each item in the Guide to configuring SRM box, we can move on to testing site failover. The array based replication, Protection Groups, and Recovery Plans settings can all be changed, or new ones created, using the menus on the left handside of the Site Recovery Manager home page.

complete.PNG

Site Failover

SRM allows us to do a test failover, as well as an actual failover in the event of a planned or unplanned site outage. The test failover brings online the replicated volumes and starts up the virtual machines, using VMware Tools to confirm the OS is responding. It does not connect the network or impact the production VMs.

  • Log in to the vSphere web client for the vCenter Server located at the DR site.
  • Click Site Recovery, click Recovery Plans and select the appropriate recovery plan.
    • To test the failover plan click the green start button (Test Recovery Plan).
    • Once the test has completed click the cleanup icon (Cleanup Recovery Plan) to remove the test data, previous results can still be viewed under History.
  • To initiate an actual fail over click the white start button inside a red circle (Run Recovery Plan).
  • Select the tick-box to confirm you understand the virtual machines will be moved to different infrastructure.
  • Select the recovery type; if the primary site is available then use Planned migration, datastores will be synced before fail over. If the primary site is unavailable then use Disaster recovery, datastores will be brought online using the most recent replica on the storage array.
  • Click Next and then Finish.

failover

During the failover you will see the various tasks taking place in vSphere. Once complete the placeholder virtual machines in the DR site are replaced with the live virtual machines. The virtual machines are brought online in the priority specified when we created the Recovery Plan.

failover1

Ensure the virtual machines are protected again as soon as the primary site is available by following the re-protection steps below.

Site Re-Protection

When the primary site is available the virtual machines must be re-protected to allow failback. Likewise after failing back to the primary site the virtual machines must be re-protected to allow failover again to the DR site.

  • Log in to the vSphere web client for either site and click Site Recovery, Recovery Plans and select the appropriate Recovery Plan.
  • Under Monitor, Recovery Steps, the Plan status needs to show Recovery complete, before we can re-protect.

reprotect1

If the status shows incomplete then you can troubleshoot which virtual machine(s) are causing the problem under Related Objects, Virtual Machines. VMware Tools must be running on the VMs to detect the full recovery process.

  • To re-protect virtual machines click Reprotect from the Actions menu at the top of the page.
  • Click the tick-box to confirm you understand the machines will be protected based on the sites specified.

reprotect2

  • Click Next and Finish. The re-protect job will now run, follow the status in the Monitor tab.

reprotect3

Once complete the Plan Status, and Recovery Status, will show Complete. The virtual machine Protection Status will show Ok. The VMs are now protected and can be failed over to the recovery site. If you are failing back to the primary site follow the same steps as outlined in the SRM Failover section above. Remember to then re-protect the VMs so they can failover to the DR site again in the event of an outage. When a Protection Plan is active the status will show Ready, the plan is ready for test or recovery.

reprotect4

_______________

Part 1 – Nimble Storage Integration with SRM

Part 2 – Site Recovery Manager Install Guide

Part 3 – Site Recovery Manager Configuration and Failover Guide

Site Recovery Manager 6.x Install Guide

This post will walk through the installation of Site Recovery Manager (SRM) to protect virtual machines from site failure. SRM plugs into vCenter to protect virtual machines replicated to a failover site using array based replication or vSphere replication. In the event of a site outage, or outage of components within a site meaning production virtual machines can no longer run there; SRM brings online the replicated datastore and VMs in vSphere, with a whole bunch of automated customisation options such as assigning new IP addresses, boot orders, dependencies, running scripts, etc. After a failover SRM can reverse the replication direction and protect virtual machines ready to fail back, all from within the vSphere web client.

Requirements

  • SRM is installed on a Windows machine at the protected site and the recovery site. SRM requires an absolute minimum of 2vCPU, 2 GB RAM and 5 GB disk available, more is recommended for large environments and installations with an embedded database.
  • The Windows server should have User Access Control (UAC) disabled (in the registry, not just set to never notify) as this interferes with the install.
  • Each SRM installation requires its own database, this can be embedded for small deployments, or external for large deployments.
  • A vCenter Server must be in place at both the protected site and the recovery site.
  • SRM supports both embedded and external Platform Services Controller deployments. If the external deployment method is used ensure the vCenter at the failover site is able to connect to the Platform Services Controller (i.e. it isn’t in the primary site). For more information click here.
  • The vCenter Server, Platform Services Controller, and SRM versions must be the same on both sites.
  • You will need the credentials of the vCenter Server SSO administrator for both sites.
  • For vCenter Server 6.0 U2 compatibility use SRM v6.1.1, vCenter Server 6.0 U3 use SRM v6.1.2 and for vCenter Server 6.5 and 6.5 U1 use v6.5 or v6.5.1 of SRM.
  • Check compatibility of other VMware products using the Product Interoperability Matrix.
  • If there any firewalls between the management components review the ports required for SRM in this KB.
  • SRM can be licensed in packs of 25 virtual machines, or for unlimited virtual machines on a per CPU basis with vCloud Suite. Read more about SRM licensing here.
  • Array based replication or vSphere Replication should be in place before beginning the SRM install. If you are using array based replication contact your storage vendor for best practices guide and the Storage Replication Adapter which is installed on the same server as SRM.

As well as the requirements listed above the following points are best practices which should also be taken into consideration:

  • Small environments can host the SRM installation on the same server as vCenter Server, for large environments SRM should be installed on a different system.
  • For vCenter Server, Platform Services Controller, Site Recovery Manager servers, and vSphere Replication (if applicable) use FQDN where possible rather than IP addresses.
  • Time synchronization should be in place across all management nodes and ESXi hosts.
  • It is best practice to have Active Directory and DNS servers already running at the failover site.

Installation

In this example we will be installing Site Recovery Manager using Nimble array based replication. There is a vCenter Server with embedded Platform Services Controller already installed at each site. The initial screenshots are from an SRM v6.1.1 install, but I have also validated the process with SRM v6.5.1 and vCenter 6.5 U1.

SRM

The virtual machines we want to protect are in datastores replicated by the Nimble array. For more information on the storage array pre-installation steps see the Nimble Storage Integration post referenced below. The Site Recovery Manager install, configuration, and failover guides have no further references to Nimble and are the same for all vendors and replication types.

Part 1 – Nimble Storage Integration with SRM

Part 2 – Site Recovery Manager Install Guide

Part 3 – Site Recovery Manager Configuration and Failover Guide

Installing SRM

The installation is pretty straight forward, download the SRM installer and follow the steps below for each site. We’ll install SRM on the Windows server for the primary / protected site first, and repeat the process for the DR / failover site. We can then pair the two sites together and create recovery plans.

SRM 6.5.1 (vSphere 6.5 U1) Download | Release Notes | Documentation

SRM 6.5 (vSphere 6.5) Download | Release Notes | Documentation

SRM 6.1.2 (vSphere 6.0 U3) Download | Release Notes | Documentation

SRM 6.1.1 (vSphere 6.0 U2) Download | Release Notes | Documentation

Log into the Windows server where SRM will be installed as an administrator, and right click the downloaded VMware-srm-version.exe file. Select Run as aministrator. If you are planning on using an external database then the ODBC data source must be configured, for SQL integrated Windows authentication make sure you log into the Windows server using the account that has database permissions to configure the ODBC data source, and run the SRM installer.

Select the installer language and click Ok.

SRM1

Click Next to begin the install wizard.

SRM2

Review the patent information and click Next.

SRM3

Accept the EULA and click Next.

SRM4

Confirm you have read the prerequisites located at http://pubs.vmware.com/srm-61/index.jsp by clicking Next.

SRM5

Select the destination drive and folder, then click Next.

SRM6

Enter the IP address or FQDN of the Platform Services Controller that will be registered with this SRM instance, in this case the primary site. If possible use the FQDN to make IP address changes easier if required at a later date. Enter valid credentials to connect to the PSC and click Next. If your vCenter Server is using an embedded deployment model then enter your vCenter Server information.

SRM7

Accept the PSC certificate when prompted. The vCenter Server will be detected from the PSC information provided. Confirm this is correct and click Next. Accept the vCenter certificate when prompted.

SRM8

Enter the site name that will appear in the Site Recovery Manager interface, and the SRM administrator email address. Enter the IP address or FQDN of the local server, again use the FQDN if possible, and click Next.

SRM11

In this case as we are using a single protected site and recovery site we will use the Default Site Recovery Manager Plug-in Identifier. For environments with multiple protected sites create a custom identifier. Click Next.

SRM12

Select Automatically generate a certificate, or upload one of your own if required, and click Next.

SRM13

Select an embedded or external database server and click Next. If you are using an external database you will need a DSN entry configured in ODBC data sources on the local Windows server referencing the external data source. Click Next.

SRM14

If you opted for the embedded database you will be prompted to enter a new database name and create new database credentials. Click Next.

SRM15

Configure the account to run the SRM services, if applicable, and click Next.

SRM10

Click Install to begin the installation.

SRM9

Site Recovery Manager is now installed. Repeat the process to install SRM on the Windows server in the DR / recovery site, referencing the local PSC and changing the site names as appropriate. If you are using storage based replication you also need to install the Storage Replication Adapter (SRA) on the same server as Site Recovery Manager. In this example I have installed the Nimble SRA, available from InfoSight downloads, which is just a next and finish installer.

After each site installation of SRM you will see the Site Recovery Manager icon appear in the vSphere web client for the corresponding vCenter Server.

SRMvsphereSRMvsphere2

Providing the datastores are replicated, either using vSphere replication or array based replication, we can now move on to pairing the sites and creating recovery plans in Part 3.

_______________

Part 1 – Nimble Storage Integration with SRM

Part 2 – Site Recovery Manager Install Guide

Part 3 – Site Recovery Manager Configuration and Failover Guide

NSX with Log Insight Integration

This post covers the steps required to configure NSX with Log Insight integration. The versions used are NSX 6.2.5 and Log Insight 4.0, for assistance with getting these products up and running see the NSX Install Guide and vRealize Log Insight Install Guide posts. Log Insight is available to NSX customers entitled to use v6.2.4 and above, at no extra cost. The Log Insight for NSX license allows for the collection of vSphere and NSX log data.

The first step is to install the NSX Content pack on the Log Insight instance, then we’ll configure NSX Manager, the NSX Controllers, and any NSX Edges to use Log Insight as a syslog server.

NSX Content Pack

Browse to the IP address or FQDN of the Log Insight appliance and log in as admin.

loginsight

Click the menu option in the top right hand corner of the page.

admin

If you need to configure vSphere integration click Administration and vSphere under the Integration menu on the left hand navigation pane. Enter the connection details of the vCenter Server. To configure only specific hosts to send logs to Log Insight click Advanced options. Test the connection and when you’re ready click Save.

vsphereint

To install the NSX Content Pack select Content Packs from the menu option in the right hand corner of the page. Under Marketplace locate the VMware NSX-vSphere Content Pack.

contentpacks

Select the content pack, accept the license agreement and click Install.

contentpacksinstall

The next message informs you to setup vSphere Integration, which we covered above, and log forwarding for the NSX Manager, Controllers, and Edge components, which we’ll cover next. Click Ok.

contentpacksinstall2

The NSX Content Pack gives us additional dashboards accessible by clicking the drop down menu next to General on the Dashboards page. We won’t see any data there yet, as we need to configure the NSX components to use syslog.

nsxcontent

NSX Manager

Browse to the IP address or FQDN of the NSX Manager and login as admin.

nsxmanager

Click Manage Appliance Settings.

log1

From the General tab locate Syslog Server and click Edit.

log2

Enter the syslog server name or IP address and use port 514 protocol UDP. Click Ok to save the settings.

log3

NSX Controllers

Configuration of a syslog server for NSX Controllers is done through an API call. For the initial configuration a REST client is required. In this example we’ll use Postman for Google Chrome. Download the Postman app from the Chrome Web Store. When you first open the app click skip to use without creating an account. On the Authorisation tab set the authorisation type to Basic Auth. Enter the admin username and password of the NSX Manager.

log7

Click the Headers tab, in the key field type Content-Type, in the value field type application/xml. (The Authorization key in the screenshot automatically generates after configuring authorisation).

headers

To view the configured syslog server of an NSX Controller enter the URL https://NSX/api/2.0/vdn/controller/controller-1/syslog, replacing NSX with the NSX Manager name, you can also update the controller if required (i.e. controller-2, controller-3, and so on). Ensure Get is selected and click Send, the output will list the syslog configuration and is displayed in the Response field.

log7

To configure the syslog server change Get to Post in the drop down menu. Then click the Body tab and select raw. Enter the following text, replacing LOG with the correct syslog server.

<controllerSyslogServer>
<syslogServer>LOG</syslogServer>
<port>514</port>
<protocol>UDP</protocol>
<level>INFO</level>
</controllerSyslogServer>

Click Send. The new syslog server will be set. Change the controller-1 section of the URL to controller-2 and click Send to configure the same syslog server for controller-2, and again for controller-3. It is important that each NSX Controller is configured with the IP address of the Log Insight server. You can change Post to Get to view the syslog server configuration again once complete.

NSX Edges

NSX Edge Service Gateways and Distributed Logical Routers can be configured for syslog in the vSphere web client. From the home page click Networking & Security, select NSX Edges.

log4

Double click the ESG or DLR and open the Manage tab, Settings, Configuration. In the Details pane next to Syslog servers click Change.

log5

Enter the syslog server name or IP, ensure the protocol is UDP and click Ok.

log6

The syslog configuration is now complete, after a few minutes you should see events start to appear in the Log Insight dashboards.

loginsightnsx

Reconfiguring vCenter Server for External PSC

An external Platform Services Controller (PSC) can provide scalability and high availability across sites. A vCenter Server initially deployed with an embedded PSC can be reconfigured to use an external PSC by following the steps outlined below. Multiple external Platform Services Controllers can be deployed and an environment can be mixed between the appliance and Windows versions of vCenter Server and PSC.

externalpsc

Considerations

  • The vCenter Server must be running at least version 6.0 Update 1.
  • The process involves the installation of an external PSC as a new target for vCenter Server. The PSC must be in the same Single Sign-On site and domain as the vCenter Server.
  • Ensure you have good backups of your vCenter Server. If the vCenter Server is virtual take a snapshot before starting the process, likewise after deploying the new PSC take a snapshot.
  • If the process fails for any reason revert back to the snapshots.
  • An external PSC deployment model cannot be converted into an embedded PSC.
  • If vCenter HA is enabled then disable and reconfigure after the process is complete. For more information see Configuring vCenter 6.5 High Availability.
  • The commands outlined below are the same for the vCenter Server Appliance and Windows vCenter Server, unless specified. Take into account the following environmental variables:
    • For Windows all commands should be run as an administrator in an elevated command prompt.
    • For the appliance use the root account for all commands, enable BASH and launch the shell by running shell.set -enabled True followed by shell.

Process

The first step is to determine the Single Sign-On site by running the following commands on the vCenter Server: vCenter Server Appliance: /usr/lib/vmware-vmafd/bin/vmafd-cli get-site-name --server-name localhost. Windows vCenter Server: "C:\Program Files\VMware\vCenter Server\vmafdd\vmafd-cli" get-site-name --server-name localhost.

Make a note of the SSO site. Next deploy the new external Platform Services Controller, if you require assistance with this see the Deploying an External Platform Services Controller post. The new PSC must be configured with the same Single Sign-On site and domain as the vCenter Server you want to reconfigure.

joindomain

joinsite

Once the external PSC is up and running go back to the vCenter Server. Confirm the Platform Services Controller services are running, for Windows first navigate to the correct directory by using:

cd "C:\Program Files\VMware\vCenter Server\bin".

For both the appliance and Windows versions run the following command:

service-control --status --all

Check that the VMware License Service, VMware Identity Management Service, VMware Security Token Service, VMware Certificate Service, and VMware Directory Services are running.

cmd

To reconfigure the vCenter Server to use the new PSC use the following command, replacing newpsc with the IP or FQDN (case sensitive) of the new PSC, username, domainname, and password with the relevant SSO domain and user details.

cmsso-util reconfigure --repoint-psc newpsc --username username --domain-name domainname --passwd password

If the external PSC is configured to use a custom port then add [--dc-port port] where port is the port number. Check the configuration results.

results

Confirm the vCenter is accessible by logging in to the vSphere web client. The process is complete, if you disabled vCenter HA then you can now go ahead and reconfigure.

Deploying an External Platform Services Controller

This post will walk through the process of deploying an external Platform Services Controller (PSC) appliance. The PSC was introduced with vSphere 6.0 to deal with infrastructure services such as Single Sign-On, Certificate Authority, and licensing.  For more information on the Platform Services Controller review this KB.

The PSC can be either embedded within the vCenter Server, or external to allow scale out for larger environments. When deciding if an embedded or external PSC is appropriate review the vCenter Server deployment models here. The external PSC can be installed as a virtual appliance, or installed on a Windows server (virtual or physical). Environments can be mixed, for example a PSC virtual appliance can be deployed where a physical Windows vCenter currently exists. You may also want to review the following posts:

Installation Process

Downloaded the VMware vCenter Server Appliance here: v6.0, v6.5.

Mount the ISO on your computer. The VCSA 6.5 installer is compatible with Mac, Linux, and Windows. Browse to the corresponding directory for your operating system, e.g. \vcsa-ui-installer\win32. Right click Installer and select Run as administrator. As we are installing a new instance click Install.

psc1

On the welcome page click Next. Accept the license agreement and click Next.

psc3

For the deployment type we need to select Platform Services Controller under the External Platform Services Controller heading. Click Next.

psc4

Enter details of the vCenter or ESXi host where the appliance will be deployed, click Next.

psc5

Select a location for the virtual appliance and click Next.

psc6

Select the compute resource for the virtual appliance and click Next.

psc7

Enter a name for the virtual appliance and configure the root password, click Next.

psc8

Select the storage to use and click Next.

psc9

Select the VM network to use and configure the network settings, click Next.

psc10

Review the deploy Platform Services Controller summary page and click Finish. The Platform Services Controller appliance will now be deployed.

stage2

In stage 2 we configure the new appliance, click Next.

config

Configure the NTP server(s) and click Next.

config1

The SSO configuration page is where we determine if the PSC should be joined to an existing SSO domain or if you are creating a new SSO domain. Enter the SSO domain details and click Next.

config2

Tick or untick the Customer Experience Improvement Program and click Next.

config3

On the summary page click Finish and Ok. The PSC virtual appliance will now be configured.

config4

Once complete we can access the Platform Services Controller in 2 different ways. For the appliance management portal browse to https://IP:5480 where IP is the IP or FQDN of the virtual appliance. Login with the root account.

root

Here we can configure settings specific to the virtual appliance, such as networking, SSH, syslog, etc.

root2

To access the user interface browse to https://IP/psc where IP is the IP or FQDN of the virtual appliance. Login with the administrator@vsphere.local account created or defined in the installation wizard.

psc

Here we can configure Platform Services Controller related settings, such as permissions, certificates, etc. To join the PSC to an Active Directory domain browse to Appliance Settings, and Manage. Under Active Directory click Join.

domain

The Platform Services Controller has now been deployed and configured. Multiple PSC instances can be placed behind a load balancer to provide High Availability, as outlined in this KB.

vRealize Log Insight 4.x Install Guide

vRealize Log Insight is a powerful log management and analytics tool, natively integrating with VMware products such as vRealize Automation, vRealize Operations, and vSphere, as well as providing a heterogeneous platform for third party products. By collecting logs at operating system, virtual machine, host, and vCenter level, as well as for third party products, Log Insight is able to compile dashboards, and perform data analysis to help administrators troubleshoot quickly and effectively. To read more see the product page here. In this post we will install a new Log Insight appliance, additional appliances can also be added to scale out the solution.

charts2

If you are using vRA and/or NSX see also the NSX with Log Insight Integration and vRealize Automation with Log Insight Integration guides.

Requirements

  • vRealize Log Insight can be licensed in packs of operating system instances, per CPU, or as part of vRealize and vCloud suites. A 60 day free trial can be obtained here.
  • The licensing editions of vRealize Log Insight can be found on the product page here. Advanced features are included with NSX, vRealize suites, and vCloud suites.
  • Version 4.0, 4.3, and 4.5 of the Log Insight appliance can be deployed to vCenter Server and ESXi versions 5.5 – 6.5. Only versions 4.3 and 4.5 are compatible with vSphere 6.5 U1.
  • For other VMware products check the Product Interoperability Matrixes here.
  • Access over the following ports is required for syslog: 514 (TCP/UDP), 1514 (TCP SSL), and the following ports for API: 9000 (TCP), 9543 (TCP SSL).
  • The virtual appliance comes pre-configured, when sizing the installation consider the following:
    • Extra small – 2 vCPU, 4 GB RAM, 132 GB disk (thick provisioned), vm hardware 7. Test or proof of concept, supports up to 20 ESXi hosts, 200 events per second, or 3 GB a day.
    • Small – 4 vCPU, 8 GB RAM, 510 GB disk (thick provisioned), vm hardware 7. Small production workloads, supports up to 200 ESXi hosts, 2000 events per second, or 30 GB a day.
    • Medium – 8 vCPU, 16 GB RAM, 510 GB disk (thick provisioned), vm hardware 7. Medium production workloads or Log Insight clusters, up to 500 ESXi hosts, 5000 events per second, or 75 GB a day.
    • Large – 16 vCPU, 32 GB RAM, 510 GB disk (thick provisioned), must be upgraded to vm hardware 8. Large production workloads or Log Insight clusters, supports up to 1500 ESXi hosts, 15000 events per second, or 225 GB a day.
  • Review the vRealize Log Insight Release Notes: v4.0 | v4.3 | v4.5
  • Download vRealize Log Insight: v4.0 | v4.3 | v4.5
  • For more information visit the vRealize Log Insight Information Center: v4.0 | v4.3 | v4.5

Installation

Download the required version of the VMware vRealize Log Insight virtual appliance. Log into the vSphere web client and right click the host or cluster where the appliance will be deployed, select Deploy OVF Template. Browse to the location of the downloaded OVA file and click Next. Review the template details and click Next.

log1

Accept the license agreement and click Next.

log2

Configure a name and location for the virtual appliance, click Next.

log3

Select the appropriate deployment configuration and click Next. See above for sizing assistance.

log4

Ideally the disk format should be changed to Thick Provisioned Eager Zeroed. Select the datastore to use and click Next. Select the network to use and click Next.

log5

Enter the network settings for the virtual appliance. Expand Other properties and configure a root password. Once complete click Next. When adding DNS servers do not specify more than 2 DNS entries.

log6

Review the summary page, tick Power on after deployment, and click Finish. The appliance console has a similar look and feel to ESXi. If you ever need to use the command line login with the root account. The password should be set during the OVA deployment, if you missed it then the root password is blank.

console

Open a web browser and connect to the IP address or FQDN of the newly deployed appliance. The setup wizard will autostart, click Next.

setup1

Click Start New Deployment.

setup2

setup3

Enter an email address and new password for the admin user, click Next.

setup4

Enter a license key and click Save and Continue.

setup5

Configure system notification settings and click Save and Continue.

setup6

Enter the NTP server(s) to use and click Test. If the test succeeds click Save and Continue.

setup7

Configure the SMTP server to use and click Save and Continue.

setup8

On the setup complete page click Finish.

setup9

The vRealize Log Insight appliance is now deployed and can begin collecting data. In this example we will be configuring vSphere Integration to automatically collect logs and events from vCenter Server and ESXi hosts. Click Configure vSphere Integration.

setup10

Enter the connection details of the vCenter Server. To configure only specific hosts to send logs to Log Insight click Advanced options. Test the connection and when you’re ready click Save.

vsphereint

Other administrative menus are located on the left hand side. The administration page can be accessed at any time by clicking the three line menu in the top right hand corner of the page.

admin

You can also access the Content Pack Marketplace from this menu. Content packs can be added to collect data from other VMware and third party products.

contentpacks

To add a content pack select it and click Install.

contentpacksinstall

For example to collect NSX logs and events we can install the NSX content pack.

contentpacksinstall2

With our Log Insight collecting data we can now flick through the various dashboards and available data. For more information on getting the most out of vRealize Log Insight, and a comprehensive user guide, see the Information Center: v4.0 | v4.3 | v4.5.

charts1

vSphere Management Assistant Guide

The vSphere Management Assistant (vMA) can be used to remotely manage and troubleshoot multiple hosts from the command line. The vSphere Management Assistant is a SUSE Linux Enterprise based virtual appliance deployed within your vSphere infrastructure, it allows centralised management and troubleshooting of multiple ESXi hosts with automatic login, and scripting tools for developers. The vMA appliance includes the vSphere Command Line Interface (vCLI), vSphere SDK for Perl, and components for logging and authentication. The vCLI can also be installed separately on a machine of your choice running Windows or Linux. The standalone vCLI installation allows administrators to run all the commands available within the vMA, if you’re interested in installing vCLI standalone v6.5 can be downloaded here as a simple executable install. Review the release notes here for system requirements.

This post will cover the installation and configuration of vSphere Management Assistant 6.5; compatible with vSphere 5.0 and above. For managing individual hosts, locally or remotely, the ESXi Shell can be used, see the Troubleshooting with ESXi Shell post.

Installing vMA

vSphere Management Assistant v6.5 can be downloaded here, review the release notes here. Unzip the contents of the download and make a note of the file location.

In order to deploy the virtual appliance we need an available Network Protocol Profile. In the vSphere web client browse to the datacentre level where the appliance will reside, select the Manage tab and click Network Protocol Profiles. Click the green plus symbol to create a new profile, follow the wizard and assign the relevant network and settings to the profile.

networkprofile

The vSphere Management Assistant is a simple OVF deployment.

  • In the vSphere web client right click the host or cluster where the virtual appliance will reside. Click Deploy OVF Template.
  • Browse to the downloaded OVF file which was extracted from the .zip download and click Next.
  • Review the details of the appliance and click Next.
  • Accept the license terms and click Next.
  • Enter a name and location for the virtual appliance, click Next.
  • Select the storage to be used and click Next.
  • Select the network to use for the virtual machine and choose the IP allocation (DHCP or static). If static is selected enter the DNS servers, gateway and subnet mask. An additional page prompts for the IP address. Click Next.
  • On the summary page tick Power on after deployment and click Finish.

ovf1

If no Network Protocol profile is present and associated to the network in use then the virtual appliance is unable to power on, you will receive the error Cannot initialize propery ‘vami.netmask0.vSphere_Management_Assistant_(vMA)’. Network ‘VM Network’ has no associated protocol profile. In this case you should ensure the profile has been created and correctly configured.

Once the appliance is powered on open the console. Enter 0 to check the configuration, use the relevant numbers to configure the default gateway, hostname, DNS, and IP address allocation. Once complete enter 1 to exit the setup program.

vma

You will be prompted to change the default password for the vi-admin account, enter the old password vmware and a new password. Once loaded you can connect to the vSphere Management Assistant using an SSH client such as Putty. You can manage the virtual appliance by browsing to https://:5480 where is the IP address or FQDN of the appliance.

Configuring vMA

Open an SSH connection to the IP address or FQDN of the vSphere Management Assistant. Login as the vi-admin user and the password you changed during setup.

The vMA allows administrators to store credentials for automatic authentication when managing ESXi hosts. Using a component called vi-fastpass two accounts are created and the passwords stored in an unreadable format; vi-admin (administrator account) and vi-user (read only). These accounts prevent the user from having to log in to each host and facilitate unattended scheduled script operations.

Alternatively vMA can be configured to use Active Directory for authentication, providing more security controls. To use AD authentication the domain must be accessible from the vMA and DNS must be in place. The following commands are useful for AD tasks in vMA:

  • Join vMA to the domain: sudo domainjoin-cli join domain user where domain is the domain to join and user is the domain user with appropriate privileges.
  • Check the domain status: sudo domainjoin-cli query.
  • Remove vMA from the domain: sudo domainjoin-cli leave.

We can add ESXi hosts or vCenter Servers to vMA using the following commands:

  • To add a system to vMA using the default fastpass authentication: vifp addserver server -authpolicy fpauth -username user -password password where server is the ESXi host or vCenter Server to add, and user and password are the credentials to authenticate with.
  • To add a system to vMA using AD authentication: vifp addserver server –authpolicy adauth –username domain\\user where server is the FQDN of the server and domain\\user is the domain and user to authenticate with.
  • To list the systems added to vMA: vifp listservers.

With the systems authenticated and added to vMA we can now set a target system for executing vCLI commands or vSphere SDK for Perl scripts.

  • Use vifptarget -s server where server is the IP address or FQDN of the vCenter Server or ESXi host. The target system is shown in the command prompt.
  • You can add multiple targets and execute commands across multiple ESXi hosts using the bulkAddServers and mcli scripts, explained in this post by William Lam.

Using vMA

The same commands available to the ESXi shell, such as esxcli, esxcfg, esxtop (resxtop since we are connecting remotely), can be used with vCLI. Furthermore the vCLI includes a subset of vmware-cmd and vicfg commands. You can use more and less commands to assist with truncating information. For example esxcli –help | more and esxcli –help | less. More allows for scrolling down only, use enter to scroll one line at a time and space to scroll a page at a time. Less allows for scrolling both backwards (ctrl + b) and forward (ctrl +f), use q to return back to the command line. The following VMware documentation will get you started with the command line interface.

Let’s take a look at some of the most popular commands. The vmware-cmd command can be used for virtual machine operations, vicfg is primarily used for host operations and is intended to replace esxcfg long term. The main set of commands for managing the vSphere environment you will see is esxcli. The command set is broken down into namespaces, to view the available namespaces just enter esxcli.

namespaces

This propogates down the chain, for example use esxcli storage to view the options within the storage namespace. You can use –help at any level of esxcli for assistance.

storagenamespaces

You can view a full list of esxcli commands by entering esxcli esxcli command list. The screenshot below has been cropped and isn’t a full list, it may be beneficial to drill down through the relevant individual sections using the method outlined above.

list

As you can see the range of esxcli commands is vast, let’s take a look at a few examples.

  • esxcli hardware allows us to view and change the physical server hardware information and configuration. Use esxcli hardware cpu global set to enable or disable hyperthreading.

hardware

  • esxcli system allows us to view and change the ESXi system configuration. To enable or disable maintenance mode use esxcli system maintenanceMode set.

maintenance-mode

  • esxcli storage can be used for storage related tasks, use esxcli storage core path list to view attached LUNs, or esxcli storage vmfs upgrade to upgrade VMFS.

vmfs.PNG

  • esxcli network allows us to perform network related tasks, use esxcli network vswitch standard to create a new standard virtual switch.

switch

For details on patching or upgrading ESXi from the command line see the ESXi Command Line Upgrades post. I also found this great blog post by Chanaka Ekanayake who has put together some of the most useful commands and examples for use with vMA and vCLI.