Tag: vCenter

vCenter

Updating vCenter Server with External PSC

by esxsiLeave a Comment on Updating vCenter Server with External PSC

VMware vSphere 6.0 reached end of general support 12 March 2020, with vSphere 6.5 scheduled for 15 October 2022, both referenced in the VMware Lifecycle Matrix. See also How to Install vSphere 7.0. Upgrade to vSphere 7 can be achieved directly from vSphere 6.5.0 and above, whereas vSphere 6.0 requires an intermediate upgrade to 6.5 or 6.7 first. For more information see the VMware Upgrade Matrix. Finally, the Windows vCenter Server and external PSC deployment models are now depreciated and not available with vSphere 7.0.

The following post demonstrates the update process for applying minor updates to a vSphere environment running multiple vCenter Server appliances and external Platform Services Controllers.

In this instance we are updating vCenter to 6.5 U1e as one of the remediation actions for the Branch Target Injection issue (CVE-2017-5715) commonly known as Spectre. For more information on Meltdown and Spectre see this blog post, VMwares responses can be found here, on the VMware Security & Compliance Blog here, as well as VMware Security Announcement VMSA-2018-0004.2 here.

meltdown-spectre-vmware

Pre-Update Checks

When upgrading vSphere with an external Platform Services Controller (PSC), upgrade the PSC first, then the vCenter Server, then the ESXi hosts, and finally the virtual machines (hardware versions, VMware Tools).

Prior to updating vCenter ensure you have verified the compatibility of any third party products such as backups, anti-virus, monitoring, etc. Also cross-check the compatibility of other VMware products using the Product Interoperability Matrix. Since we are applying a minor update to vCenter Server the usual pre-requisites such as FQDN resolution, time synchronization, relevant ports open, etc. should already be in place. For vCenter 6.5 U1e all hosts must be running at least ESXi version 5.5. For more information on the requirements for vCenter Server 6.5, or if you are upgrading from an earlier version, the following posts may be of use:

Before beginning the update process take a backup and snapshot of the vCenter Server Appliance. There is downtime during the update but this is minimal – around 10 mins to update and reboot using an ISO as an update source, when using the online repository the update time may vary depending on your internet connection.

Review the version release notes and the VMware Docs site here.

VAMI Update

Platform Services Controller (PSC) appliances that are replicating should all be updated before the vCenter Server appliances. The easiest way of updating the vCenter Servers and Platform Services Controllers is through the VAMI (vCenter Server Appliance Management Interface). Browse to https://PSC:5480, where PSC is the FQDN or IP address of the external Platform Services Controller. Log in as the root user.

VAMI1

Select the Update option from the navigator.

vcupgrade2

Click the Check Updates drop-down. If the VCSA has internet access then select Check Repository to pull the update direct from the VMware online repository.

If the VCSA does not have internet access, or you’d prefer to provide the update manually then download the relevant update from VMware here (in this case VMware-vCenter-Server-Appliance-6.5.0.14000-7515524-patch-FP.iso) and attach the ISO to the CD/DVD drive of the VCSA in the virtual machine settings. Back in the VAMI update page select the Check Updates drop-down and click Check CDROM.

VAMI3

Details of the available update from either the online repository or attached ISO are displayed. Click Install Updates. Accept the EULA and click Install to begin the installation.

vcupgrade3

When the update process has completed click OK. From an attached ISO the update took around 5 minutes. The updated version and release date should now be displayed in the current version details. Finally, to complete the upgrade reboot the vCenter Server Appliance. Select Summary from the navigator and click Reboot.

vcupgrade4

If you are running multiple external PSCs then repeat the above process for each PSC in the SSO domain. Do not update the vCenter Server appliances until all PSC appliances are running the same updated version.

Once all external PSC appliances that replicate between one another have been upgraded then move on to the vCenter Server appliances. Repeat the above process for each vCenter Server in the SSO domain.

CLI Update

Alternatively the vCenter Server Appliance can be updated from the command line. Again, either using the online repository or by downloading the update from VMware here (VMware-vCenter-Server-Appliance-6.5.0.10000-5973321-patch-FP.iso or latest version) and attaching the ISO to the CD/DVD drive of the VCSA in the virtual machine settings. For more information on updating the vCenter Server Appliance using the appliance shell see this section of VMware docs.

Platform Services Controller (PSC) appliances that are replicating should all be updated before the vCenter Server appliances. Log in to the external Platform Services Controller appliance as root. First stage the patches from your chosen source using either:

  • software-packages stage --iso --acceptEulas stages software packages from ISO and accepts EULA.
  •  software-packages stage --url --acceptEulas stages software packages from the default VMware online repository and accepts EULA.

Next, review the staged packages, install the update, and reboot the VCSA.

  • software-packages list --staged lists the details of the staged software package.
  • software-packages install --staged installs the staged software package.
  • shutdown reboot -r update reboots the VCSA where ‘update’ is the reboot reason. Use -d to add a delay.
CLI4

If you are running multiple external PSCs then repeat the above process for each PSC in the SSO domain. Do not update the vCenter Server appliances until all PSC appliances are running the same updated version.

Once all external PSC appliances that replicate between one another have been upgraded then move on to the vCenter Server appliances. Repeat the above process for each vCenter Server in the SSO domain.

NSX

Configuring VMware Cross-vCenter NSX

by esxsiLeave a Comment on Configuring VMware Cross-vCenter NSX

This post provides an overview of cross-vCenter NSX and walks through the configuration steps. Cross-vCenter NSX allows central management of network virtualization and security policies across multiple vCenter Server systems. Cross vCenter NSX introduces universal objects; such as universal logical switches, universal logical routers, and universal distributed firewall rules. Universal objects are able to span multiple sites or vCenter Server instances, enhancing workload mobility by allowing cross vCenter and long distance vMotion for virtual machines, whilst keeping the same network settings and firewall rules. This improves DR capabilities, overcomes scale limits of vCenter Server, and gives administrators more control over resource pooling and the separation of environments.

Cross vCenter-NSX was introduced in NSX v6.2 and requires vSphere v6.0 or later. As normal NSX Manager is deployed with vCenter server in a 1:1 pairing.  In a single site NSX deployment the NSX Manager is given the standalone role by default. When configuring cross-vCenter NSX one NSX Manager is assigned the primary role, and up to seven other NSX Managers are assigned the secondary role. NSX Managers configured for cross-vCenter NSX must all be running the same version. The primary NSX Manager is responsible for deploying the Universal Controller Cluster; forming the control plane across the NSX Managers. The Universal Controller Cluster runs in the site of the primary NSX Manager. Universal objects are created on the primary NSX Manager and automatically synchronized across the multi-site NSX environment.

Configuring Cross-vCenter NSX

The steps below assume you have already deployed and registered the NSX Managers, and have a good understanding of NSX. This post is intended as add on to the NSX Install Guide to provide an outline of the additional or different steps required for a cross-vCenter NSX install, further resources are listed at the bottom of the page. If you are using vCenter enhanced linked mode then multiple NSX Manager instances are displayed within the same interface, or single pane of glass, when managing the Network & Security section of the vSphere web client. Enhanced linked mode is not a requirement for cross-vCenter NSX however, and vCenter Server systems not in enhanced linked mode can still be configured for cross-vCenter NSX.

From the Networking & Security page of the vSphere web client select Installation, highlight the NSX Manager in the primary site, from the Actions menu select Assign Primary Role.

NSX_Promote

The secondary NSX Manager(s) synchronize with the primary using the Universal Synchronization Service. These sites do not run any NSX Controllers, although they can be redeployed easily in the event of a primary site outage. Before assigning the secondary role you should ensure there are no existing NSX Controllers deployed in the associated vCenter. If you have already assigned a segment ID pool to the NSX Managers then ensure the segment ID pools do not overlap. Select the primary NSX Manager and from the Actions menu click Add Secondary NSX Manager. Enter the secondary NSX Manager information and admin password.

NSX2

Review the table of NSX Managers, the roles have now changed accordingly.

NSX_Roles

The universal controller cluster is formed by individually deploying the NSX controllers from the primary NSX Manager, the method of deploying the controllers is the same (see NSX Install Guide Part 1 – Mgmt and Control Planes for further assistance). Once the controllers are deployed you will notice placeholder controllers listed against the secondary NSX Manager, these are not connected or deployed. In the event of a site failure the configuration is synchronized between NSX Managers so you can simply re-deploy the controllers in the DR site. To see the failover process review this blog post. VMware recommend deploying 3 controllers on different hosts with anti-affinity rules.

NSX_Controllers

The next part of the install process is to follow the host preparation and VXLAN configuration steps as normal (see NSX Install Guide Part 2 – Data Plane for further assistance). Create the segment ID pools for each NSX Manager, making sure they do not overlap. On the primary NSX Manager you will also assign a universal segment ID pool.

In order for us to deploy universal logical switches we need to create a universal transport zone. A universal transport zone determines which hosts a universal logical switch can reach, spanning multiple vCenters. From the Logical Network Preparation tab open Transport Zones, ensure the primary NSX Manager is selected and click the plus symbol. Select Mark this object for Universal Synchronization, and enter the configuration for the universal transport zone. All universal objects must be created on the primary NSX Manager, change the NSX Manager to the secondary site and you will see the universal transport zone has synchronized there also.

NSX_TZ

Next we will create a universal logical switch for the transit network. Local objects such as logical switches, logical routers, and Edge Services Gateways can still be deployed from each NSX Manager, although by design they are only local to the vCenter linked to that specific NSX Manager, and cannot be deployed or edited elsewhere. From the left hand navigation pane in Networking & Security select Logical Switches, ensure the primary NSX Manager is selected and click the plus symbol. Enter a name for the transit network and select the universal transport zone we created earlier.

NSX_Universal_Transit

At this stage you can also deploy another universal logical switch, connecting a couple of test VMs on a private subnet, and have them ping one another to confirm connectivity. Now that we have a transit network and test universal logical switches connected to our universal transport zone we can go ahead and create a universal DLR. In this particular environment we have already deployed an ESG in each site. For further assistance with deploying an ESG and DLR see NSX Install Guide Part 3 – Edge and DLR.

From the Networking & Security page click NSX Edges, ensure the primary NSX Manager is selected and click the plus symbol. The control VM for the DLR is deployed to the primary site, again the configuration is synchronized and this can be re-deployed to the DR site in the event of a primary site outage. Select Universal Logical Router and follow the wizard as normal, if local egress is required then check the appropriate box. Sites configured in a cross-vCenter NSX environment can use the same physical routers for egress traffic, or have the local egress feature enabled within a universal logical router. The local egress feature allows routes to be customized at host, cluster, or router level.

NSX_UDLR

From the NSX Edges page double click the new universal DLR, select Manage, Settings, Interfaces and click the add button. In order for traffic to route from the universal DLR to the ESG(s) we must add an uplink interface connecting them to the universal transit network. Change the logical router interface to Uplink, in the Connected To field select the transit network universal logical switch we created earlier. Configure the IP and MTU settings of the interface per your own environment.

NSX_UDLR_Interface

You can also add Internal interfaces here corresponding with universal logical switches for virtual machine subnets. Before these subnets can route out follow the same process to add an Internal interface to the ESG(s) connecting them to the same transit network.

A virtual machine connected to the test universal logical switch can now vMotion between sites keeping the same IP addressing, providing L2 over L3 capability. As well as remaining on the same logical network a virtual machine can also be migrated across sites without any additional firewall rules, this is achieved with the use of universal firewall rules. Universal firewall rules require a dedicated section creating under the Firewall section of Networking & Security, you must select Mark this section for Universal Synchronization. For assistance with creating universal firewall rules see here.

NSX_Universal_Firewall

Additional Resources

To plan a cross-vCenter NSX installation review the VMware Cross-vCenter NSX Design Guide, Cross-vCenter NSX Topologies Guide, and the VMware Cross-vCenter Installation Guide. For more information on cross-vCenter NSX design see the following blog posts:

vCenter

Installing vCenter Internal CA signed SSL Certificates

by esxsi5 Comments on Installing vCenter Internal CA signed SSL Certificates

This post will walk through the process of replacing the default self-signed certificates in vCenter with SSL certificates signed by your own internal Certificate Authority (CA). In previous versions of vSphere the certificate replacement procedure was so complex that many administrators ignored it completely. Now with the certificate tool improvements in vSphere 6.x, and the ever increasing security threat of todays digital world, applying SSL certificates takes on an enhanced significance for verifying servers, solutions, and users are who they say they are.

The procedure outlined below is specific to installing Microsoft intermediate CA signed certificates on VCSA 6.5 with embedded PSC, protecting us against man in the middle attacks with a secure connection which we can see in the screenshot below. From v6.0 onwards the VMware Certificate Authority (VMCA) was also introduced, for more information on using the VMCA see this blog post, or to read how to use the VMCA as an intermediate CA see here. VMware documentation for replacing self-signed certificates can be reviewed from this KB article.

Trusted_vSphere

Before beginning the replacement certificate process ensure you have a good backup, and snapshot of the VCSA. The following links are the official VMware guides and this blog post provides a good overview of the certificates we’re actually going to be replacing. Replacing default certificates with CA signed SSL certificates in vSphere 6.x (2111219), Replacing a vSphere 6.x Machine SSL certificate with a Custom Certificate Authority Signed Certificate (2112277), How to replace the vSphere 6.x Solution User certs with CA signed certs (2112278).

728x90

Generate CSR

The first thing we need to do is generate a Certificate Signing Request (CSR). Open an SSH connection to the VCSA using an SSH client such as Putty, and login as root – if you need to enable SSH you can do so from the VAMI (https://vCenterIPorFQDN:5480) under Access; enable both SSH Login and Bash Shell. Run the following command to open the VMware built in Certificate Manager tool:

/usr/lib/vmware-vmca/bin/certificate-manager

Cert_Tool_1

Select the appropriate option. In this case we first want to replace the machine SSL certificate with a custom certificate, option 1. When prompted enter the SSO administrator username and password. Enter 1 again to generate certificate signing request(s) and Key(s) for machine SSL certificate, and enter the output directory. In the example below we are using the /tmp directory. Fill in the required values for the certool.cfg file.

Cert_Tool_2

The CSR and key are generated in the location specified. Change the shell to /bin/bash using chsh -s "/bin/bash" root and open an SCP connection to the VCSA using WinSCP. Copy the vmca_issued_csr.csr file to your local machine, you can use Notepad to view the contents of the file. Leave the WinSCP session open as we’ll need it to copy the certificate chain back to the VCSA.

Request Certificate

The next step is to use the CSR to request a certificate from your internal Certificate Authority (official KB here). A Microsoft CA template needs creating with the settings specified here (official KB here) before requesting the certs. Once this is done open a web browser to the Microsoft Certificate Services page (normally https://CAServer/certsrv) and select Request a Certificate.

Internal_CA_1

Then we want to Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file. The next page allows us to enter the CSR generated earlier to request a certificate with the pre-configured vSphere 6.5 certificate template.

Internal_CA_2

Click Submit and then select Base 64 encoded and Download certificate and Download certificate chain. A .cer file will be downloaded, I have renamed this machine_name_ssl.cer, and a .p7b. Double click the .p7b file to open in certmgr, locate and right click the root certificate, select All Tasks, Export. Export the root certificate in Base-64 encoded X.509 (.CER) format, in this example I have named the file Root64.cer. Using WinSCP copy the machine and root certificate files to the VCSA.

Install Certificate

Go back to Certificate Manager and enter 1 to continue to importing custom certificate(s) and key(s) for machine SSL certificate. Enter the file for the machine SSL certificate we copied, I have used /tmp/machine_name_ssl.cer. Enter the associated custom key that was generated with the CSR request, in this case /tmp/vmca_issued_key.key. Finally, enter the signing certificate of the machine SSL certificate, in this case /tmp/Root64.cer. When prompted enter y to replace the default machine SSL certificate with the custom certificate.

Cert_Tool_3

The certificate will now be installed, when finished a success message will be displayed. If certificate installation fails at 0% see this KB article.

Cert_Tool_4

To verify the machine certificate open a web browser to the vCenter FQDN, the connection will now show secure. Depending on the browser used you can view the certificate properties to verify it is correct, alternatively browse to https://vCenterFQDN/psc and log in with an SSO administrator account. Open Certificate Management and Machine Certificates, select the installed machine certificate and click Show Details, verify the certificate properties are correct.

Certificate_Management

Solution User Certificates

Repeat the steps above for the solution user certificates (official KB here). Replacing the solution user certificates may break some external plugins, such as SRM, in which case you should review this KB article for corrective action. To recap: /usr/lib/vmware-vmca/bin/certificate-manager. This time select option 5 replace solution user certificates with custom certificates. Generate the CSRs and keys, you will notice that for the solution user certs 4 CSR and key files are created; machine, vsphere-webclient, vpxd, and vpxd-extension.

Cert_Tool_5

Using WinSCP copy the files to your local machine and repeat the certificate request process from the Microsoft Certificate Services page. Copy the new certificates to the VCSA and repeat the install process. Solution User certificates can be viewed on the PSC web interface under Certificate Management, Solution User Certificates.

Solution_User_Management