Tag Archives: VCSA

Updating vCenter Server with External PSC

The following post demonstrates the update process for applying minor updates to a vSphere environment running multiple vCenter Server appliances and external Platform Services Controllers.

In this instance we are updating vCenter to 6.5 U1e as one of the remediation actions for the Branch Target Injection issue (CVE-2017-5715) commonly known as Spectre. For more information on Meltdown and Spectre see this blog post, VMwares responses can be found here, on the VMware Security & Compliance Blog here, as well as VMware Security Announcement VMSA-2018-0004.2 here.

meltdown-spectre-vmware

Pre-Update Checks

When upgrading vSphere with an external Platform Services Controller (PSC), upgrade the PSC first, then the vCenter Server, then the ESXi hosts, and finally the virtual machines (hardware versions, VMware Tools).

Prior to updating vCenter ensure you have verified the compatibility of any third party products such as backups, anti-virus, monitoring, etc. Also cross-check the compatibility of other VMware products using the Product Interoperability Matrix. Since we are applying a minor update to vCenter Server the usual pre-requisites such as FQDN resolution, time synchronization, relevant ports open, etc. should already be in place. For vCenter 6.5 U1e all hosts must be running at least ESXi version 5.5. For more information on the requirements for vCenter Server 6.5, or if you are upgrading from an earlier version, the following posts may be of use:

Before beginning the update process take a backup and snapshot of the vCenter Server Appliance. There is downtime during the update but this is minimal – around 10 mins to update and reboot using an ISO as an update source, when using the online repository the update time may vary depending on your internet connection.

Review the version release notes and the VMware Docs site here.

VAMI Update

Platform Services Controller (PSC) appliances that are replicating should all be updated before the vCenter Server appliances. The easiest way of updating the vCenter Servers and Platform Services Controllers is through the VAMI (vCenter Server Appliance Management Interface). Browse to https://PSC:5480, where PSC is the FQDN or IP address of the external Platform Services Controller. Log in as the root user.

VAMI1

Select the Update option from the navigator.

vcupgrade2

Click the Check Updates drop-down. If the VCSA has internet access then select Check Repository to pull the update direct from the VMware online repository.

If the VCSA does not have internet access, or you’d prefer to provide the update manually then download the relevant update from VMware here (in this case VMware-vCenter-Server-Appliance-6.5.0.14000-7515524-patch-FP.iso) and attach the ISO to the CD/DVD drive of the VCSA in the virtual machine settings. Back in the VAMI update page select the Check Updates drop-down and click Check CDROM.

VAMI3

Details of the available update from either the online repository or attached ISO are displayed. Click Install Updates. Accept the EULA and click Install to begin the installation.

vcupgrade3

When the update process has completed click OK. From an attached ISO the update took around 5 minutes. The updated version and release date should now be displayed in the current version details. Finally, to complete the upgrade reboot the vCenter Server Appliance. Select Summary from the navigator and click Reboot.

vcupgrade4

If you are running multiple external PSCs then repeat the above process for each PSC in the SSO domain. Do not update the vCenter Server appliances until all PSC appliances are running the same updated version.

Once all external PSC appliances that replicate between one another have been upgraded then move on to the vCenter Server appliances. Repeat the above process for each vCenter Server in the SSO domain.

CLI Update

Alternatively the vCenter Server Appliance can be updated from the command line. Again, either using the online repository or by downloading the update from VMware here (VMware-vCenter-Server-Appliance-6.5.0.10000-5973321-patch-FP.iso or latest version) and attaching the ISO to the CD/DVD drive of the VCSA in the virtual machine settings. For more information on updating the vCenter Server Appliance using the appliance shell see this section of VMware docs.

Platform Services Controller (PSC) appliances that are replicating should all be updated before the vCenter Server appliances. Log in to the external Platform Services Controller appliance as root. First stage the patches from your chosen source using either:

  • software-packages stage --iso --acceptEulas stages software packages from ISO and accepts EULA.
  •  software-packages stage --url --acceptEulas stages software packages from the default VMware online repository and accepts EULA.

Next, review the staged packages, install the update, and reboot the VCSA.

  • software-packages list --staged lists the details of the staged software package.
  • software-packages install --staged installs the staged software package.
  • shutdown reboot -r update reboots the VCSA where ‘update’ is the reboot reason. Use -d to add a delay.

CLI4

If you are running multiple external PSCs then repeat the above process for each PSC in the SSO domain. Do not update the vCenter Server appliances until all PSC appliances are running the same updated version.

Once all external PSC appliances that replicate between one another have been upgraded then move on to the vCenter Server appliances. Repeat the above process for each vCenter Server in the SSO domain.

Installing vCenter Internal CA signed SSL Certificates

This post will walk through the process of replacing the default self-signed certificates in vCenter with SSL certificates signed by your own internal Certificate Authority (CA). In previous versions of vSphere the certificate replacement procedure was so complex that many administrators ignored it completely. Now with the certificate tool improvements in vSphere 6.x, and the ever increasing security threat of todays digital world, applying SSL certificates takes on an enhanced significance for verifying servers, solutions, and users are who they say they are.

The procedure outlined below is specific to installing Microsoft intermediate CA signed certificates on VCSA 6.5 with embedded PSC, protecting us against man in the middle attacks with a secure connection which we can see in the screenshot below. From v6.0 onwards the VMware Certificate Authority (VMCA) was also introduced, for more information on using the VMCA see this blog post, or to read how to use the VMCA as an intermediate CA see here. VMware documentation for replacing self-signed certificates can be reviewed from this KB article.

Trusted_vSphere

Before beginning the replacement certificate process ensure you have a good backup, and snapshot of the VCSA. The following links are the official VMware guides and this blog post provides a good overview of the certificates we’re actually going to be replacing. Replacing default certificates with CA signed SSL certificates in vSphere 6.x (2111219)Replacing a vSphere 6.x Machine SSL certificate with a Custom Certificate Authority Signed Certificate (2112277)How to replace the vSphere 6.x Solution User certs with CA signed certs (2112278).

Migrating Windows vCenter Server to VCSA 6.7

468x60

Generate CSR

The first thing we need to do is generate a Certificate Signing Request (CSR). Open an SSH connection to the VCSA using an SSH client such as Putty, and login as root – if you need to enable SSH you can do so from the VAMI (https://vCenterIPorFQDN:5480) under Access; enable both SSH Login and Bash Shell. Run the following command to open the VMware built in Certificate Manager tool:

/usr/lib/vmware-vmca/bin/certificate-manager

Cert_Tool_1

Select the appropriate option. In this case we first want to replace the machine SSL certificate with a custom certificate, option 1. When prompted enter the SSO administrator username and password. Enter 1 again to generate certificate signing request(s) and Key(s) for machine SSL certificate, and enter the output directory. In the example below we are using the /tmp directory. Fill in the required values for the certool.cfg file.

Cert_Tool_2

The CSR and key are generated in the location specified. Change the shell to /bin/bash using chsh -s "/bin/bash" root and open an SCP connection to the VCSA using WinSCP. Copy the vmca_issued_csr.csr file to your local machine, you can use Notepad to view the contents of the file. Leave the WinSCP session open as we’ll need it to copy the certificate chain back to the VCSA.

Request Certificate

The next step is to use the CSR to request a certificate from your internal Certificate Authority (official KB here). A Microsoft CA template needs creating with the settings specified here (official KB here) before requesting the certs. Once this is done open a web browser to the Microsoft Certificate Services page (normally https://CAServer/certsrv) and select Request a Certificate.

Internal_CA_1

Then we want to Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file. The next page allows us to enter the CSR generated earlier to request a certificate with the pre-configured vSphere 6.5 certificate template.

Internal_CA_2

Click Submit and then select Base 64 encoded and Download certificate and Download certificate chain. A .cer file will be downloaded, I have renamed this machine_name_ssl.cer, and a .p7b. Double click the .p7b file to open in certmgr, locate and right click the root certificate, select All Tasks, Export. Export the root certificate in Base-64 encoded X.509 (.CER) format, in this example I have named the file Root64.cer. Using WinSCP copy the machine and root certificate files to the VCSA.

Install Certificate

Go back to Certificate Manager and enter 1 to continue to importing custom certificate(s) and key(s) for machine SSL certificate. Enter the file for the machine SSL certificate we copied, I have used /tmp/machine_name_ssl.cer. Enter the associated custom key that was generated with the CSR request, in this case /tmp/vmca_issued_key.key. Finally, enter the signing certificate of the machine SSL certificate, in this case /tmp/Root64.cer. When prompted enter y to replace the default machine SSL certificate with the custom certificate.

Cert_Tool_3

The certificate will now be installed, when finished a success message will be displayed. If certificate installation fails at 0% see this KB article.

Cert_Tool_4

To verify the machine certificate open a web browser to the vCenter FQDN, the connection will now show secure. Depending on the browser used you can view the certificate properties to verify it is correct, alternatively browse to https://vCenterFQDN/psc and log in with an SSO administrator account. Open Certificate Management and Machine Certificates, select the installed machine certificate and click Show Details, verify the certificate properties are correct.

Certificate_Management

Solution User Certificates

Repeat the steps above for the solution user certificates (official KB here). Replacing the solution user certificates may break some external plugins, such as SRM, in which case you should review this KB article for corrective action. To recap: /usr/lib/vmware-vmca/bin/certificate-manager. This time select option 5 replace solution user certificates with custom certificates. Generate the CSRs and keys, you will notice that for the solution user certs 4 CSR and key files are created; machine, vsphere-webclient, vpxd, and vpxd-extension.

Cert_Tool_5

Using WinSCP copy the files to your local machine and repeat the certificate request process from the Microsoft Certificate Services page. Copy the new certificates to the VCSA and repeat the install process. Solution User certificates can be viewed on the PSC web interface under Certificate Management, Solution User Certificates.

Solution_User_Management

vCenter Server Appliance Integrated TFTP Server

This post covers the steps required to use the vCenter Server Appliance for Auto Deploy, with the built in TFTP server in vSphere 6.5. For more information on Auto Deploy, and to see the process for creating ESXi images and deploy rules to boot hosts, see the VMware Auto Deploy 6.x Guide. This post assumes that you have a working vCenter Server Appliance, and may be of use if you have recently migrated from Windows vCenter to VCSA.

Enable Auto Deploy

Open the vSphere web client and click System Configuration, Nodes. Select the vCenter Server and open the Related Objects tab. The Auto Deploy, ImageBuilder Service, and VMware vSphere ESXi Dump Collector services should all be set to Automatic and Running.

To start a service right click and select Start, then select Edit Startup Type and choose Automatic.

servicesLog out of the web client and log back in. You should now see the Auto Deploy icon on the home page.

autodeploy1

Enable TFTP

Now that Auto Deploy is enabled we can configure the TFTP server. Enable SSH on the VCSA by browsing to the Appliance Management page: https://VCSA:5480 where VCSA is the IP or FQDN of your appliance.

Log in as the root account. From the Access page enable SSH Login and Bash Shell.

SSH

SSH onto the vCenter Appliance, using a client such as Putty, and log in with the root account. First type shell and hit enter to launch Bash.

To start the TFTP service enter service atftpd start. Check the service is started using service atftpd status.

startservice

To allow TFTP traffic through the firewall on port 69; we must run iptables -A port_filter -p udp -m udp –dport 69 -j ACCEPT. Validate traffic is being accepted over port 69 using iptables -nL | grep 69.

firewall

The TFTP server will now work, however we need to make a couple of additional changes to make the configuration persistent after the VCSA is rebooted. There isn’t an official VMware way of doing this, and as it’s done in Linux there may be more than one way of achieving what we want. Basically I am going to backup iptables and create a script to restore iptables and start the TFTP service when the appliance boots. The steps are outlined below and this worked for me, however as a reminder this is not supported by VMware, and if you are a Linux expert you’ll probably find a better way round it.

The following commands are all run in Bash on the vCenter Server Appliance, you can stay in the existing session we were using above.

First make a copy of the existing iptables config by running iptables-save > /etc/iptables.rules.

Next change the directory by running cd /etc/init.d, and create a new script: vi scriptname.sh, for example: vi starttftp.sh.

Press I to begin typing. I used the following, which was copied from the Image Builder Service startup script, and modified for TFTP.

#! /bin/sh
#
# TFTP Start/Stop the TFTP service and allow port 69
#
# chkconfig: 345 80 05
# description: atftpd

### BEGIN INIT INFO
# Provides: atftpd
# Required-Start: $local_fs $remote_fs $network
# Required-Stop:
# Default-Start: 3 5
# Default-Stop: 0 1 2 6
# Description: TFTP
### END INIT INFO

service atftpd start
iptables-restore -c < /etc/iptables.rules

The file must be in the above format to be compatible with chkconfig which runs the script at startup. I left the defaults in from the Image Builder Service as it made sense they started at the same time and had the same dependencies. If you wish to modify further see the following sources: Bash commands, Script Options, Startup, and vmwarebits.com for the iptables commands.

Press escape to leave the editor and :wq to save the file and quit.

Next set execute permissions on the script by running chmod +x scriptname.sh, for example: chmod +x starttftp.sh.

To set the script to run at startup use chkconfig –add scriptname.sh, for example: chkconfig –add starttftp.sh.

Reboot the vCenter appliance to test the script is running. If successful the atftpd service will be started and port 69 allowed, you can check these with service atftpd status and iptables -nL | grep 69.

Close the session and disable SSH if required.

Configure DHCP

In this example I will be using PXE boot to boot the ESXi hosts using a DHCP reservation. On the DHCP scope that will be serving the hosts I have configured reservations and enabled options 066 and 067. In the value for option 066 (Boot Server Host Name) goes the IP address or FQDN of the vCenter Server where TFTP is running. In the value for option 067 (Bootfile Name) I have entered the BIOS DHCP File Name (undionly.kpxe.vmw-hardwired).

DHCP

Now that Auto Deploy is up and running using the built-in components of VCSA 6.5 you can begin creating ESXi images and deploy rules to boot hosts; using the Auto Deploy GUI. See the VMware Auto Deploy 6.x Guide.