Migrating Windows vCenter Server to VCSA 6.7

VMware vCenter Server pools ESXi host resources to provide a rich feature set delivering high availability and fault tolerance to virtual machines. The vCenter Server is a centralised management application and can be deployed as a virtual appliance or Windows machine. It should be noted that vCenter 6.7 is the final release where Windows modules will be available, see here for more information. All future releases will only be available as vCenter Server Appliance (VCSA) which is the preferred deployment method of vCenter Server. This post gives a walk through on migrating from a Windows based vCenter Server (VCS) to the Photon OS based vCenter Server Appliance (VCSA).

vCenter 6.7: Download | Release Notes | What’s New | VMware DocsvSphere Central

About VCSA

migrate2vcsaThe VCSA is a pre-configured virtual appliance built on Project Photon OS. Since the OS has been developed by VMware it benefits from enhanced performance and boot times over the previous Linux based appliance. Furthermore the embedded vPostgres database means VMware have full control of the software stack, resulting in significant optimisation for vSphere environments and quicker release of security patches and bug fixes. The VCSA scales up to 2000 hosts and 35,000 virtual machines. A couple of releases ago the VCSA reached feature parity with its Windows counterpart, and is now the preferred deployment method for vCenter Server. Features such as Update Manager are bundled into the VCSA, as well as file based backup and restore, and vCenter High Availability. The appliance also saves operating system license costs and is quicker and easier to deploy and patch.

Migrating to VCSA involves the deployment of a new appliance and migration of all configuration (including distributed switches) and historical data using the upgrade installer. The VCSA uses a temporary IP address during migration before switching to the IP and host name of the VCS, the Windows box is then powered off.

Software Considerations

  • The Windows VCS must be v.6.0 or v6.5 (any build / patch) to migrate to VCSA 6.7. Both physical and virtual vCenter Server installations are compatible.
  • Any database, internal or external, supported by VCS can be migrated to the embedded vPostgres database within the target VCSA.
  • The ESXi host or vCenter where VCSA will be deployed must be running v5.5 or above. However, all hosts you intend to connect to vCenter Server 6.7 should be running ESXi 6.0 or above, hosts running 5.5 and earlier cannot be managed by vCenter 6.7 and do not have a direct upgrade path to 6.7.
  • The Windows server is powered off once the VCSA is brought online, this means any other components, VMware or third party, need to be migrated off the Windows server in advance or they will no longer work (don’t forget to move and update any scripts that may live on the Windows server).
  • If you are using Update Manager the VCSA now includes an embedded Update Manager instance.
  • You must check compatibility of any third party products and plugins that might be used for backups, anti-virus, monitoring, etc. as these may need upgrading for vSphere 6.7 compatibility.
  • To check version compatibility with other VMware products see the Product Interoperability Matrix.
  • The points above are especially important since at the time of writing vSphere 6.7 is new enough that other VMware and third party products may not have released compatible versions. Verify before installing vSphere 6.7 and review the Release Notes and Important information before upgrading to vSphere 6.7 KB.

Hardware Considerations

  • The VCSA with embedded PSC requires the following hardware resources (disk can be thin provisioned)
    • Tiny (up to 10 hosts, 100 VMs) – 2 CPUs, 10 GB RAM.
    • Small (up to 100 hosts, 1000 VMs) – 4 CPUs, 16 GB RAM.
    • Medium (up to 400 hosts, 4000 VMs) – 8 CPUs, 24 GB RAM.
    • Large (up to 1000 hosts, 10,000 VMs) – 16 CPUs, 32 GB RAM.
    • X-Large (up to 2000 hosts, 35,000 VMs) – 24 CPUs, 48 GB RAM – new to v6.5.
  • Storage requirements for the smallest environments start at 250 GB and increase depending on your specific database requirements. See the Storage Requirements document for further details.
  • Where the PSC is deployed as a separate appliance this requires 2 CPUs, 4 GB RAM, 60 GB disk.
  • Environments with ESXi host(s) with more than 512 LUNs and 2048 paths should be sized large or x-large.
  • To help with selecting the appropriate storage size for the appliance calculate the size of your existing VCS database here.

Architectural Considerations

  • The migration tool supports different deployment topologies but can not, make changes to the topology and SSO domain configuration.
  • For more information on the deployment topologies available with vCenter 6.x see vCenter Server and Platform Services Controller Deployment Types.
  • A series of videos covering vCenter Server and Platform Services Architecture can be found here. If you require further assistance with vCenter planning see also the vSphere Topology and Upgrade Planning Tool here,
  • Most deployments will include the vCenter Server and PSC in one appliance, following the embedded deployment model, which I will use in this guide.
  • Consider if the default self-signed certificates are sufficient or if you want to replace with custom CA or VMware CA signed certs, see Installing vCenter Internal CA signed SSL Certificates for more information.

embedded

Other Considerations

  • Ensure you have a good backup of the vCenter Server and the database.
  • Variables such as FQDN resolution, database permissions and access to the licensing portal should all be in place since we are upgrading an existing vCenter solution.
  • All vSphere components should be configured to use an NTP server. The installation can fail or the vCenter Server Appliance vpxd service may not be able to start if the clocks are unsynchronized.
  • The ESXi host on which you deploy the VCSA should not be in lockdown or maintenance mode.
  • You will need the SSO administrator login details and if the Windows VCS service runs as a service account then the account must have replace a process level token permission.
  • Local Windows users that have vSphere permissions are not migrated since they are specific to the Windows server, all SSO users and permissions are migrated.
  • The upgrade can be easily rolled back by following this KB.
  • Migration of vCenter using DHCP, or services with custom ports, is not supported. The settings of only one physical network adapter are migrated.
  • Downtime varies depending on the amount of data you are migrating and is calculated when running the migration wizard.
  • A list of Required Ports for vCenter Server and PSC can be found here.
  • The configuration maximums for vSphere 6.7 can be found here.
  • In vSphere 6.7 TLS 1.2 is enabled by default. TLS 1.0 and TLS 1.1 are disabled by default, review the Release Notes for more information.
  • There are a number of Intel and AMD CPUs no longer supported with vSphere 6.7, review the Release Notes for a full list of unsupported processors.

Process

Before we begin if your existing Windows vCenter is virtual it may be beneficial to rename the vCenter virtual machine name in the vSphere inventory to include -old or equivalent. While the hostname and IP are migrated the vSphere inventory name of the VM cannot be a duplicate. The old server is powered down but not deleted so that we have a back out.

Download the VMware vCenter Server Appliance 6.7 ISO from VMware downloads: v6.7.0. Mount the ISO on your computer. The VCSA 6.5 installer is compatible with Mac, Linux, and Windows. Copy the migration-assistant folder to the Windows vCenter Server (and PSC server if external). If the PSC is running on a different Windows server then you must run the Migration Assistant on the PSC server first and migrate following the instructions below, then complete the same process on the Windows vCenter Server.

Start the VMware-Migration-Assistant and enter the SSO Administrator credentials to start running pre-checks.

VCSA_Migration_1

If all checks complete successfully the Migration Assistant will finish at ‘waiting for migration to start’.

On a different machine from your Windows vCenter and PSC server(s) open the vcsa-ui-installer folder file located on the root of the ISO. Browse to the corresponding directory for your operating system, e.g. \vcsa-ui-installer\win32. Right click Installer and select Run as administrator. The vCenter Server Appliance Installer will open, click Migrate.

VCSA_Migration_2

The migration is split into 2 stages; stage 1 deploys the new appliance with temporary network settings, there is no outage to the Windows vCenter at this stage. Stage 2 migrates data and network settings over to the new appliance and shuts down the Windows server. We begin with deploying the appliance. Click Next.

VCSA_Migration_3

Accept the license terms and click Next.

VCSA_Migration_4

Enter the details of the vCenter Server to migrate, then click Next.

VCSA_Migration_5

Enter the FQDN or IP address of the host, or vCenter upon which you wish to deploy the new VCSA. Enter the credentials of an administrative or root user and click Next. The installer will validate access, if prompted with an untrusted SSL certificate message click Yes to continue. Tip – connect to the vCenter for visibility of any networks using a distributed switch, connecting to the host direct will only pull back networks using a standard switch.

VCSA_Migration_6

Enter the virtual appliance VM name, this is the name that appears in the vSphere inventory as mentioned earlier. The host name of the vCenter Server will automatically be migrated. Click Next.

VCSA_Migration_7

Select the appropriate deployment size for your environment and click Next.

VCSA_Migration_8

Select the datastore to locate the virtual appliance and click Next. Configure the temporary network settings for the appliance. These will only be used during migration of the data, once complete the temporary settings are discarded and the VCSA assumes the identity, including IP settings, of the Windows vCenter Server. Click Next.

VCSA_Migration_9

Review the settings on the summary page and click Finish. The VCSA will now be deployed. Once complete click Continue to being the second stage of the migration.

VCSA_Migration_10

Click Next to begin the migration wizard.

VCSA_Migration_11

The source vCenter details are imported from stage 1.

VCSA_Migration_12

As my source Windows vCenter was joined to a domain I am prompted for credentials to join the VCSA to the domain.

VCSA_Migration_13

Select the data to migrate and click Next.

VCSA_Migration_14

Select whether or not to join the VMware Customer Experience Improvement Program and click Next.

VCSA_Migration_15

Review the summary page and click Finish. Data will now be migrated to the VCSA, once complete the Windows vCenter Server will be powered off and the network settings transferred to the VCSA. If you urgently need to power back on the Windows server to retrieve files or such like, then do so with the vNICs disconnected, otherwise you will cause an IP/host name conflict on the network.

VCSA_Migration_16

Post-Installation

Connect to the vCenter post install using the IP or FQDN of the vCenter. Access vSphere by clicking either Launch vSphere Client (HTML5) or Launch vSphere Web Client (FLEX). As the web client will be depreciated in future versions, and the HTML5 client is now nearly at full feature parity, we will use the HTML5 vSphere client.

Windows_vCenter67_14

Management features of the VCSA can be accessed by browsing to the IP or FQDN of the vCenter on port 5480. The login is the root account we configured a password for during the migration wizard.

VCSA_Management

vCenter Server Appliance 6.7 Install Guide

VMware vCenter Server pools ESXi host resources to provide a rich feature set delivering high availability and fault tolerance to virtual machines. The vCenter Server is a centralised management application and can be deployed as a virtual appliance or Windows machine. It should be noted that vCenter 6.7 is the final release where Windows modules will be available, see here for more information. All future releases will only be available as vCenter Server Appliance (VCSA) which is the preferred deployment method of vCenter Server. An existing Windows vCenter can be migrated to VCSA by following the steps in Migrating Windows vCenter Server to VCSA 6.7 This post gives a walk through on a clean installation of VCSA 6.7.

vCenter 6.7: Download | Release Notes | What’s New | VMware DocsvSphere Central

About VCSA

migrate2vcsaThe VCSA is a pre-configured virtual appliance built on Project Photon OS. Since the OS has been developed by VMware it benefits from enhanced performance and boot times over the previous Linux based appliance. Furthermore the embedded vPostgres database means VMware have full control of the software stack, resulting in significant optimisation for vSphere environments and quicker release of security patches and bug fixes. The VCSA scales up to 2000 hosts and 35,000 virtual machines. A couple of releases ago the VCSA reached feature parity with its Windows counterpart, and is now the preferred deployment method for vCenter Server. Features such as Update Manager are bundled into the VCSA, as well as file based backup and restore, and vCenter High Availability. The appliance also saves operating system license costs and is quicker and easier to deploy and patch.

Software Considerations

  • VCSA 6.7 must be deployed to an ESXi host or vCenter running v5.5 or above. However, all hosts you intend to connect to vCenter Server 6.7 should be running ESXi 6.0 or above, hosts running 5.5 and earlier cannot be managed by vCenter 6.7 and do not have a direct upgrade path to 6.7.
  • You must check compatibility of any third party products and plugins that might be used for backups, anti-virus, monitoring, etc. as these may need upgrading for vSphere 6.7 compatibility.
  • To check version compatibility with other VMware products see the Product Interoperability Matrix.
  • The points above are especially important since at the time of writing vSphere 6.7 is new enough that other VMware and third party products may not have released compatible versions. Verify before installing vSphere 6.7 and review the Release Notes and Important information before upgrading to vSphere 6.7 KB.

Architectural Considerations

  • When implementing a new vSphere 6.7 environment you should plan your topology in accordance with the VMware vCenter Server and Platform Services Controller Deployment Types.
  • A series of videos covering vCenter Server and Platform Services Architecture can be found here. If you require further assistance with vCenter planning see also the vSphere Topology and Upgrade Planning Tool here,
  • Most deployments will include the vCenter Server and PSC in one appliance, following the embedded deployment model, which I will use in this guide.
  • Greenfield deployments of vSphere 6.7 can take advantage of Embedded PSC with Enhanced Linked Mode, providing native vCenter Server HA support, and removal of SSO site boundaries.
  • Consider if the default self-signed certificates are sufficient or if you want to replace with custom CA or VMware CA signed certs, see Installing vCenter Internal CA signed SSL Certificates for more information.

embedded

Other Considerations

  • The VCSA with embedded PSC requires the following hardware resources (disk can be thin provisioned)
    • Tiny (up to 10 hosts, 100 VMs) – 2 CPUs, 10 GB RAM.
    • Small (up to 100 hosts, 1000 VMs) – 4 CPUs, 16 GB RAM.
    • Medium (up to 400 hosts, 4000 VMs) – 8 CPUs, 24 GB RAM.
    • Large (up to 1000 hosts, 10,000 VMs) – 16 CPUs, 32 GB RAM.
    • X-Large (up to 2000 hosts, 35,000 VMs) – 24 CPUs, 48 GB RAM – new to v6.5.
  • Storage requirements for the smallest environments start at 250 GB and increase depending on your specific database requirements. See the Storage Requirements document for further details.
  • Where the PSC is deployed as a separate appliance this requires 2 CPUs, 4 GB RAM, 60 GB disk.
  • Environments with ESXi host(s) with more than 512 LUNs and 2048 paths should be sized large or x-large.
  • The ESXi host on which you deploy the VCSA should not be in lockdown or maintenance mode.
  • All vSphere components should be configured to use an NTP server. The installation can fail or the vCenter Server Appliance vpxd service may not be able to start if the clocks are unsynchronized.
  • FQDN resolution should be in place when deploying vCenter Server.
  • A list of Required Ports for vCenter Server and PSC can be found here.
  • The configuration maximums for vSphere 6.7 can be found here.
  • In vSphere 6.7 TLS 1.2 is enabled by default. TLS 1.0 and TLS 1.1 are disabled by default, review the Release Notes for more information.
  • There are a number of Intel and AMD CPUs no longer supported with vSphere 6.7, review the Release Notes for a full list of unsupported processors.

Installation

Download the VMware vCenter Server Appliance 6.7 ISO from VMware downloads: v6.7.0.

Mount the ISO on your computer. The VCSA 6.7 installer is compatible with Mac, Linux, and Windows. Browse to the corresponding directory for your operating system, e.g. \vcsa-ui-installer\win32. Right click Installer and select Run as administrator. As we are installing a new instance click Install.

VCSA_1

The installation is split into 2 stages, we begin with deploying the appliance. Click Next.

VCSA_2

Accept the license agreement and click Next.

VCSA_3

Select the deployment model, in this example we will be using an embedded deployment combining the vCenter Server and Platform Services Controller in one appliance, click Next.

VCSA_4

Enter the FQDN or IP address of the host, or vCenter upon which you wish to deploy the new VCSA. Enter the credentials of an administrative or root user and click Next. The installer will validate access, if prompted with an untrusted SSL certificate message click Yes to continue. Tip – connect to the vCenter for visibility of any networks using a distributed switch, connecting to the host direct will only pull back networks using a standard switch.

VCSA_5

Enter the VM name for the VCSA and a root password, click Next.

VCSA_6

Select the deployment size in line with the number of hosts and virtual machines that will be managed, click Next.

VCSA_7

Select the datastore where the VCSA will be deployed, select thin provisioning if required, and click Next. Configure the network settings for the appliance and click Next.

VCSA_8

On the summary page click Finish. The appliance will now be deployed.

VCSA_9

With the VCSA now deployed we can move on to stage 2, click Continue.

VCSA_10

Click Next to being the VCSA setup.

VCSA_11

Configure the NTP servers, enable SSH access if required, and click Next.

VCSA_12

Enter a unique SSO domain name, the default is vsphere.local. The SSO domain name should not be the same as your Active Directory Domain. Configure a password for the SSO administrator, click Next.

VCSA_13

Select or deselect the customer experience improvement program box and click Next.

VCSA_14

Review the details on the summary page and click Finish. Click Ok to acknowledge that the VCSA setup cannot be paused or stopped once started. When the installer is complete click Close to close the wizard.

Post-Installation

Connect to the vCenter post install using the IP or FQDN of the vCenter. Access vSphere by clicking either Launch vSphere Client (HTML5) or Launch vSphere Web Client (FLEX). As the web client will be depreciated in future versions, and the HTML5 client is now nearly at full feature parity, we will use the HTML5 vSphere client.

Windows_vCenter67_14

You must apply a new vCenter license key within 60 days. If you have purchased vCenter Server then log into your licensing portal here. If the license key does not appear then check with your VMware account manager. Log in to the vSphere Web Client using the SSO administrator login.  From the Menu drop-down click Administration,

Windows_vCenter67_16

Under Licensing select Licenses. First we need to add a new license key, click Add New Licenses. Enter the new license key for vCenter Server, click Next. If applicable assign a name to the licence, click Next. Click Finish to add the license key.

Windows_vCenter67_15

Switch to Assets, the vCenter Server is listed in evaluation mode. Highlight the vCenter and click Assign License. Select the license key and click Ok.

Windows_vCenter67_17

If you have an Active Directory domain then vCenter can use this as an identity source. First ensure the vCenter is joined to the domain; from the Menu drop-down click Administration. Under Single Sign On click Configuration. Select the Active Directory Domain tab and verify the vCenter is domain joined. Change to the Identity Sources tab and click Add Identity Source. Fill in the Active Directory details for your domain and click Ok.

Windows_vCenter67_18

You can now add permissions to vCenter objects such as datacenters, clusters, folders, individual virtual machines, etc. for Active Directory users and groups. To learn more about vSphere permissions click here.

To start adding ESXi hosts to vCenter click the Menu drop-down and select Hosts and Clusters. Right click the vCenter and select New Datacenter, give the datacenter a name and click Ok. Right click the datacenter and select Add Host. Follow the onscreen wizard to add a host. Creating clusters and configuring vCenter is beyond the scope of this post, for assistance follow the documentation links at the top of the page.

Windows_vCenter67_19

Updating vCenter Server with External PSC

The following post demonstrates the update process for applying minor updates to a vSphere environment running multiple vCenter Server appliances and external Platform Services Controllers.

In this instance we are updating vCenter to 6.5 U1e as one of the remediation actions for the Branch Target Injection issue (CVE-2017-5715) commonly known as Spectre. For more information on Meltdown and Spectre see this blog post, VMwares responses can be found here, on the VMware Security & Compliance Blog here, as well as VMware Security Announcement VMSA-2018-0004.2 here.

meltdown-spectre-vmware

Pre-Update Checks

When upgrading vSphere with an external Platform Services Controller (PSC), upgrade the PSC first, then the vCenter Server, then the ESXi hosts, and finally the virtual machines (hardware versions, VMware Tools).

Prior to updating vCenter ensure you have verified the compatibility of any third party products such as backups, anti-virus, monitoring, etc. Also cross-check the compatibility of other VMware products using the Product Interoperability Matrix. Since we are applying a minor update to vCenter Server the usual pre-requisites such as FQDN resolution, time synchronization, relevant ports open, etc. should already be in place. For vCenter 6.5 U1e all hosts must be running at least ESXi version 5.5. For more information on the requirements for vCenter Server 6.5, or if you are upgrading from an earlier version, the following posts may be of use:

Before beginning the update process take a backup and snapshot of the vCenter Server Appliance. There is downtime during the update but this is minimal – around 10 mins to update and reboot using an ISO as an update source, when using the online repository the update time may vary depending on your internet connection.

Review the version release notes and the VMware Docs site here.

VAMI Update

Platform Services Controller (PSC) appliances that are replicating should all be updated before the vCenter Server appliances. The easiest way of updating the vCenter Servers and Platform Services Controllers is through the VAMI (vCenter Server Appliance Management Interface). Browse to https://PSC:5480, where PSC is the FQDN or IP address of the external Platform Services Controller. Log in as the root user.

VAMI1

Select the Update option from the navigator.

vcupgrade2

Click the Check Updates drop-down. If the VCSA has internet access then select Check Repository to pull the update direct from the VMware online repository.

If the VCSA does not have internet access, or you’d prefer to provide the update manually then download the relevant update from VMware here (in this case VMware-vCenter-Server-Appliance-6.5.0.14000-7515524-patch-FP.iso) and attach the ISO to the CD/DVD drive of the VCSA in the virtual machine settings. Back in the VAMI update page select the Check Updates drop-down and click Check CDROM.

VAMI3

Details of the available update from either the online repository or attached ISO are displayed. Click Install Updates. Accept the EULA and click Install to begin the installation.

vcupgrade3

When the update process has completed click OK. From an attached ISO the update took around 5 minutes. The updated version and release date should now be displayed in the current version details. Finally, to complete the upgrade reboot the vCenter Server Appliance. Select Summary from the navigator and click Reboot.

vcupgrade4

If you are running multiple external PSCs then repeat the above process for each PSC in the SSO domain. Do not update the vCenter Server appliances until all PSC appliances are running the same updated version.

Once all external PSC appliances that replicate between one another have been upgraded then move on to the vCenter Server appliances. Repeat the above process for each vCenter Server in the SSO domain.

CLI Update

Alternatively the vCenter Server Appliance can be updated from the command line. Again, either using the online repository or by downloading the update from VMware here (VMware-vCenter-Server-Appliance-6.5.0.10000-5973321-patch-FP.iso or latest version) and attaching the ISO to the CD/DVD drive of the VCSA in the virtual machine settings. For more information on updating the vCenter Server Appliance using the appliance shell see this section of VMware docs.

Platform Services Controller (PSC) appliances that are replicating should all be updated before the vCenter Server appliances. Log in to the external Platform Services Controller appliance as root. First stage the patches from your chosen source using either:

  • software-packages stage --iso --acceptEulas stages software packages from ISO and accepts EULA.
  •  software-packages stage --url --acceptEulas stages software packages from the default VMware online repository and accepts EULA.

Next, review the staged packages, install the update, and reboot the VCSA.

  • software-packages list --staged lists the details of the staged software package.
  • software-packages install --staged installs the staged software package.
  • shutdown reboot -r update reboots the VCSA where ‘update’ is the reboot reason. Use -d to add a delay.

CLI4

If you are running multiple external PSCs then repeat the above process for each PSC in the SSO domain. Do not update the vCenter Server appliances until all PSC appliances are running the same updated version.

Once all external PSC appliances that replicate between one another have been upgraded then move on to the vCenter Server appliances. Repeat the above process for each vCenter Server in the SSO domain.

Installing vCenter Internal CA signed SSL Certificates

This post will walk through the process of replacing the default self-signed certificates in vCenter with SSL certificates signed by your own internal Certificate Authority (CA). In previous versions of vSphere the certificate replacement procedure was so complex that many administrators ignored it completely. Now with the certificate tool improvements in vSphere 6.x, and the ever increasing security threat of todays digital world, applying SSL certificates takes on an enhanced significance for verifying servers, solutions, and users are who they say they are.

The procedure outlined below is specific to installing Microsoft intermediate CA signed certificates on VCSA 6.5 with embedded PSC, protecting us against man in the middle attacks with a secure connection which we can see in the screenshot below. From v6.0 onwards the VMware Certificate Authority (VMCA) was also introduced, for more information on using the VMCA see this blog post, or to read how to use the VMCA as an intermediate CA see here. VMware documentation for replacing self-signed certificates can be reviewed from this KB article.

Trusted_vSphere

Before beginning the replacement certificate process ensure you have a good backup, and snapshot of the VCSA. The following links are the official VMware guides and this blog post provides a good overview of the certificates we’re actually going to be replacing. Replacing default certificates with CA signed SSL certificates in vSphere 6.x (2111219)Replacing a vSphere 6.x Machine SSL certificate with a Custom Certificate Authority Signed Certificate (2112277)How to replace the vSphere 6.x Solution User certs with CA signed certs (2112278).

468x60 vmware

Generate CSR

The first thing we need to do is generate a Certificate Signing Request (CSR). Open an SSH connection to the VCSA using an SSH client such as Putty, and login as root – if you need to enable SSH you can do so from the VAMI (https://vCenterIPorFQDN:5480) under Access; enable both SSH Login and Bash Shell. Run the following command to open the VMware built in Certificate Manager tool:

/usr/lib/vmware-vmca/bin/certificate-manager

Cert_Tool_1

Select the appropriate option. In this case we first want to replace the machine SSL certificate with a custom certificate, option 1. When prompted enter the SSO administrator username and password. Enter 1 again to generate certificate signing request(s) and Key(s) for machine SSL certificate, and enter the output directory. In the example below we are using the /tmp directory. Fill in the required values for the certool.cfg file.

Cert_Tool_2

The CSR and key are generated in the location specified. Change the shell to /bin/bash using chsh -s "/bin/bash" root and open an SCP connection to the VCSA using WinSCP. Copy the vmca_issued_csr.csr file to your local machine, you can use Notepad to view the contents of the file. Leave the WinSCP session open as we’ll need it to copy the certificate chain back to the VCSA.

Request Certificate

The next step is to use the CSR to request a certificate from your internal Certificate Authority (official KB here). A Microsoft CA template needs creating with the settings specified here (official KB here) before requesting the certs. Once this is done open a web browser to the Microsoft Certificate Services page (normally https://CAServer/certsrv) and select Request a Certificate.

Internal_CA_1

Then we want to Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file. The next page allows us to enter the CSR generated earlier to request a certificate with the pre-configured vSphere 6.5 certificate template.

Internal_CA_2

Click Submit and then select Base 64 encoded and Download certificate and Download certificate chain. A .cer file will be downloaded, I have renamed this machine_name_ssl.cer, and a .p7b. Double click the .p7b file to open in certmgr, locate and right click the root certificate, select All Tasks, Export. Export the root certificate in Base-64 encoded X.509 (.CER) format, in this example I have named the file Root64.cer. Using WinSCP copy the machine and root certificate files to the VCSA.

Install Certificate

Go back to Certificate Manager and enter 1 to continue to importing custom certificate(s) and key(s) for machine SSL certificate. Enter the file for the machine SSL certificate we copied, I have used /tmp/machine_name_ssl.cer. Enter the associated custom key that was generated with the CSR request, in this case /tmp/vmca_issued_key.key. Finally, enter the signing certificate of the machine SSL certificate, in this case /tmp/Root64.cer. When prompted enter y to replace the default machine SSL certificate with the custom certificate.

Cert_Tool_3

The certificate will now be installed, when finished a success message will be displayed. If certificate installation fails at 0% see this KB article.

Cert_Tool_4

To verify the machine certificate open a web browser to the vCenter FQDN, the connection will now show secure. Depending on the browser used you can view the certificate properties to verify it is correct, alternatively browse to https://vCenterFQDN/psc and log in with an SSO administrator account. Open Certificate Management and Machine Certificates, select the installed machine certificate and click Show Details, verify the certificate properties are correct.

Certificate_Management

Solution User Certificates

Repeat the steps above for the solution user certificates (official KB here). Replacing the solution user certificates may break some external plugins, such as SRM, in which case you should review this KB article for corrective action. To recap: /usr/lib/vmware-vmca/bin/certificate-manager. This time select option 5 replace solution user certificates with custom certificates. Generate the CSRs and keys, you will notice that for the solution user certs 4 CSR and key files are created; machine, vsphere-webclient, vpxd, and vpxd-extension.

Cert_Tool_5

Using WinSCP copy the files to your local machine and repeat the certificate request process from the Microsoft Certificate Services page. Copy the new certificates to the VCSA and repeat the install process. Solution User certificates can be viewed on the PSC web interface under Certificate Management, Solution User Certificates.

Solution_User_Management

vCenter Server Appliance Integrated TFTP Server

This post covers the steps required to use the vCenter Server Appliance for Auto Deploy, with the built in TFTP server in vSphere 6.5. For more information on Auto Deploy, and to see the process for creating ESXi images and deploy rules to boot hosts, see the VMware Auto Deploy 6.x Guide. This post assumes that you have a working vCenter Server Appliance, and may be of use if you have recently migrated from Windows vCenter to VCSA.

Enable Auto Deploy

Open the vSphere web client and click System Configuration, Nodes. Select the vCenter Server and open the Related Objects tab. The Auto Deploy, ImageBuilder Service, and VMware vSphere ESXi Dump Collector services should all be set to Automatic and Running.

To start a service right click and select Start, then select Edit Startup Type and choose Automatic.

servicesLog out of the web client and log back in. You should now see the Auto Deploy icon on the home page.

autodeploy1

Enable TFTP

Now that Auto Deploy is enabled we can configure the TFTP server. Enable SSH on the VCSA by browsing to the Appliance Management page: https://VCSA:5480 where VCSA is the IP or FQDN of your appliance.

Log in as the root account. From the Access page enable SSH Login and Bash Shell.

SSH

SSH onto the vCenter Appliance, using a client such as Putty, and log in with the root account. First type shell and hit enter to launch Bash.

To start the TFTP service enter service atftpd start. Check the service is started using service atftpd status.

startservice

To allow TFTP traffic through the firewall on port 69; we must run iptables -A port_filter -p udp -m udp –dport 69 -j ACCEPT. Validate traffic is being accepted over port 69 using iptables -nL | grep 69.

firewall

The TFTP server will now work, however we need to make a couple of additional changes to make the configuration persistent after the VCSA is rebooted. There isn’t an official VMware way of doing this, and as it’s done in Linux there may be more than one way of achieving what we want. Basically I am going to backup iptables and create a script to restore iptables and start the TFTP service when the appliance boots. The steps are outlined below and this worked for me, however as a reminder this is not supported by VMware, and if you are a Linux expert you’ll probably find a better way round it.

The following commands are all run in Bash on the vCenter Server Appliance, you can stay in the existing session we were using above.

First make a copy of the existing iptables config by running iptables-save > /etc/iptables.rules.

Next change the directory by running cd /etc/init.d, and create a new script: vi scriptname.sh, for example: vi starttftp.sh.

Press I to begin typing. I used the following, which was copied from the Image Builder Service startup script, and modified for TFTP.

#! /bin/sh
#
# TFTP Start/Stop the TFTP service and allow port 69
#
# chkconfig: 345 80 05
# description: atftpd

### BEGIN INIT INFO
# Provides: atftpd
# Required-Start: $local_fs $remote_fs $network
# Required-Stop:
# Default-Start: 3 5
# Default-Stop: 0 1 2 6
# Description: TFTP
### END INIT INFO

service atftpd start
iptables-restore -c < /etc/iptables.rules

The file must be in the above format to be compatible with chkconfig which runs the script at startup. I left the defaults in from the Image Builder Service as it made sense they started at the same time and had the same dependencies. If you wish to modify further see the following sources: Bash commands, Script Options, Startup, and vmwarebits.com for the iptables commands.

Press escape to leave the editor and :wq to save the file and quit.

Next set execute permissions on the script by running chmod +x scriptname.sh, for example: chmod +x starttftp.sh.

To set the script to run at startup use chkconfig –add scriptname.sh, for example: chkconfig –add starttftp.sh.

Reboot the vCenter appliance to test the script is running. If successful the atftpd service will be started and port 69 allowed, you can check these with service atftpd status and iptables -nL | grep 69.

Close the session and disable SSH if required.

Configure DHCP

In this example I will be using PXE boot to boot the ESXi hosts using a DHCP reservation. On the DHCP scope that will be serving the hosts I have configured reservations and enabled options 066 and 067. In the value for option 066 (Boot Server Host Name) goes the IP address or FQDN of the vCenter Server where TFTP is running. In the value for option 067 (Bootfile Name) I have entered the BIOS DHCP File Name (undionly.kpxe.vmw-hardwired).

DHCP

Now that Auto Deploy is up and running using the built-in components of VCSA 6.5 you can begin creating ESXi images and deploy rules to boot hosts; using the Auto Deploy GUI. See the VMware Auto Deploy 6.x Guide.

Configuring vCenter 6.5 High Availability

The vCenter Server Appliance now provides vCenter High Availability with vSphere 6.5 onwards. By implementing vCenter HA you can protect your vCenter from host and hardware failures, and significantly reduce down time during patching due to the active / standby nature of the vCenter cluster.

The vCenter HA architecture is made up of the components in the vSphere image below. The vCenter Server Appliance is cloned out to create passive and witness nodes. Updated data is replicated between the active and passive nodes. In the event of an outage to the active vCenter the passive vCenter automatically assumes the active role and identity. Management connections still route to the same IP address and FQDN, however they have now failed over to the replica node. When the outage is resolved and the vCenter that failed comes back online; it then takes on the role of the passive node, and receives replication data from the active vCenter Server.

vcenterha

468x60 vmware

Requirements

  • vCenter HA was introduced with the vCenter Server Appliance 6.5 .
  • The vCenter deployment size should be at least small, and therefore 4 vCPU 16 GB RAM.
  • A minimum of three hosts.
  • The hosts should be running at least ESXi 5.5.
  • The management network should be configured with a static IP address and reachable FQDN.
  • SSH should be enabled on the VCSA.
  • A port group for the HA network is required on each ESXi host.
  • The HA network must be on a different subnet to the management network.
  • Network latency between the nodes must be less than 10ms.
  • vCenter HA is compatible with both embedded deployment model and external PSC.
  • For further information on vCenter HA performance and best practises see this post.

Configuration Types

When setting up vCenter HA we are given the option of basic configuration or advanced. The correct deployment type depends on your environment. If the VCSA is managing its own ESXi host and virtual machine, or is managed by another vCenter Server in the same SSO domain then the basic deployment method should be used. This automatically clones the vCenter, and creates DRS anti-affinity rules.

If the VCSA is on a separate vCenter in a different SSO domain then the advanced deployment method should be used. In this case we need to manually add an additional NIC and clone the VCSA. The basic and advanced configuration types produce the same end result, but with a different process for different environments.

Both the embedded PSC and external PSC deployment models are supported. In this post we will walk through the advanced and basic configuration steps for vCenter with embedded PSC. For external PSC a load balancer can be implemented to provide HA, you can read more about implementing vCenter HA with the external deployment model here. If you are configuring vCenter HA in a cluster with less than the required number of physical hosts, such as in a home lab, you can add a parameter to override the anti-affinity setting; see this post by William Lam.

Basic Configuration Process

Log into the vSphere web client. Right click the top level vCenter Server in the inventory and select vCenter HA Settings. Click Configure in the top right hand corner.

ha1

Select the configuration type, in this example we are going to use Basic. Click Next.

basic1

An additional NIC will automatically be added to the active VCSA. Select the HA network to use and enter an IP address, remember this must be a separate subnet to the management network or the configuration wizard will error. Click Next.

basic2

Once the configuration wizard is complete the active VCSA will be cloned to create passive, and witness nodes. On this page we need to specify the HA IP addresses to use for each node, then click Next. You do not need to manually add any NICs during the basic configuration, this is all done for you. However as per the pre-requisites you do need to make sure a network is available to use for HA traffic.

basic3

Review the deployment page, if applicable you may need to change the compute or datastore locations by clicking Edit to ensure each component is spread across the vSphere cluster.

basic4

As you can see on the final page clone tasks will automatically be created. The new VMs are named VCSA-peer and VCSA-witness, where VCSA is the VM name of your current vCenter Server Appliance. Click Finish.

basic5

Monitor the tasks pane, vCenter HA may take around 5 minutes to clone and deploy the cluster nodes, depending on the speed of your underlying infrastructure. Once complete the vCenter HA status will show Enabled, and all nodes in the cluster will show Up.

basic6

You can edit the status of vCenter HA at any time by going back into the vCenter HA menu and clicking Edit. These are the available options.

edit

Advanced Configuration Process

The advanced deployment process takes longer as it involves much more manual configuration. The first thing we need to do is add an additional network adapter to our existing vCenter Server Appliance, and configure a vCenter HA IP address. Log into the vSphere web client of the vCenter managing the VCSA. Locate and right click the VCSA, select Edit Settings. From the New device drop down select Network and click Add. Select the port group to use, remember this needs to be a separate subnet to the management network, ensure Connect is ticked and click Ok.

Now we can configure the network settings using the Appliance Management portal. Browse to https:// :5480 where is the IP address or FQDN of your vCenter Server Appliance. Log in with the root password.

backup1

Select Networking on the left hand navigation menu.

backup2

Open the Manage tab and click Edit next to the Networking Interfaces box. Expand nic1, note that the status is down, configure the IP settings and click Ok.

ip

Verify that nic1 is now showing a status of Up.

interfaces

We can now start the vCenter HA configuration wizard. Open the vSphere web client of the VCSA for which you want to configure HA. Right click the top level vCenter Server in the inventory and select vCenter HA Settings. Click Configure in the top right hand corner.

ha1

Select the configuration type, in this example we are going to use Advanced. Click Next.

advanced1

Enter the IP address settings for the passive and witness nodes, on the HA network, then click Next.

advanced2

Now we need to do some manual cloning, go back to the vSphere client of the vCenter Server managing the VCSA. Locate the VCSA in the inventory, right click and select Clone, Clone to Virtual Machine.

Run through the clone wizard, let’s create the passive node first. During the clone wizard we configure all settings, including management IP address and host name, to be the same as the active VCSA except for the HA IP address. Each node has a unique IP address on the HA network.

  • Enter a name and location for the virtual appliance.
  • Select different compute resource and datastores to the active VCSA if possible.
  • On the clone options page select Customise the operating system, Power on virtual machine after creation.

clone1

  • On the customise guest OS page click the create new specification icon.
  • Enter a name and description for the new customisation.

clone2

  • Enter the same OS host name and domain as the active node.
  • Configure the same time zone as the active node.
  • On the network page edit the settings for NIC1, select use the following IP settings, and enter the management network settings of the active vcsa. This network adapter will be used to assume the identity of the active VCSA in the event of a fail over.

nic1

  • Edit the settings for NIC2, select prompt the user for an address when the specification is used. Enter the subnet mask and leave the gateway blank. This adapter will be used for the HA network, we will configure the unique IP address shortly.

nic2

  • On the DNS and domain settings page of the wizard enter the domain name and DNS server(s) that the interface will connect to, click Finish.
  • You will be returned to the clone virtual machine wizard. Select the newly created customisation profile.
  • Enter the IP address for NIC1. This is the HA IP for the passive node we input during the vCenter HA configuration wizard earlier.

usersettings

  • Accept the default virtual hardware and vApp properties.

The VCSA will now be cloned to create the passive node. Repeat the steps above for the witness node, however this time select the existing guest customisation that we created first time round.

customisation

Enter the unique HA IP address for the witness node that we specified during the vCenter HA configuration wizard.

usersettings

When these manual steps have been completed go back to the vCenter HA configuration wizard and click Finish. Monitor the Configure a vCenter HA Cluster task in the recent tasks pane.

clone

Once complete the vCenter HA status will show Enabled, and all nodes in the cluster will show Up.

basic6

For more information on vCenter HA or configuring different aspects of the advanced deployment; see the vCenter High Availability section of the vSphere 6.5 Documentation Centre.

The final step is to configure an anti-affinity rule to stop the vCenter Server appliances from running on the same hosts. Log into the vSphere web client and browse to Hosts and Clusters. Click the vSphere cluster and select the Manage tab. Under Configuration click VM/Host Rules. Under VM/Host Rules click Add.

Enter a name for the rule, such as vCenter HA, ensure Enable rule is ticked and select Separate Virtual Machines as the rule type. Click Add and select the vCenter Server nodes. Click Ok.

drs

This rule will ensure DRS does not place nodes on the same hosts in a vSphere cluster.

vCenter Appliance 6.5 Upgrade

VMware vCenter Server pools ESXi host resources to provide a rich feature set delivering high availability and fault tolerance to virtual machines. The vCenter Server is a centralised management application and can be deployed as a virtual appliance or Windows machine. This post walks through an upgrade of the vCenter Server Appliance from v5.5 or v6.0 to v6.5. See also vCenter Server Appliance 6.5 Install Guide, or Migrating Windows vCenter Server.


The latest vSphere version is now 6.7, updated posts:

vCenter Server Appliance 6.7 Install Guide

Windows vCenter Server 6.7 Install Guide

Migrating Windows vCenter Server to VCSA 6.7

About VCSA

The VCSA is a pre-configured virtual appliance; as of v6.5 the operating environment is built on Project Photon OS 1.0. Since the OS has been developed by VMware it benefits from enhanced performance and boot times over the previous Linux based appliance. Furthermore the embedded Postgre database means VMware have full control of the software stack, resulting in significant optimisation for vSphere environments and quicker release of security patches and bug fixes. The VCSA scales up to 2000 hosts and 35,000 virtual machines.

In vSphere 6.0 the VCSA reached feature parity with its Windows counterpart, 6.5 begins to pave the way for VCSA to become the preferred deployment method for vCenter Server. One key addition is the inclusion of Update Manager bundled into the VCSA, as well as vCenter High Availability, Backup and Restore, and other features. The appliance also saves operating system license costs and is quicker and easier to deploy and patch.

Upgrading to VCSA 6.5 involves the deployment of a new appliance and migration of all configuration and historical data (optional) using the upgrade installer. The VCSA uses a temporary IP address during migration before switching to the IP and host name of the new VCSA, the old appliance is then powered off.

vcs

Software Considerations

  • VCSA 6.5 must be deployed to an ESXi host running v5.5 or above. All hosts you intend to connect to vCenter Server 6.5 should also be running ESXi v5.5 or above.
  • The VCSA to be upgraded can be either v5.5 or v6.0.
  • VCSA 6.5 does not support the use of an external database. Where a system using an external database is upgraded, the data is imported into the internal Postgres database within VCSA 6.5.
  • You must check compatibility of any third party products and plugins that might be used for backups, anti-virus, monitoring, etc. as these may need upgrading for use with vSphere 6.5.
  • If you are unsure check the Product Interoperability Matrix.

Architectural Considerations

  • From vSphere 6 onwards the Platform Services Controller (PSC) was introduced to the vSphere architecture. The PSC contains infrastructure services such as Single Sign On, Certificate Authority, licensing, etc. The PSC is deployed internally with vCenter Server or as an external component. Read more about the PSC in this KB.
  • When implementing a new vSphere 6.5 environment you should plan your topology in accordance with the VMware vCenter Server and PSC Deployment Types. Larger environments may require an external PSC.
  • When upgrading vCenter the deployment model already in place will be migrated, the upgrade supports different deployment topologies but can not make changes to the topology and SSO domain configuration.
  • In this post we will be upgrading vCenter Server 6.0 using the embedded deployment model. If you are using an external deployment model the PSC appliance must be upgraded before the vCenter Server.
  • Consider if the default self-signed certificates are sufficient or if you want to replace with custom CA or VMware CA signed certs, see Installing vCenter Internal CA signed SSL Certificates for more information.

embedded

Other Considerations

  • VCSA 6.5 with embedded PSC requires the following hardware resources (disk can be thin provisioned)
    • Tiny (up to 10 hosts, 100 VMs) – 2 CPUs, 10 GB RAM.
    • Small (up to 100 hosts, 1000 VMs) – 4 CPUs, 16 GB RAM.
    • Medium (up to 400 hosts, 4000 VMs) – 8 CPUs, 24 GB RAM.
    • Large (up to 1000 hosts, 10,000 VMs) – 16 CPUs, 32 GB RAM.
    • X-Large (up to 2000 hosts, 35,000 VMs) – 24 CPUs, 48 GB RAM – new to v6.5.
  • Storage requirements for the smallest environments start at 250 GB and increase depending on your specific database requirements. See the Storage Requirements document for further details.
  • Where the PSC is deployed as a separate appliance this requires 2 CPUs, 4 GB RAM, 60 GB disk.
  • Environments with ESXi host(s) with more than 512 LUNs and 2048 paths should be sized large or x-large.
  • The ESXi host on which you deploy the new appliance should not be in lockdown or maintenance mode.
  • All vSphere components should be configured to use an NTP server. The installation can fail or the vCenter Server Appliance vpxd service may not be able to start if the clocks are unsynchronized.
  • FQDN resolution should be in place when deploying vCenter Server.
  • Review the list of Required Ports for vCenter Server and PSC.
  • Official resources – vSphere 6.5 Documentation Centre, vSphere 6.5 Release Notes.
  • Read the Important information before upgrading to vSphere 6.5 KB.

Installation

Download the VMware vCenter Server Appliance 6.5 ISO from VMware downloads: v6.5.0 | v6.5.0 U1.

Unlike the Windows vCenter installer, which hasn’t changed much in v6.5; the VCSA installer has had a complete overhaul. You’ll notice straight away that the GUI is much cleaner, and multiple deployment options (install, upgrade, migrate, restore) are now bundled into one installer.

Mount the ISO on your computer. The VCSA 6.5 installer is compatible with Mac, Linux, and Windows. Browse to the corresponding directory for your operating system, e.g. \vcsa-ui-installer\win32. Right click Installer and select Run as administrator. As we are upgrading an existing system click Upgrade.

vcsa1

The installation is split into 2 stages, we begin with deploying a new appliance. The second stage migrates all data and settings. Click Next.

upgrade1

Accept the EULA and click Next.

upgrade2

Enter the details for the existing vCenter Server Appliance and the host or vCenter it is managed by. Click Next.

upgrade3

Enter the FQDN or IP address of the host, or vCenter upon which you wish to deploy the new VCSA, then click Next. The installer will validate access, if prompted with an untrusted SSL certificate message click Yes to continue. Tip – connect to the vCenter for visibility of any networks using a distributed switch, connecting to the host direct will only pull back networks using a standard switch.

upgrade4

Enter a VM name and root password for the new appliance, and click Next.

upgrade5

Configure the deployment size of the new appliance and click Next.

upgrade6

Select the datastore to locate the virtual appliance and click Next. Configure the temporary network settings for the appliance. These will only be used during migration of the data, once complete the temporary settings are discarded and the VCSA assumes the identity, including IP settings, of the old appliance. Click Next.

upgrade7

The new VCSA will now be deployed, once complete click Finish.

upgrade8

Stage 2 migrates data and identity across to the new VCSA, click Next.

upgrade9

Select the data to migrate and click Next.

upgrade10

Select whether or not to join the VMware Customer Experience Improvement Program and click Next.

upgrade11

Review the summary page, confirm you have taken a backup of the vCenter and click Finish.Click OK to the shut down warning.

upgrade12

Data will now be migrated to the new VCSA, once complete the old VCSA will be powered off and the network settings transferred.

upgrade13

When finished click Close, the vCenter upgrade is complete.

upgrade14

Post-Installation

Connect to the vCenter post install using the IP or FQDN of the vCenter. Access vSphere by clicking either the vSphere Web Client (Flash) or the vSphere Client (HTML5). Connect to the vSphere Web Client to manage your system, the thick client (Windows) is no longer supported.

vsphereweb

Log in to the vSphere Web Client using the SSO administrator login. Verify the installed version is correct under the Summary tab when selecting the vCenter, you can also go to Help > About.

vsphereclient

You must apply a new vCenter license key within 60 days. From the Hosts and Clusters view select the vCenter Server. Click Actions and Assign License. Select a license or use the green plus button to add a new license and click Ok.

You can obtain a 60 day trial license for vCenter Server here. If you have purchased vCenter Server then log into your licensing portal here. If the license key does not appear then check with your VMware account manager.

client