Tag Archives: NSX

NSX 6.4.1 Upgrade Guide

This post will walk through upgrading to NSX 6.4.1. If upgrading from 6.4.0 then the new Upgrade Coordinator feature can be used, allowing simultaneous upgrade planning of multiple NSX objects, see the NSX 6.4.x Upgrade Coordinator post for more information. If upgrading from an earlier version than 6.4.0 then the steps outlined below are applicable. When performing an upgrade the NSX components must be upgraded in the following order: NSX Manager, NSX Controllers, Host Clusters, NSX Edge, Service Virtual Machines (such as Guest Introspection).

Review the operational impacts of NSX upgrades for each component here when planning your upgrade, it is best practise to limit all operations in the environment until the upgrade is complete. Make sure NSX Manager is backed up before starting an upgrade, and be aware that after a successful upgrade NSX cannot be downgraded. You should also review the VMware NSX for vSphere 6.4.1 Release Notes here and NSX for vSphere Documentation Center here.

Requirements

Requirements specific to NSX 6.4.1 are listed below. As we are doing an upgrade the assumption is that the vSphere and NSX environment is already setup and working, you can validate the existing NSX configuration here. You should also ensure an underlying network with IP connectivity and an MTU size of 1600 or above, FQDN resolution, connectivity, and time synchronisation between NSX and vSphere components, syslog, monitoring, and backups are all in place. In addition review the basic system requirements for NSX here and the full list of network port requirements here.

  • NSX 6.4.1 is compatible with vSphere versions 6.0 U2 and above, also note; if you are using 6.0 then U3 is recommended, the minimum supported version for 6.5 is 6.5a, support for 5.5 has now been removed
  • Supported upgrade paths to NSX 6.4.1 are from 6.2.4 onwards, there is a workaround for upgrading from 6.2.0, 6.2.1, or 6.2.2 which can be found here
  • Review the VMware Upgrade Path page here and also fully review the NSX 6.4.1 Release Notes here, as there are a number of things to be aware of when upgrading from 6.2.x or 6.3.x
  • Check compatibility with VMware products using the VMware Interoperability page here
  • Check compatibility with other third party products such as partner services for Guest Introspection using the VMware Compatibility Guide here
  • Before starting the upgrade make sure existing appliances meet the recommended hardware requirements:
    • NSX Manager 16 GB RAM (24 GB for large deployments), 4 vCPU (8 vCPU for large deployments), and 60 GB disk, a large deployment is typically 256+ hosts or 2000+ VMs
    • NSX Controllers 4 GB RAM, 4 vCPU, and 28 GB disk
    • NSX Edge Compact: 512 MB RAM, 1 vCPU, 584 MB + 512 MB disks. Large: 1 GB RAM, 2 vCPU, 584 MB + 512 MB disks. Quad Large: 2 GB RAM, 4 vCPU, 584 MB + 512 MB disks. X-Large: 8 GB RAM, 6 vCPU, 584 MB + 2 GB + 256 MB disks.
  • Verify the existing NSX Manager has sufficient space by connecting to the CLI (if using SSH service may need starting on the summary page of NSX Manager appliance page) and running show filesystems
  • Maximum latency between NSX components and NSX and vSphere components should be 150 ms RTT or below
  • NSX Data Security is no longer supported, it should be removed if installed prior to the upgrade
  • If you are using Cross-vCenter NSX then each component should be upgraded in the order listed here
  • Enabling DRS on the vSphere cluster allows running VMs to be automatically migrated when each host is placed into maintenance mode for the NSX VIB upgrades. This process can of course be undertaken manually if DRS is not in use
  • A completed upgrade can be validated following the steps listed here

Backups

Before we start take a backup of the vCenter Server and NSX Manager. NSX configuration can be backed up using FTP/SFTP, see this post for more information. From version 6.4.1 a configuration backup is automatically taken at the start of the upgrade process, this is intended as a fall back and you should still take your own backup before beginning. You can also take a snapshot of the NSX Manager incase we need to revert back the NSX Manager upgrade. For extra peace of mind export the vSphere Distributed Switch configuration by following the instructions here.

In the event you do need to restore from an NSX backup a new appliance should be deployed and the configuration restored, click here for further details.

Upgrade Process

As noted above make sure you have read all the linked documentation, specifically the release notes and operational impacts for each component upgrade. The steps below will not list the operational impact for each step of the upgrade.

Download the NSX for vSphere 6.4.1 Upgrade Bundle from the download page here to a location accessible from the NSX Manager. Browse to the NSX Manager and log in as admin. From the home page click Upgrade.

Click Upload Bundle and browse to the upgrade bundle downloaded earlier, click Continue. Once the bundle is uploaded you can (optional) select to enable SSH and/or join the Customer Experience Improvement Program. Click Upgrade to start the upgrade.

NSX64_1

The installer will now upgrade NSX Manager, once complete you will be returned to the login page.

NSX64_2

Log back into NSX Manager and click Upgrade. Verify the upgrade state is complete and the version number is correct. Click Summary and verify the health of the NSX Manager.

NSX64_3

Log into the vSphere Client, if you were already logged in then log out and back in, or you may need to clear your browser cache. From the Menu drop-down select Networking and Security.

Before upgrading any other components we need to upgrade the NSX Controller Cluster. On the Dashboard tab confirm there are 3 controller nodes all connected, the upgrade cannot commence if any nodes are in a disconnected state.

NSX64_5

Click Installation and Upgrade and select the Management and NSX Managers tab. Check the NSX Manager version is correct, in the Controller Cluster Status column click Upgrade Available.

NSX641_1

Each controller is upgraded and rebooted one at a time. From NSX 6.3.3 onwards the underlying operating system of the controller nodes changed to Photon-OS. If you are upgrading from 6.3.3 onwards an in-place upgrade is applied. If you are upgrading from 6.3.2 or earlier then the controller nodes are redeployed, any DRS rules anti-affinity rules are lost and will need to be reapplied.

Click Yes to being the Controller Cluster upgrade.

NSX641_2

Monitor the status in the NSX Controller Nodes tab. After all the controller nodes have been upgraded validated the Status, Peers, and Upgrade Status are all green. Confirm the Software Version is correct.

NSX641_3

Next we can upgrade the host NSX VIBs, click the Host Preparation tab. Clusters running NSX are displayed, upgrades are initiated on a per cluster basis. Select the cluster and click Upgrade to begin the upgrade.

Hosts running NSX 6.2.x require a reboot for the installation of new VIBs, hosts running NSX 6.3.0 and above do not need a reboot but must be placed into maintenance mode. You can either manually place hosts into maintenance mode and vMotion / power off VMs yourself, or allow DRS to live migrate VMs and remediate hosts one at a time.

NSX641_4

Click Yes to commence the cluster upgrade.

NSX641_5

At this stage if hosts are not in maintenance mode the NSX Installation will show Not Ready. If you have DRS enabled on the cluster click Actions and Resolve All, this will automatically vMotion running machines from a host, place into maintenance mode, update the VIBs, and exit maintenance mode, one host at a time. Alternatively you can select individual hosts and click Resolve if you want to control the order of the upgrades.

NSX641_6

Monitor the status of the NSX Installations in the Hosts table. You can also monitor Recent Tasks to make sure a host is not taking too long to enter maintenance mode, if a host cannot be evacuated due to DRS rules, or a VM that cannot be migrated then manual intervention may be required (in this case see here).

If you are using stateless images with Auto Deploy you should also update your ESXi image with the latest NSX VIBs or they will be lost at next reboot, for guidance see this post.

NSX641_7

The next step is to upgrade NSX Edges. Before commencing with validate the status of all NSX prepared hosts is green and they are showing successfully upgraded to the correct version. During Edge upgrades a replacement appliance is deployed which means 2 appliances (or 4 if running in HA mode) are powered on at the same time, ensure your cluster has sufficient compute resource.

NSX641_8

At the time of writing (v6.4.1) NSX Edges still need to be upgraded using the vSphere web client. Log into the vSphere web client and click Networking & Security, NSX Edges, deployed Edges are displayed .If you have multiple NSX Managers ensure the correct NSX Manager is selected in the drop-down. Select the NSX Edge to upgrade and from the Actions menu click Upgrade Version.

NSX641_9

The upgraded version will be deployed from OVF, you can follow the progress in the Recent Tasks pane and also the Status column for the Edge. Repeat this process for each Edge Services Gateway (ESG) and Distributed Logical Router (DLR) you wish to upgrade.

NSX641_10

The final stage is to upgrade Guest Introspection. This can either be done in the vSphere web client or by going back into the HTML5 web client. From the Menu drop-down select Networking and Security, click Installation and Upgrade and the Service Deployment tab. Existing service deployments are listed, the Installation Status for Guest Introspection shows Upgrade Available. Select the Guest Introspection deployment and click Upgrade, once complete verify the Installation Status and Service Status are both green and the version number is correct.

NSX641_11

After all NSX components are upgraded if you want to follow additional verification steps then see the upgrade validation KB here, or the post upgrade tasks listed here. You should take a further backup of NSX Manager after completion of the upgrade. Any third party appliances for Guest Introspection or Network Introspection that require an update can now be upgraded.

NSX 6.4.x Upgrade Coordinator

This post will walk through an upgrade to NSX 6.4.1 using the new Upgrade Coordinator feature allowing simultaneous upgrade planning of multiple NSX components. If you are upgrading from an earlier version of NSX, see the NSX 6.4.1 Upgrade Guide post for details on upgrading individual components. From version 6.4 onwards upgrade plans can be used to upgrade host clusters, controller clusters, Edge Service Gateways (ESGs), Distributed Logical Routers – including Universal (DLRs and UDLRs), and Service Virtual Machines such as Guest Introspection. Upgrade plans consist of either a one click system managed upgrade, or planning your own upgrade where objects and options can be customised.

Review the operational impacts of NSX upgrades for each component here when planning your upgrade, it is best practise to limit all operations in the environment until the upgrade is complete. Make sure NSX Manager is backed up before starting an upgrade, and be aware that after a successful upgrade NSX cannot be downgraded. You should also review the VMware NSX for vSphere 6.4.1 Release Notes here and NSX for vSphere Documentation Center here.

Requirements

Requirements specific to NSX 6.4.1 are listed below. As we are doing an upgrade the assumption is that the vSphere and NSX environment is already setup and working, you can validate the existing NSX configuration here. You should also ensure an underlying network with IP connectivity and an MTU size of 1600 or above, FQDN resolution, connectivity, and time synchronisation between NSX and vSphere components, syslog, monitoring, and backups are all in place. In addition review the basic system requirements for NSX here and the full list of network port requirements here.

  • NSX 6.4.1 is compatible with vSphere versions 6.0 U2 and above, also note; if you are using 6.0 then U3 is recommended, the minimum supported version for 6.5 is 6.5a, support for 5.5 has now been removed
  • Supported upgrade paths to NSX 6.4.1 are from 6.2.4 onwards, there is a workaround for upgrading from 6.2.0, 6.2.1, or 6.2.2 which can be found here
  • Review the VMware Upgrade Path page here and also fully review the NSX 6.4.1 Release Notes here, as there are a number of things to be aware of when upgrading from 6.2.x or 6.3.x
  • Check compatibility with VMware products using the VMware Interoperability page here
  • Check compatibility with other third party products such as partner services for Guest Introspection using the VMware Compatibility Guide here
  • Before starting the upgrade make sure existing appliances meet the recommended hardware requirements:
    • NSX Manager 16 GB RAM (24 GB for large deployments), 4 vCPU (8 vCPU for large deployments), and 60 GB disk, a large deployment is typically 256+ hosts or 2000+ VMs
    • NSX Controllers 4 GB RAM, 4 vCPU, and 28 GB disk
    • NSX Edge Compact: 512 MB RAM, 1 vCPU, 584 MB + 512 MB disks. Large: 1 GB RAM, 2 vCPU, 584 MB + 512 MB disks. Quad Large: 2 GB RAM, 4 vCPU, 584 MB + 512 MB disks. X-Large: 8 GB RAM, 6 vCPU, 584 MB + 2 GB + 256 MB disks.
  • Verify the existing NSX Manager has sufficient space by connecting to the CLI (if using SSH service may need starting on the summary page of NSX Manager appliance page) and running show filesystems
  • Maximum latency between NSX components and NSX and vSphere components should be 150 ms RTT or below
  • NSX Data Security is no longer supported, it should be removed if installed prior to the upgrade
  • If you are using Cross-vCenter NSX then each component should be upgraded in the order listed here
  • Enabling DRS on the vSphere cluster allows running VMs to be automatically migrated when each host is placed into maintenance mode for the NSX VIB upgrades. This process can of course be undertaken manually if DRS is not in use
  • A completed upgrade can be validated following the steps listed here

Backups

Before we start take a backup of the vCenter Server and NSX Manager. NSX configuration can be backed up using FTP/SFTP, see this post for more information. From version 6.4.1 a configuration backup is automatically taken at the start of the upgrade process, this is intended as a fall back and you should still take your own backup before beginning. You can also take a snapshot of the NSX Manager incase we need to revert back the NSX Manager upgrade. For extra peace of mind export the vSphere Distributed Switch configuration by following the instructions here.

In the event you do need to restore from an NSX backup a new appliance should be deployed and the configuration restored, click here for further details.

Upgrade Process

As noted above make sure you have read all the linked documentation, specifically the release notes and operational impacts for each component upgrade. The steps below will not list the operational impact for each step of the upgrade.

Download the NSX for vSphere 6.4.1 Upgrade Bundle from the download page here to a location accessible from the NSX Manager. Browse to the NSX Manager and log in as admin. From the home page click Upgrade.

Click Upload Bundle and browse to the upgrade bundle downloaded earlier, click Continue. Once the bundle is uploaded you can (optional) select to enable SSH and/or join the Customer Experience Improvement Program. Click Upgrade to start the upgrade.

NSX64_1

The installer will now upgrade NSX Manager, once complete you will be returned to the login page.

NSX64_2

Log back into NSX Manager and click Upgrade. Verify the upgrade state is complete and the version number is correct. Click Summary and verify the health of the NSX Manager.

NSX64_3

Log into the vSphere Client, if you were already logged in then log out and back in, or you may need to clear your browser cache. From the Menu drop-down select Networking and Security.

For any upgrade plan the NSX Controller Cluster upgrade is mandatory and performed first. On the Dashboard tab confirm there are 3 controller nodes all connected, the upgrade cannot commence if any nodes are in a disconnected state.

NSX64_5

Click Installation and Upgrade and select the Upgrade tab. Review the components, any warnings, and current and target version details.

NSX64_4

To start an upgrade plan click Plan Upgrade.

Upgrade Coordinator puts objects of the same type in default upgrade groups when planning an upgrade. These groups and other settings can be modified by planning your own upgrade (controller upgrades are mandatory) or you can allow the system to upgrade everything using a one click upgrade. Select the desired upgrade plan and click Next.

NSX64_7

The default options for the one click upgrade are to upgrade Host Clusters and Service VMs individually (serial), and to upgrade NSX Edges all together (parallel). There is no pause between components or pause on error. If you are happy with these settings then click Start Upgrade to being the upgrade process, otherwise go back to Plan Your Upgrade.

NSX64_8

Select your own upgrade to choose which components are upgraded, controller upgrades are mandatory and are done first. You can also pause the upgrade between components or pause the upgrade if an error is returned.

NSX64_9

The next 3 pages of the Upgrade Coordinator allow you to manage upgrade groups for Host Clusters, NSX Edges, and Service VMs. When planning your upgrade take into consideration the following:

  • Objects of the same type can be added to or removed from an upgrade group
  • The order of object upgrades within a group can be changed
  • All components included in an upgrade group must be upgraded before the next component type can be upgraded, e.g. all hosts included in an upgrade plan must be upgraded before moving onto Edges, and so on
  • Excluding an object within an upgrade group is useful for multiple maintenance windows, where you want to add an object to an upgrade plan but exclude them from this upgrade session
  • If the upgrade order within group is set to Serial then each object is upgraded one at a time, if it is Parallel then multiple objects within that group are upgraded at the same time

Controller Upgrades: each controller is upgraded and rebooted one at a time. From NSX 6.3.3 onwards the underlying operating system of the controller nodes changed to Photon-OS. If you are upgrading from 6.3.3 onwards an in-place upgrade is applied. If you are upgrading from 6.3.2 or earlier then the controller nodes are redeployed, any DRS rules anti-affinity rules are lost and will need to be reapplied.

Host Upgrades: hosts running NSX 6.2.x require a reboot for the installation of new VIBs, hosts running NSX 6.3.0 and above do not need a reboot but must be placed into maintenance mode. You can either manually place hosts into maintenance mode and vMotion / power off VMs yourself, or allow DRS to live migrate VMs and remediate hosts one at a time. Monitor the status of the NSX Installations on the Upgrade tab. You can also monitor Recent Tasks to make sure a host is not taking too long to enter maintenance mode, if a host cannot be evacuated due to DRS rules, or a VM that cannot be migrated then manual intervention may be required (in this case see here).

If you are using stateless images with Auto Deploy you should also update your ESXi image with the latest NSX VIBs or they will be lost at next reboot, for guidance see this post.

NSX64_10

Configure your upgrade plan based on the components you want to upgrade in this session, and review the final plan. When you’re read click Start Upgrade to begin the upgrade process.

NSX64_13

Monitor the status of the upgrade on the Upgrade page. If any warnings or errors are displayed during the upgrade process see the Monitor and Troubleshoot Your Upgrade page here. If you selected Pause between components you must Resume or Replan after each stage of the upgrade.

nsx64_14

An in-progress upgrade plan can still be paused to make modifications; when paused the object currently being upgraded will continue and the upgrade plan pauses when this object upgrade succeeds or fails.

nsx64_15

After the upgrade is complete verify the Upgrade page shows the system upgrade status successful.

nsx64_16

Verify the NSX health from the Dashboard page. After all NSX components are upgraded if you want to follow additional verification steps then see the upgrade validation KB here, or the post upgrade tasks listed here. You should take a further backup of NSX Manager after completion of the upgrade. Any third party appliances for Guest Introspection or Network Introspection that require an update can now be upgraded.

Setting Manual DFW Override for NSX Restore

The recommended restore process for NSX Manager is to deploy a new OVA of the same version, and restore the configuration. After a recent failed upgrade we needed to restore NSX Manager, so deployed a new OVA with the same network settings. After the new NSX Manager was powered on we were unable to ping the IP address, this was because there were no default rules allowing access to the VM, and since the existing NSX Manager was down we were unable to connect to the UI or API to add the required firewall rules. NSX Manager is normally excluded from Distributed Firewall (DFW) by default, however at this point the hosts saw it as any other VM, since we had not yet restored the configuration. Therefore we needed to add a manual override to clear the filters applied to the new NSX Manager, allowing us to connect and restore the configuration. The following commands were run on the host where the new NSX Manager OVA was deployed, using SSH. For further guidance on the backup and restore process of NSX see the NSX Backup and Restore post.

Disclaimer: the steps below are advanced commands using vsipfwcli which is an extremely powerful tool. You should engage VMware GSS if doing this on anything other than a lab environment, you should also understand the impact of stopping the vsfwd service on the host and the impact this may have on any other VMs with a DFW policy of fail closed.

net-stats -l lists the NIC details of the VMs running on the host, verify the new NSX Manager is present.

/etc/init.d/vShield-Stateful-Firewall stop stops the vsfwd user world agent, you can also use status to display the status.

vsfwd

summarize-dvfilter lists port and filter details, we need the port name for the VM, e.g. nic-38549-eth0-vmware-sfw.2.

DFW_1

vsipioctl getrules -f nic-38549-eth0-vmware-sfw.2 lists the existing filters applied to the port, replace the port name with your own, from the output check to confirm the ruleset name, e.g. ruleset domain-c17.

DFW_2

vsipioctl vsipfwcli -f nic-38549-eth0-vmware-sfw.2 -c "create ruleset domain-c17;" creates a new empty ruleset with the same name, overriding the previous ruleset applied to the port. Replace the port name with your own and the ruleset name if it is different.

vsipioctl getrules -f nic-38549-eth0-vmware-sfw.2 again lists the existing filters applied to the port, the ruleset should now be empty as no filters are applied.

DFW_3

The NSX Manager is now pinging and the normal restore process can resume; connect to the web interface by browsing to the IP address or FQDN of the NSX Manager.

Restore_NSX_1

Select Backup & Restore.

Restore_NSX_2

Select the appropriate restore point and click Restore. Click Yes to confirm.

Restore_NSX_3

The restore generally takes 5-10 minutes, once complete you will see a restore completed successfully message in a blue banner on the Summary page. You may need to log out and log back in after the config is restored.

Restore_NSX_4

Once the NSX Manager services have started you can manage the DFW from the vSphere web client as normal. Remember to start the vsfwd service again on the host, after the vsfwd service is started the empty ruleset we created earlier is replaced with the original ruleset when the host syncs with NSX Manager.

/etc/init.d/vShield-Stateful-Firewall start starts the vsfwd user world agent, you can also use status to display the status.