Setting Manual DFW Override for NSX Restore

The recommended restore process for NSX Manager is to deploy a new OVA of the same version, and restore the configuration. After a recent failed upgrade we needed to restore NSX Manager, so deployed a new OVA with the same network settings. After the new NSX Manager was powered on we were unable to ping the IP address, this was because there were no default rules allowing access to the VM, and since the existing NSX Manager was down we were unable to connect to the UI or API to add the required firewall rules. NSX Manager is normally excluded from Distributed Firewall (DFW) by default, however at this point the hosts saw it as any other VM, since we had not yet restored the configuration. Therefore we needed to add a manual override to clear the filters applied to the new NSX Manager, allowing us to connect and restore the configuration. The following commands were run on the host where the new NSX Manager OVA was deployed, using SSH. For further guidance on the backup and restore process of NSX see the NSX Backup and Restore post.

Disclaimer: the steps below are advanced commands using vsipfwcli which is an extremely powerful tool. You should engage VMware GSS if doing this on anything other than a lab environment, you should also understand the impact of stopping the vsfwd service on the host and the impact this may have on any other VMs with a DFW policy of fail closed.

net-stats -l lists the NIC details of the VMs running on the host, verify the new NSX Manager is present.

/etc/init.d/vShield-Stateful-Firewall stop stops the vsfwd user world agent, you can also use status to display the status.

vsfwd

summarize-dvfilter lists port and filter details, we need the port name for the VM, e.g. nic-38549-eth0-vmware-sfw.2.

DFW_1

vsipioctl getrules -f nic-38549-eth0-vmware-sfw.2 lists the existing filters applied to the port, replace the port name with your own, from the output check to confirm the ruleset name, e.g. ruleset domain-c17.

DFW_2

vsipioctl vsipfwcli -f nic-38549-eth0-vmware-sfw.2 -c "create ruleset domain-c17;" creates a new empty ruleset with the same name, overriding the previous ruleset applied to the port. Replace the port name with your own and the ruleset name if it is different.

vsipioctl getrules -f nic-38549-eth0-vmware-sfw.2 again lists the existing filters applied to the port, the ruleset should now be empty as no filters are applied.

DFW_3

The NSX Manager is now pinging and the normal restore process can resume; connect to the web interface by browsing to the IP address or FQDN of the NSX Manager.

Restore_NSX_1

Select Backup & Restore.

Restore_NSX_2

Select the appropriate restore point and click Restore. Click Yes to confirm.

Restore_NSX_3

The restore generally takes 5-10 minutes, once complete you will see a restore completed successfully message in a blue banner on the Summary page. You may need to log out and log back in after the config is restored.

Restore_NSX_4

Once the NSX Manager services have started you can manage the DFW from the vSphere web client as normal. Remember to start the vsfwd service again on the host, after the vsfwd service is started the empty ruleset we created earlier is replaced with the original ruleset when the host syncs with NSX Manager.

/etc/init.d/vShield-Stateful-Firewall start starts the vsfwd user world agent, you can also use status to display the status.

EMC Networker Restore

This post explains the process of restoring a physical machine that has been backed up prior by Networker. The restore process will be a bare metal restore and is specific to environments where two NICs are used for backup, for example a private backup VLAN and a public VLAN. The Networker Bare Metal Recovery Wizard can only configure one physical NIC, therefore we need to use the CLI to configure the second NIC.

Procedure

To restore a physical server using a bare metal restore we first need to boot from the EMC Networker BMR ISO. Mount the ISO through the iLO or load the physical media into the server.

Once you have booted from the ISO you will be presented with the Networker Bare Metal Recovery Wizard. This may take 5 – 10 minutes to load after the command prompt window.

Set the current date and time, click Next.

restore1

Select the network interface that you want to configure, for this you should select the NIC that is connected to the public VLAN (i.e. the server OS netork settings) and click Next.

restore2

In the host and network screen enter the server name and domain if applicable. Enter the servers network settings and click Next. If the network is not configured correctly the restore will fail.

Confirm the physical disk on the server that you want to restore to and click Next. Note that the disk will be formatted and any existing data on the disk will be overwritten.

restore3

Before selecting the restore server we need to take some additional steps. In the background you will see a command prompt, you will need to do the following:

Configure the backup VLAN IP address using:

  • netsh interface ip set address name=”Ethernet Adapter” static <IP> <Subnet Mask> replacing Ethernet Adapter with the name of the interface connected to the backup VLAN, and the IP and subnetmask accordingly.
  • Run ipconfig /all to confirm the network settings are correct for both interfaces.
  • Test network connectivity by pinging other IP addresses on the same subnet.
  • If the server does not have access to DNS then you can type notepad.exe in command prompt and then open Boot (X:) > Windows\System32\drivers\etc\hosts. Add the IP and host names of any servers you need to resolve and save the file.

Go back to the Networker Bare Metal Recovery Wizard. You should be on the Select Server page. Enter the name of the Networker server you are restoring data from and click Next.

If you entered incorrect network settings in the previous step then this lookup will fail. If the client is able to connect then you will be asked to select the backup you want to restore from.

restore4

Confirm the partitions to be restored on the next page and click Next.

Double check the information on the summary page and start the restore. Again note the warning that data on the disk(s) will be overwritten.

restore5

When the restore has completed the results page will appear. If the result was successful then click Reboot, the server will be restarted and boot to the recovered operating system.

restore6