How to Migrate VMware Virtual Machines to VMware Cloud on AWS

This post pulls together the workload migration planning and lessons learned notes made during a real-life customer use case of evacuation an on-premise data centre to VMware Cloud (VMC) on AWS (Amazon Web Services). The content is a work in progress and intended as a generic list of considerations and useful links for both VMware and AWS, it is not a comprehensive guide. Cloud, more-so than traditional infrastructure, is continuously changing. Features are implemented regularly and transparently so always validate against official documentation. This post was last updated on September 16th 2019.

Part 1: SDDC Deployment

Part 2: Migration Planning & Lessons Learned

See Also: VMware Cloud on AWS Security One Stop Shop

1. Virtual Machine Migrations

The following points should help with the planning of Virtual Machine (VM) workload migrations to VMware Cloud on AWS. An assumption is made that the Software-Defined Data Centre (SDDC) is stood up and operational with monitoring, backups, Anti-Virus, etc. in place. Review Part 1: SDDC Deployment for more information. I found the SDDC deployment and getting the environment available was the easy part. Internal processes and complexity of the existing environment are going to determine how quickly you can migrate workloads to the SDDC.

We started by exporting a list of Virtual Machines from each vCenter, from that we identified the service it was running and the service owner or a business owner. The biggest surprise here was the number of servers deployed by, or for, people who had left the organisation. These servers were still being hosted, maintained, patched, but no longer needed. We were able to decommission more workloads than expected due to years of VM sprawl. While VMware Cloud on AWS isn’t directly responsible for this, the project forced us to evaluate each server we hosted. For remaining workloads, we put together a migration flow which identified the following criteria:

  • CPU, RAM, storage requirements: specified a baseline to automatically accept and then anything above our baseline would require a manual check.
  • Network dependencies: is there a large amount of data in transit, is IP retention required, is the VLAN stretched using Hybrid Cloud Extension (HCX), load balancer requirements.
  • Data flows: used vRealize Network Insight to identify potential egress costs and additional service dependencies.
  • Additional application or organisation specific considerations: e.g. data classification, tagging / charge-back model, backups, security, monitoring, DNS, authentication, licensing or support.
  • Service Management considerations: is the service platinum/gold/silver/bronze or unclassified, do the platform Service Level Agreements (SLAs) fulfil the existing SLAs in place for each service, is the proposed migration type (i.e. the amount of downtime) taking this into consideration. Involving Service Management right from the start was useful as they were able to advise on internal processes for Service Acceptance and Business Continuity.
  • Service Owner considerations: if the technical criteria above are met then the next step was to meet with service owners and get their buy-in for the migration. We migrated internal services we owned first and then used that as a success story to onboard other services. This process involved meeting with various departments, presenting the solution and the benefits over their existing hosting, in our case DR and performance improvements, and migrating dev or test workloads first to build confidence.
  • Migration passport: one of our Senior Engineers came up with this concept as a one-pager for each service that was migrated, it consisted of migration details (change ID, date, status), migration scope (server names, locations, and notes), firewall rules, vRNI outputs, and other information such as associated documentation.

Each environment is different, so these are provided as example considerations only. Use resources such as those outlined below, and, to develop your own migration strategy.

Workload_Mobility

2. Network Design

  • Updated Feb 2020 – see also AWS Native Services Integration With VMware Cloud on AWS
  • Research the differences and limitations around the different VMware on AWS connection types, especially under 1Gbps – Configuring AWS Direct Connect with VMware Cloud on AWS
  • Make sure you understand the terminology around a Virtual Interface (VIF) and the difference between a Standard VIF, Hosted VIF, and Hosted Connection: What’s the difference between a hosted virtual interface (VIF) and a hosted connection? It is important to consider that VMware Cloud on AWS requires a dedicated Virtual Interface (VIF) – or a pair of VIFs for resilience. If you have a standard 1Gbps or 10Gbps connection direct from Amazon then you can create and allocate VIFs for this purpose. If you are using a hosted connection from an Amazon Partner Network (APN) for sub-1G connectivity then you may need to procure additional VIFs, or a dedicated Direct Connect with the ability to have multiple VIFs on a single circuit. This is a discussion you should have with your APN partner.
  • The Virtual Private Cloud (VPC) provided by the shadow AWS account cannot be used as a transit VPC. In other words, if you want to connect to private IP addressing of native AWS services, you cannot hop via VMware Cloud. In this instance, a Transit Gateway can be used.
  • At the time of writing a VPN attachment must be created to connect the SDDC to a Transit Gateway, if Direct Connect is in use, then the minimum requirement is 1Gbps.
  • If there is a requirement to connect multiple existing AWS VPCs, or multiple SDDCs, with on-premise networks then definitely check out VMware Cloud on AWS with Transit Gateway Demo.
  • If a backup VPN is in use, then you may be able to reduce failover time using Bidirectional Forwarding Detection (BFD) which is automatically enabled by AWS, in our case, it was not supported by our third-party provider.
  • Use vRealize Network Insight to get an idea of dependencies and data flows that you can use to plan firewall rules and estimate egress or cross-AZ charges. In general, my experience with these charges is that they have been minimal, this depends entirely on your own environment but should be considered when calculating overall VMware on AWS pricing.
  • If you want to update your default route see How to Set the Default Route in VMware Cloud on AWS: Part 1 & Part 2.
  • VMware Cloud on AWS: NSX Networking and Security eBook

3. Load Balancing & Security

  • Update Feb 2020 – see also VMware Cloud on AWS Security One Stop Shop
  • With the acquisition of Avi Networks, we can expect Avi Networks services as a paid add-on for VMware Cloud: VMware Cloud on AWS: NSX and Avi Networks Load Balancing and Security.
  • Third-party load balancers such as virtual F5 can be deployed in a virtual appliance format. If you are planning on using AWS Elastic Load Balancer (ELB) on a private IP address accessible on-premise ensure you have a connectivity method as outlined above.
  • The NSX Distributed Firewall (DFW) feature is included in the price of VMware Cloud, the paid-for message is removed from SDDC v1.8 onwards, this was announced at VMworld 2019.
  • Another VMworld 2019 announcement was the inclusion of syslog forwarding with the free version of VMware Cloud Log Intelligence (SaaS offering for log analytics). However, for troubleshooting NSX DFW logs you still need the paid-for version.
  • If you are using HCX, this product uses its own IPSec tunnel and therefore we could not get it working with the private IP address over a backup VPN. It was assumed that HCX would also not work with the private IP address via Transit Gateway either, due to the SDDC VPN requirement, and would need to be reconfigured to use the public IP address.
  • Another HCX migration consideration is that when you are stretching a network, all traffic goes via the HCX Interconnects. This means you are encapsulating everything in port UDP 4500, and essentially bypassing your on-premise firewall rules while the network is stretched. It is essential to double-check all rules are correct before eventually moving the gateway to VMC.
  • Again if you are doing VMware HCX migrations, remember to remove stretched networks once complete. This involves shutting down the gateway on-premise, removing the L2 stretch, and changing the network in the SDDC to routed, for us the downtime was around 30 seconds. The deployment of HCX in our environment, although covered by vSphere High Availability (HA), didn’t have resilience built-in; therefore we decided to minimise the amount of time they were in use by planning a migration strategy around each subnet.
  • If you use NSX Service Deployments for Anti-Virus, i.e. Guest Introspection for agentless AV then you will need to deploy an agent on each VM, as this feature is still currently unavailable.

4. General

  • The Cloud Services Portal (CSP) can be integrated with enterprise federation, allowing you to control access using your organisational policies, hopefully, therefore, enforcing Multi-Factor Authentication (MFA) and removing access as part of a leavers process. Federation will only work with a tenant, it will not work with a master organisation.
  • It is not possible at the time of writing to easily transfer an SDDC deployed in the root/master organisation into a tenant. The process currently is a redeploy and migrate.
  • Druva offers a product that will backup Virtual Machines from VMware Public Cloud direct into an S3 bucket they manage, for a greenfield deployment if you are not transferring any existing licenses this could be a good option as you only pay for the capacity you use. Having a backup environment setup in AWS has many benefits but also adds a management overhead and the consideration of replicating between Availability Zones.
  • In general internal support was good once teams were educated on the platform and the slightly different operating model we were implementing. In terms of external support, we have not encountered any compatibility issues yet. There was one application vendor with a published KB article stating they support running the application on VMware Cloud on AWS, who then retracted support stating the vSphere version being run was not GA.

Watch a Failover from Direct Connect to Backup VPN for VMware Cloud on AWS

This post demonstrates a simulated failure of Amazon Direct Connect, with VMware Cloud (VMC) on Amazon Web Services (AWS). In this setup, the standby VPN has been configured to provide connectivity in the event of a Direct Connect failure. The environment consists of a 6 host stretched cluster in the eu-west-2 (London) region, across Availability Zones eu-west-2a and eu-west-2b.

In this instance, a pair of hosted private Virtual Interfaces (VIFs) are provided by a Cloud Connect service from a single third-party provider. A Route-Based VPN has been configured. Direct Connect with VPN as standby was introduced in SDDC v1.7. For more information, see Nico Vibert’s post here.

AWS Direct Connect: “Using AWS Direct Connect, you can establish private connectivity between AWS and your datacenter, office, or colocation environment, which in many cases can reduce your network costs, increase bandwidth throughput, and provide a more consistent network experience than Internet-based connections.”

AWS VPN: “AWS Virtual Private Network (AWS VPN) lets you establish a secure and private tunnel from your network or device to the AWS global network.”

DX_VPN_Setup

Direct Connect Outage

Before beginning, it is worth re-iterating that the following screenshots do not represent a process. Providing the backup VPN is configured correctly, then the customer/consumer of the service does not need to intervene; in the event of a real-world outage, everything highlighted below happens automatically. You may also want to review further reading: How to Deploy and Configure VMware Cloud on AWS (Part 1), How to Migrate VMware Virtual Machines to VMware Cloud on AWS (Part 2), plus additional demo post Watch VMware vSphere HA Recover Virtual Machines Across AWS Availability Zones.

Taking down the primary and secondary VIFs was carried out by the hosting third party, to help with providing evidence of network resilience. When we start out in this particular environment, the VIFs are attached and available. Servers in VMware Cloud are contactable from on-premise across the Direct Connect. The backup VPN is enabled.

DX_VPN_1DX_VPN_2

Following disabling of the interfaces by our third-party provider, the BGP and Direct Connect status changes to down.

DX_VPN_3DX_VPN_5

This is confirmed in the AWS console as both the BGP status and therefore the VIF state is down.

DX_VPN_4

With the Direct Connect down routes are redistributed using the backup VPN. The Direct Connect BGP hold timer is 90 seconds, and the BGP keepalive is 30 seconds. After 90 seconds, the VIF(s) BGP hold time expires, and traffic starts to flow through the VPN connection.

In the screenshot below you can see an on-premise monitoring solution reporting on a server hosted in VMware Cloud on AWS. The server is available over the Direct Connect, drops, and is then available over the backup VPN after we disable the interfaces to simulate a failure. The test was conducted twice.

VPN_Monitor

How VMware is Accelerating NHS Cloud Adoption

This post provides an overview of how the UK National Health Service (NHS) can unlock the benefits of cloud computing with VMware Cloud (VMC) on Amazon Web Services (AWS).

Part 1: How VMware is Accelerating NHS Cloud Adoption

Part 2: Bridging the Gap Between NHS and Public Cloud with VMware Cloud on AWS

In November 2014 the National Information Board and Department of Health and Social Care published the Personalised Health Care 2020 paper, outlining a framework to support the NHS with making better use of data and technology to improve health and care services. The paper endorsed the use of digital cloud services, backing the UK Government cloud first strategy, introduced in 2013. In January 2018 NHS Digital released guidance for NHS and social care data: off-shoring and the use of public cloud services, along with a toolset for identifying and assessing data risk classification. The NHS and social care data: off-shoring and the use of public cloud services guidance paper published by NHS Digital states; ‘NHS and social care organisations can safely put health and care data, including non-personal data and confidential patient information, into the public cloud’. The NHS and social care providers may use cloud computing services for NHS data, providing it is hosted in the UK, or European Economic Area (EEA), or in the US where covered by Privacy Shield.

The Information Governance (IG) report for Amazon Web Services was updated in 2018, the score approves Amazon Web Services to host and process NHS patient data. VMware Cloud on AWS leverages Amazon’s infrastructure to provide an integrated cloud offering, delivering a highly scaleable and secure solution for NHS trusts and other organisations to migrate workloads and extend their on-premise infrastructure. Steps for understanding the data type, assessing migration risks, and implementing and monitoring data protection controls are also included in the documentation linked above. Each individual data controller organisation is responsible for implementing and reviewing their own processes around data risk classifications, however to assist NHS Digital have provided a consistent health and social care data risk model. For organisations that do not yet have cloud governance in place NHS Digital have also provided guidance on the health and social care cloud risk framework.

Cloud services introduce a shared security model. NHS organisations can be compliant by implementing a cloud risk framework and proportionate controls outlined by NHS Digital; summarised in the health and social care cloud security one page overview. Security considerations for different data classifications are detailed in the health and social care cloud security – good practice guide.

The NHS can implement Secure by Design services with VMware Cloud on AWS

  • NHS organisations must be aware of the shared security model that exists between: VMware; delivering the service, Amazon Web Services (the IaaS provider); delivering the underlying infrastructure, and customers; consuming the service.
  • The NHS organisation is in complete control of the location of its data. VMware do not backup or archive customer data and therefore it is up to the NHS organisation to implement this functionality.
  • Micro-segmentation can be used to protect applications by ring-fencing virtual machines in a zero trust architecture. The risks of legacy operating systems can be mitigated by isolating them from the rest of the network. Micro-segmentation is included for all virtual machines in the default VMware on AWS pricing model.
  • NHS organisations can use Role Based Access Control (RBAC) and Multi-Factor Authentication (MFA) to control access to cloud resources. NHS organisations are in control of inbound and outbound firewall rules and can opt to route all traffic internally on private addressing.
  • VMware Cloud on AWS meets a number of security standards such as NIST, ISO, and CIS. Standard Amazon policies for physical security and secure disposal apply. Amazon use self-encrypting disks and manage the keys using Amazon Key Management Service (KMS).
  • VMware implement a number of stringent security controls, for example MFA generated time-based credentials for support staff; all logged and monitored by a Security Operations Centre (SOC), VSAN based encryption, and industry-leading commercial solutions to secure, store, and control access to tokens, secrets, passwords, etc.
  • Full details can be found at: VMware Cloud on AWS Security One Stop Shop

Additional benefits of VMware Cloud on AWS to the wider NHS, are as follows:

The NHS can save time and money by reducing physical or data centre footprint

  • NHS Digital reached an agreement in May 2019 to offer other NHS organisations discounted access to cloud services, such as favourable VMware on AWS pricing, to help accelerate their journey to the cloud. In addition, a favourable pricing structure is in place for reserved instances should organisations commit for 1 or 3 years.
  • Commissioning new space in a data centre, or even just new hardware, can be a lengthy process. With VMware Cloud an entire virtual data centre can be deployed in around 90 minutes. Extending capacity on demand takes as little as 15 minutes.

  • Existing VMware Virtual Machines (VMs) can be migrated to VMware Cloud on AWS, and back if needed, in minutes without the need to refactor applications.
  • NHS technical staff continue to use the same tools and management capabilities that they currently use day to day.
  • In most cases where products such as Monitoring, Backups, and Anti-Virus, are licensed per physical host or per number of VMs organisations can adopt a Bring Your Own Licensing (BYOL) approach.

  • VSAN replication and stretched networks can enhance Disaster Recover (DR) capabilities. The Stretched-Cluster deployment provides vSphere High Availability (HA) across 2 Amazon Availability Zones within a region with a 99.99% availability commitment. Additional DR services such as Site Recovery Manager (SRM) add-ons are also available.
  • In many cases replacing aging servers and storage infrastructure with the latest hardware and flash based VSAN can yield significant application performance benefits.
  • Physical host capacity can be scaled out dynamically and then back in when it is no longer required. NHS organisations can take advantage of easily spinning up environments to test or develop without having to manually install and configure additional hardware.

  • VMware Cloud on AWS has a private link into Amazon’s backbone network of services, ranging from storage, database, and network services, to Internet of Things (IoT), Artificial Intelligence (AI) and Machine Learning. Developers can take advantage of various managed container services, or serverless platforms.
  • Since VMware Cloud resides in Amazon’s data centres hybrid configurations can be securely implemented, for example using Amazon’s Elastic Load Balancer with the back end servers in VMC, or Amazon’s Relational Database Service with the application servers in VMC.
  • Full details can be found at AWS Native Services Integration With VMware Cloud on AWS

  • Hardware maintenance such as firmware updates, failure remediation, and upgrades are all handled by VMware, as are software updates to the hypervisor and infrastructure management layer.
  • NHS technical staff are responsible for securing applications inside the virtual machine, e.g. operating system updates and firewall configuration, ensuring that Amazon Secure by Design best practises are followed.
  • In summary VMware Cloud on AWS enables NHS organisations to seamlessly extend or migrate data centre workloads to the cloud, whilst enhancing security and availability options. In the example shown below an existing VMware vSphere environment has been extended to VMware Cloud on AWS, giving organisations the flexibility to run their workloads on the most suited platform. This approach is secure and easy for operational teams who may not yet have an established cloud governance process in place.

Additional notes on this design: The Internet Gateway for VMC is not in use, all routes are advertised internally and controlled using on-premise firewalls, in other words all ingress and egress traffic is via the on-premise data centres. Access to native AWS services uses the 25Gbps Elastic Network Interfaces (ENI) and is secured using the gateway firewall and Amazon Security Groups.

NHS_SDDC

VMware Cloud on AWS FAQs | Resources | Documentation | Factbook | Evaluation Guide | On-Boarding Handbook | Operating Principles