Tag Archives: VMware to AWS Migration

This post will walk through the example design below; building out the Amazon Web Services (AWS) framework enabling VMware Cloud on AWS customers to start using AWS at scale, alongside VMware workloads. The key focus will be around the control of multiple accounts, using AWS Organizations and Service Control Policies, and cross-account connectivity, with Transit Gateway and the role of the VMware Cloud-connected Virtual Private Cloud (VPC).

Example VMC AWS Setup

To enlarge the image right click and select open image in new tab.

VMware Cloud on AWS Focus

This article assumes you already have a working knowledge of VMware Cloud on AWS and have either deployed or are planning the deployment of, your Software-Defined Data Centre (SDDC). If you are unclear about the requirements for the connected AWS account and VPC review the VMware Cloud documentation here.

In the example architecture, we are working on, a Stretched Cluster has been deployed in the eu-west-2 (London) region. During the SDDC deployment, I connected an existing AWS account, aws.sddc@myorg.com, and now have a 25 Gbps cross-VPC link between VMware Cloud and my own VPC using the Elastic Network Interface (ENI). More information on how the connected VPC works can be found in AWS Native Services Integration With VMware Cloud on AWS.

VMCFocus

In this setup, I have also configured some Elastic Compute Cloud (EC2) instances to back up my Virtual Machines (VMs) to Simple Storage Services (S3). Great, so how do I start deploying AWS services at scale, and onboard the rest of my business that wants to begin creating their own AWS accounts?

AWS Organizations & Service Control Policies

AWS Organizations is a good starting point for those wanting to implement policies and governance across multiple accounts, for compliance, security, and standardised environments. Organizations can consolidate billing for your accounts, and automate the creation of new accounts as your environments grow. There is no additional charge for using AWS Organizations or Service Control Policies (SCP). An AWS Organization can be deployed manually, or as part of a Landing Zone which is discussed in the next section.

First, log into the AWS console with the account you will assign as the master. This account is used for account and policy management (it is itself exempt from Service Control Policies we will cover shortly), and assumes the role of a payer account for charges accrued by accounts within the organization hierarchy. Once the master account is set, it cannot be changed.

From the Services drop-down locate AWS Organizations, click Create Organization. Name your organization and select either consolidated billing only features or all features. From the Accounts tab, you can create and manage AWS accounts, and add existing accounts to your organization. A member, and master, account can only be a member of one organization. When you create an account with AWS Organizations, an Identity and Access Management (IAM) role is created with full administrative permissions in the new account. The master account can assume this IAM role if required to gain access.

The Organize Accounts tab is where you can start creating the hierarchy of Organizational Units (OU). The canvas starts with the top-most container, which is the administrative root. OUs, and nested OUs (up to 5 levels deep including root), are added for separate groupings of departments or business units allowing policies to be applied to groups of accounts. An OU will inherit policies from the parent OU in addition to any policies assigned directly to it.

Organization1

A Service Control Policy contains statements defining the controls that are applied to an account or group of accounts. SCPs can only be used for organizations created with all features enabled, they are not available with consolidated billing only. Multiple SCPs can be attached or inherited to accounts in the hierarchical OU chain, however, a deny will always override any allow policies. SCPs can be created and managed in the Policies tab.

A default FullAWSAccess policy exists an is attached to the organization root allowing access to any operation. In this example, I have created a DenyInternet policy to be applied to my DataOps OU, who have a requirement to analyse sensitive data from data sets running in VMware Cloud. The SCP is a JSON policy that specifies the maximum available permission for accounts or grouping of accounts (OU) that the policy is attached to. You can write the JSON out yourself or use the statement filter on the left-hand side.

Organization3

Once the policy is created, I attach it to the relevant OU, where it is instantly applied to any member accounts residing in that particular OU. I attach the policy either from the Policies tab, or directly on the account, OU, or organization root.

Organization2

Now, when logging in with the user account, I am unable to create an Internet Gateway as defined in the SCP statement. For more information on Service Control Policies review the Service Control Policies User Guide which details example statements, relationship with IAM permissions, and scenarios where SCPs would not apply, such as resource-based policies, and users or roles outside of the organization.

IGWDeny

Outside of the master account, my AWS hierarchy now looks like this. With a repeatable process in place for members of the DataOps team to create new accounts which do not have internet access. Furthermore, I may want to create some root policies to limit the tampering of AWS security tools such as CloudTrail, GuardDuty, and Config. You can read more about these services in the next section.

AWSOrg

Additional Baseline AWS Services

To protect the AWS Organization, I can look to implement a security baseline across all my accounts, using central management of the services outlined below. These tools can be implemented individually or automated as part of AWS Landing Zone. For VMware Cloud, the connected AWS account that I have full control over can fall into the remit of these services, my organizational hierarchy, and Service Control Policies. However, remember that the SDDC environment is deployed to a shadow AWS account that the customer does not have access to, and this means that we need to utilise Log Insight Cloud to capture and analyse any syslog output from vCenter, NSX-T, etc. Log Insight Cloud can also pull logs from AWS as a log source, from services like CloudTrail and CloudWatch. You can read more about VMware Cloud security measures in VMware Cloud on AWS Security One Stop Shop.

IAM is a mechanism by which we can manage, control, and govern authentication, authorisation, and access to resources within your AWS account. For administrators overseeing multiple accounts, IAM can help with enforcing password policies, Multi-Factor Authentication (MFA), and Identity Federation or Single Sign-On. IAM policies can be applied to users, groups, or roles.

CloudTrail records and tracks all Application Programming Interface (API) requests in an AWS account. Each API request is captured as an event, containing associated metadata such as caller identity, timestamp, and source IP. The event is recorded and stored as a log file in an S3 bucket, with custom retention periods, and optional delivery to CloudWatch Logs for metric monitoring and alerting. CloudTrail logs for multiple accounts can be stored in a central encrypted S3 bucket for effective auditing and security analysis.

GuardDuty is a regional-based intelligent threat detection service that monitors for unusual behaviour from CloudTrail event logs, VPC flow logs, and DNS logs. Logs are assessed against multiple security feeds for anomalies and known malicious sources. GuardDuty can provide continuous security analysis, powered by machine learning, for your entire AWS environment across multiple-accounts. GuardDuty findings are presented in a dashboard with priority level and severity score and integrate with other services such as CloudWatch and Lambda for remediation automation.

Config can record and capture resource changes in your environment for Configuration Items, detail resource relationships, store configuration history, provide a snapshot of configurations, act as a resource inventory for AWS resources, and allow you to check the compliance of those resources against pre-defined and custom rules. Config can enable notifications of changes, as well as detailing who made the change and when, by integrating with CloudTrail. When coupled with rules like encryption checks, Config can become a powerful security analysis tool.

The Security Pillar White Paper of the AWS Well-Architected Framework is worth reviewing as a starting point to these services.

AWS Control Tower & Landing Zone

AWS Control Tower is a paid-for option for customers who want to quickly setup and govern new AWS environments based on AWS best practices, for those with multiple and distributed applications that will span many accounts. AWS Control Tower consists of established blueprints allowing for automated setup and configuration of your multi-account AWS environments and Identity Federation. Account Factory automates and standardises account provisioning from a configurable account template, with pre-approved network and region settings. Guardrails prevent resources from being deployed that do not conform to policies and detect and remediate non-compliant accounts and resources. Control Tower dashboards provide visual summaries to monitor security and compliance across the organization.

One of the components included with Control Tower is AWS Landing Zone. A Landing Zone can also be implemented yourself outside of Control Tower, it deploys a multi-account AWS environment based on AWS well-architected and security and compliance best practices. The Landing Zone deployment is made up of 4 accounts for AWS Organization & Single Sign-On (SSO), shared infrastructure services, log archive, and security. The good thing about AWS Landing Zone is it provides a security baseline for several security services and settings, you can see the full list here. Once again, you can create these accounts and services yourself manually if there is a need for greater customisation or granular control, however doing so is time-consuming.

SDDC Cross-Account AWS Connectivity

Not having a Landing Zone or Organization and account structure in place does not stop or delay the VMware Cloud on AWS deployment. For example, you can still create the connected AWS account, and your own central shared services or network account, if it is appropriate to your design, and retrospectively fit these accounts into the Organization hierarchy.

In the setup below, the connected AWS VPC has been reserved for SDDC operations only, in this case, VM backups. The SDDC router is connected to this VPC / account using the subnets defined in the ENI configuration at deployment, meaning backups will run over the 25 Gbps cross-VPC link with no additional data charges. Further services can be deployed to this account, but as the number of AWS services and environments (prod, dev, test, etc.) start to scale, it is good practice to use separate accounts. This is where the Transit Gateway and centralised shared network account can help.

SDDCConnection

The Transit Gateway (TGW) allows customers to connect many VPCs (including the SDDC), and on-premises networks to a single gateway. In the example architecture, we have the following, which provides connectivity between VMware Cloud on AWS, multiple VPCs and accounts, and the on-premises data centre, using a central shared network services model:

  • Direct Connect has been attached to the TGW using a Direct Connect Gateway, you can read how here.
  • VMware Cloud on AWS has been connected to the TGW using a VPN attachment. The VPN needs setting up in the Cloud Services Portal, you can read how here. Note that to my knowledge using this model in conjunction with HCX L2 extension may not be supported end to end.
  • Additional VPCs are connected to the TGW using VPC attachments, you can read how here.

Outside of VMware cloud, VPC Peering has traditionally been used to provide one to one private network connectivity between VPCs, including VPCs in different accounts. VPC Peering cannot be configured in the SDDC as we do not have access to the underlying AWS account. If it is unlikely that the VMware Cloud customer will be a heavy user of native AWS services, then using a TGW may be overkill, and the SDDC connected VPC may suffice.

For small environments, a VPN connection between additional VPCs can be configured on a one-to-one basis from the VMware Cloud Services Portal. However, as the number of VPCs and accounts begins to scale the VPN approach becomes harder to manage. VPC endpoints can also be used for targeted access to service-by-service resources in other accounts, you can see examples of this at AWS Native Services Integration With VMware Cloud on AWS.

In any case, when connecting VPCs and networks together, it is essential to remember that you should not have overlapping IP address ranges. This is relatively easy to plan for greenfield AWS environments but may need further consideration when connecting your existing on-premises networks.

Now, when we pull the shared network and account management together, we start to have the basis for the DataOps team to deploy their own AWS services with cross-environment access, governed by organizational policy and control. This post was intended as a high-level view on account and network management for VMware Cloud on AWS design integration with native AWS. Allowing connectivity into your SDDC requires correct firewall configuration, you can view examples at Connecting VMware Cloud on AWS to Amazon EC2.

Example VMC AWS Setup

AWS Native Services Integration With VMware Cloud on AWS

This post lists some of the available options for connecting VMware Cloud on Amazon Web Services (AWS) with native AWS services for hybrid cloud deployments. Read more about multi-account and VPC management at Building AWS Environments for VMware Cloud Customers.

The most common way of consuming AWS services with VMware Cloud on AWS is to use the built-in Elastic Network Interface (ENI) functionality. Each VMware Cloud Software-Defined Data Center (SDDC) can be connected to another AWS Virtual Private Cloud (VPC) during the deployment phase. A VPC is Amazon’s logical separation of virtual networks. At scale, you may choose to have many VPCs and many accounts for different applications and environments. Multiple VPCs can be connected together using an AWS Transit Gateway (TGW). A further option we will look at is VPC Endpoints, enabling you to privately connect to supported AWS services and endpoints.

1. Connected VPC

The AWS bare metal hosts deployed for VMware Cloud on AWS use a redundant 25 Gbps physical interface or Elastic Network Adaptor (ENA). The physical interface uses a trunk port to carry multiple VLANs for services like management, vMotion, NSX, and connectivity to the AWS backbone network.

The cross-linked VPC architecture is provided by a series of ENIs. Each host in the vSphere cluster uses the network adaptor outlined above to provide an individual cross-VPC ENI per physical host; supporting high-bandwidth, low latency connectivity to native AWS services.

VMC_Host_Connectivity

VMware Cloud on AWS uses the AWS VPC  as an underlay for NSX-T. The NSX Edge (Tier 0) router is a virtual router acting as the uplink to the connected VPC. The active ENI in use is the physical ESXi host where the virtual router is running. The connected VPC is owned and managed by the customer, any native services deployed are billed separately by AWS. When deploying the SDDC, the connected account and VPC is required along with a private subnet in each applicable Availability Zone (AZ). A static route is created for the defined subnets adding the connected VPC router as the next hop.

Traffic that traverses the ENI is not chargeable; however, cross-AZ charges do need to be taken into consideration if a Stretched Cluster is in use. During provisioning of the SDDC, and connection of the customer-managed AWS account, a CloudFormation template is deployed creating the necessary AWS Identity Access Management (IAM) roles and ENI configuration.

SDDC_AWS_1SDDC_AWS_2

Following the SDDC deployment, you can view the connected account, VPC, ENI, and subnet details in the Connected VPC menu under the Networking & Security tab of the SDDC, from the VMware Cloud Services Portal.

Access to and from native AWS services can be controlled and needs to be opened, using the NSX firewalls (gateway and distributed) and AWS Security Groups. To see an example configuration see the Connecting VMware Cloud on AWS to Amazon EC2 post or the Access an EC2 Instance section of the VMware Cloud on AWS Docs page.

2. VPC Endpoints

VPC Endpoints allow private connectivity between your VPC and supported AWS services or custom applications. Network traffic traversing a VPC endpoint does not leave the AWS backbone network and therefore does not require Internet Gateway, Direct Connect, or VPN.

The Access an S3 Bucket Using an S3 Endpoint section of the VMware Cloud on AWS Docs page details the process for configuring a Gateway VPC Endpoint to access AWS Simple Storage Services (S3) from VMware Cloud on AWS, without having to go out to the Internet. Furthermore, you can use Interface VPC Endpoints to connect to supported AWS services in another VPC, or VPC Endpoint Services (AWS PrivateLink) to connect to custom applications in another VPC. Here are some examples:

AWS_Integration_Examples

The general process for creating an endpoint is the same across these VPC Endpoint types. In the example below, we are connecting to a VPC Endpoint Service for Splunk, fronted by a Network Load Balancer (NLB) in another VPC. The administrator of the VPC Endpoint Service needs to grant IAM service consumer permissions and accept the incoming connection, as detailed in the AWS documentation here.

In the AWS console, I log into the connected account and select the VPC service. I choose Endpoints and Create Endpoint. To create a Gateway VPC Endpoint, e.g. for S3, or an Interface VPC Endpoint, e.g. for DynamoDB or other services, I would select the appropriate service from the AWS services service category. In this instance, I use Find service by name and enter the endpoint private service name. Either way, the key point is that I select the connected VPC from the VPC drop-down and the subnets that match up with those used for the ENI when deploying the VMware Cloud on AWS SDDC.

VPCEndpoint1VPCEndpoint1cont

By using the cross-VPC linked subnets the Virtual Machines in the SDDC will utilise the static route across the ENI outlined in the Connected VPC section above. AWS Security Groups can be used to limit this to certain source IP addresses from within the SDDC or the wider VPC if required. In this instance, we can successfully test the connection over port 443 following the creation of the VPC Endpoint.

3. Additional VPC Connectivity

Traditionally VPC Peering has been used to provide one to one private network connectivity between VPCs, including VPCs in different accounts. VPC Peering cannot be configured in the SDDC as we do not have access to the underlying AWS account. VPN connections between additional VPCs and the SDDC router (Tier 0) can be configured from the VMware Cloud Services Portal, enabling VMware Cloud on AWS connectivity with other VPC environments. As the number of VPCs and accounts begins to scale the VPN approach becomes harder to manage.

This predicament is resolved with a relatively new addition to AWS; the Transit Gateway (TGW). The native AWS TGW is available now and acts as a transit network hub allowing you to connect multiple VPCs and on-premise networks (using Direct Connect or VPN attachments). A Managed Transit Gateway is being developed by VMware to assist with multi-SDDC and multi-VPC connectivity. You can review how the native AWS Transit Gateway fits into the VMware Cloud on AWS architecture on the VMware Network Virtualization blog: VMware Cloud on AWS with Transit Gateway Demo:

VMC_TGW_Examples

Image VMware Cloud on AWS with Transit Gateway Demo. Further Resources:

VMware Cloud on AWS Security One Stop Shop

Kicking off 2020 with the theme of the year – security.

To keep this content accurate, the bulk of it has been quoted from the Cloud Security Alliance (CSA) VMware Cloud on Amazon Web Services (AWS) self-assessment. The CSA defines best practises for secure cloud computing environments. The full assessment can be found here under VMware Cloud on AWS. I have listed the key points from the CSA applicable to our customer use case below, any direct quotes are in blue text, alongside further information from the following must-read resources:

Introduction

This post is arranged into the following sections:

  • Introduction
  • Roles & Responsibilities
  • Physical Security
  • Security Operations Monitoring
  • Penetration Testing & Audit
  • Data Accessibility
  • Data Encryption

It is important to understand the service to be able to secure it appropriately. Review the VMware Cloud on AWS Service Description, where page 3/11 states:

VMware Cloud on AWS (the Service Offering or VMware Cloud) brings VMware’s
enterprise class Software-Defined Data Center software to the Amazon Web Services cloud, enabling customers to run any application across vSphere-based private, public, and hybrid cloud environments.

The Service Offering has the following components:

  • Software-Defined Data Center (SDDC) consisting of:
    • VMware vSphere running on elastic bare metal hosts deployed in AWS
    • VMware vCenter Server appliance
    • VMware NSX Data Center to power networking for the Service Offering
    • VMware vSAN aggregating host-based storage into a shared datastore
    • VMware HCX enabling app mobility and infrastructure hybridity
  • Self-service provisioning of SDDCs, on demand, from vmc.vmware.com
  • Maintenance, patching, and upgrades of the SDDC, performed by VMware

The SDDC service offering uses dedicated AWS physical hardware per tenant, each ESXi host you purchase is a dedicated physical AWS bare metal server. Each customer environment is logically and physically separated, there is no multi-tenancy or nested virtualisation. The customer is in charge of their own workloads as well as ingress/egress and user access.

This post focuses on the security of the VMware Cloud on AWS platform and collates information provided by VMware. The network design is a topic that requires addressing in its own right; connectivity, default route, Internet access, firewall and load balancing, etc. To ensure you secure the network along with user access and workloads, as outlined in the next section, review in full the above documentation, the VMware  Cloud on AWS Documentation, and Reference Architectures, as well as engaging your VMware customer success or account team.

More detail on planning the SDDC deployment can be found in How to Deploy and Configure VMware Cloud on AWSHow to Migrate VMware Virtual Machines to VMware Cloud on AWS. The following additional reading may also be of use: AWS Security by DesignSecurity, Identity, and Compliance on AWS, Humair’s BlogVMware Network Virtualisation & VMware Cloud Blog.

The deployment of any native AWS services should follow best practices outlined in the Security Pillar White Paper of the AWS Well-Architected Framework. VMware Cloud on AWS can make up part of a more comprehensive cloud framework, read more about multi-account and VPC management at Building AWS Environments for VMware Cloud Customers.

Roles & Responsibilities

Although VMware Cloud on AWS utilises Infrastructure as a Service (IaaS) from AWS, the customer consumes the platform as a whole from VMware, and therefore maintains a relationship and support contract with VMware. Support for any native AWS services deployed in the customers connected AWS Virtual Private Cloud (VPC) remains with AWS as normal. The security model, therefore, is shared between the customer and VMware. VMware separates the roles as follows:

We (VMware) will use commercially reasonable efforts to provide:

  • Information Security: We will protect the information systems used to deliver the
    Service Offering over which we (as between VMware and you) have sole administrative level control.
  • Security Monitoring: We will monitor for security events involving the underlying
    infrastructure servers, storage, networks, and information systems used in the delivery of the Service Offering over which we (as between VMware and you) have sole administrative level control. This responsibility stops at any point where you have some control, permission, or access to modify an aspect of the Service Offering.
  • Patching and Vulnerability Management: We will maintain the systems we use to
    deliver the Service Offering, including the application of patches we deem critical for the target systems. We will perform routine vulnerability scans to surface critical risk areas for the systems we use to deliver the Service Offering. Critical vulnerabilities will be addressed in a timely manner.

You (the customer) are responsible for addressing the following:

  • Information Security: You are responsible for ensuring adequate protection of the
    Content that you deploy and/or access with the Service Offering. This includes, but is not limited to, any level of virtual machine patching, security fixes, data encryption, access controls, roles and permissions granted to your internal, external, or third party users, etc.
  • Network Security: You are responsible for the security of the networks over which you have administrative level control. This includes, but is not limited to, maintaining effective firewall rules in all SDDCs that you deploy in the Service Offering.
  • Security Monitoring: You are responsible for the detection, classification, and remediation of all security events that are isolated with your deployed SDDCs, associated with virtual machines, operating systems, applications, data, or content surfaced through vulnerability scanning tools, or required for a compliance or certification program in which you are required to participate, and which are not serviced under another VMware security program.

In a nutshell, as well as user access and connectivity, the customer is ultimately responsible for what is inside the Virtual Machine. This means things like Anti-Virus, operating system and application patches, monitoring, backups, access control / privileged access, etc. The VMware Cloud Service Offerings Terms of Service backs this up, stating:

2.2 You (the customer) are responsible for taking and maintaining appropriate steps to protect the confidentiality, integrity, and security of Your Content. Those steps include (a) controlling access you provide to your Users, (b) configuring the Service Offering appropriately, (c) ensuring the security of Your Content while it is in transit to and from the Service Offering, (d) using encryption technology to protect Your Content, and (e) backing up Your Content.

It is the customer’s responsibility to secure data appropriately through accessibility and authorisation. This means securing connectivity with a Virtual Private Network (VPN) or Direct Connect and maintaining associated on-premise firewalls accordingly, as well as implementing security policies and firewall rules for the VMware Cloud Internet Gateway, VMware Cloud Compute and Management Gateways (NSX Edge Firewalls), and the NSX Distributed Firewall (Micro-Segmentation).

While the NSX Edge Firewalls protects north-south traffic; essentially anything coming in or out of the SDDC, the Distributed Firewall protects east-west traffic; between workloads inside the SDDC. Micro-Segmentation can be used to protect applications by ring-fencing virtual machines in a zero-trust architecture. The VMware Cloud on AWS NSX Networking and Security eBook goes into great detail on the NSX Edge and Distributed Firewalls with screenshots and configuration examples in chapter 6 (page 83). Both firewall types are included for all virtual machines in the default VMware Cloud on AWS pricing model.

All native AWS services deployed in the connected AWS VPC fall under the customer’s responsibility to secure as normal with Security Groups, Access Control Lists (ACLs), Identity and Access Management (IAM) groups/roles/policies, etc. This includes the cross-VPC 25Gbps Elastic Network Interfaces (ENI) deployed to connect the SDDC with the customers VPC.

VMware Cloud on AWS customers retain control and ownership of their Customer Content and it is the customer’s responsibility to manage data retention to their own requirements. VMware Cloud on AWS backs up Account Information including system configuration settings but does not provide backup services for Customer Content.

Source: Cloud Security Alliance (CSA) VMware Cloud on Amazon Web Services (AWS) self assessment

The CSA assessment and VMware Cloud Services Security Overview go into more detail on code security, change control, quality assurance, and configuration management, however, it is worth calling out patching. VMware are responsible for patching the underlying infrastructure of the platform; this includes all network, utility and security equipment. Critical security patches are ‘installed in a timely manner’, while non-critical patches are included in predefined patch schedules.

Customers have visibility into VMware SDDC products are updated from the Cloud Services Portal. In most cases updates and patches can be applied before General Availability (GA); some products run VMware Cloud specific versions and do not need to wait for the next GA release of vSphere, for example. Also VMware has subscriptions to internal vendor security and bug-tracking notification services, meaning remediation efforts are accelerated, and critical or high-risk issues prioritised, often having been applied before the vulnerability has been made public.

For more information on AWS Security by Design and the shared security model of security of the cloud and security in the cloud, start with the Introduction to AWS Security by Design Whitepaper.

Physical Security

VMware Cloud on AWS uses Amazon Web Services (AWS) geographically resilient data center hosting facilities.  Data centers are built in clusters in various global regions. VMware provides customers the flexibility to place VMware Cloud on AWS instances and store data within multiple geographic regions as well as across multiple Availability Zones within each region to minimize risk.

Physical Access is strictly controlled both at the perimeter and at building ingress/egress points and includes, but is not limited to fencing, walls, video surveillance, intrusion detection systems and other electronic biometric access controls and alarm monitoring systems managed by a 24x7x365 professional security staff.

AWS equipment is protected from outages in alignment with ISO 27001 standard. AWS has been validated and certified by an independent auditor to confirm alignment with ISO 27001 certification standard.  AWS Availability Zones are all redundantly connected to multiple tier-1 transit providers.

Customers explicitly choose which VMware Cloud on AWS data center best suits their needs, and customer data will not traverse locations without the explicit actions of the tenant administrator.

Automated processes are in place that handle media sanitization before repurposing of any hardware. Upon the explicit deletion of a production environment by a tenant, a cryptographic wipe of the hard drive is performed via destruction of keys used by the self-encrypting drives.

When a physical storage device has reached the end of its useful life, a decommissioning process that is designed to prevent customer data from being exposed to unauthorized individuals is followed using techniques detailed in NIST 800-88 (“Guidelines for Media Sanitization”) as part of the decommissioning process – the same applies when exiting the VMware Cloud service.

Source: Cloud Security Alliance (CSA) VMware Cloud on Amazon Web Services (AWS) self assessment

The exact locations of AWS data centres are generally kept secret, and they do not run data centre tours (this digital tour is about as good as it gets). AWS provides regions which contain multiple Availability Zones, consisting of one or more data centres all physically separated from one another. Electrical power systems, water, telecommunications, and internet connectivity are all designed to be fault-tolerant. Availability Zones are connected using private fibre-optic networking allowing customers to architect highly available solutions.

Physical access to data centres is restricted to those with valid and approved business justification. Site and server room access is limited to authorised individuals and requires a point in time access with multi-factor authentication. AWS implement additional perimeter security features outlined above and monitoring for things like open doors and removal of assets.

Media storage devices used to store customer data are classified by AWS as critical and treated as high-impact throughout their life-cycle. Media is decommissioned using techniques detailed in NIST 800-88 and is not removed from AWS control until it has been securely decommissioned. AWS and employees are audited by multiple compliance programs, you can download AWS Compliance Reports from the AWS Artifact service in the AWS console.

Security Operations Monitoring

VMware monitors internal platform & systems for privacy breaches and has a breach notification process to notify customers in the event of a privacy breach. If VMware becomes aware of a security incident on VMware Cloud on AWS that leads to the unlawful disclosure or access to personal information provided to VMware as a processor, we will notify customers without undue delay, and will provide information relating to a data breach as reasonably requested by our customers. VMware will use reasonable endeavors to assist customers in mitigating, where possible, the adverse effects of any personal data breach.

VMware Cloud on AWS has the capability to detect attacks that target the virtual infrastructure.  VMware Cloud on AWS has several intrusion detection mechanisms in place.  VMware log aggregation systems continuously ingest AWS firewall, AWS security services along with Cloud Trail logs, infrastructure and VPC Flow all logs. VMware continuously collects and monitors services operation logs using SIEM technologies. The 24x7x365 VMware Security Operations Center uses the SIEM to correlate information with public and private threat feeds to identify suspicious and unusual activities.

The real-time status of the VMware Cloud on AWS services along with past incidents is publicly available here. Availability reports are available to customers upon request within 45 days after a validated SLA event.

The VMware Security Operations Center (SOC) uses Log capture and SIEM tools, security monitoring technologies and intrusion detection tools in realtime to identify unauthorized access attempts or any behaviors that would indicate abnormal activity.

All changes to the virtual machine configuration are logged and available to the customer which enables detection of tampering and integrity checking.

VMware monitors AWS infrastructure and receives notifications directly from AWS in the event of a provider failure. VMware has developed processes with AWS to ensure that that we have defined disaster recovery mechanisms in place in the event that an upstream event occurs. VMware Cloud on AWS has conducted successful DR testing and continues to test annually.

Source: Cloud Security Alliance (CSA) VMware Cloud on Amazon Web Services (AWS) self assessment

See the Data Access section for more information on access logging. For ingesting logs from VMware Cloud on AWS, as well as native AWS, and other sources, customers can use Log Insight Cloud, which has both free and chargeable versions.

To address any further concerns, customers can also use their own Security Information and Event Management (SIEM) tools, such as Splunk, to continuously monitor the VMware Cloud on AWS environment for any unauthorised activity. Furthermore, tools currently used to scan or secure VMware environments on-premise can mostly be carried across to the VMware Cloud on AWS environment with IPFIX and Port Mirroring. This gives the customer unprecedented visibility under the hood of a cloud environment. The VMware Cloud on AWS NSX Networking and Security eBook goes into more detail on these operational tools in chapter 7 (page 101).

As well as Log Insight Cloud and/or the customers own SIEM tools, AWS can be used to monitor the connected VPC and services. AWS CloudTrail is a service that logs all API calls associated with your account. At the same time, AWS Config provides visibility of assets through an inventory of AWS resources and a history of configuration changes to these resources. You can use AWS Config to define rules that evaluate these configurations for compliance. VMware Cloud resources deployed to your connected VPC, such as IAM configurations for SDDC formation, and the attached ENIs, can be found in AWS Config. Your AWS logs can also be added as a content pack or log source for Log Insight Cloud.

In the example screenshots below, you can see part of the AWS CloudTrail logs for initial SDDC deployment. Highlighted in green is my user account linking the AWS account and running the CloudFormation template to create the appropriate IAM configuration, then a few minutes later in yellow the events for the ENIs being added and configured. You can view this in more detail in your own environment, the second screenshot shows AWS Config verifying there have been no changes made since initial deployment. I have had to remove most of the detail, but you get the idea.

CloudTrailAWSConfig

For a full list of AWS services that can be used to secure your native workloads and resources see the AWS Cloud Security page. The VMware Cloud on AWS NSX Networking and Security eBook also contain information on leveraging native AWS services in chapter 9 (from page 134).

It is important to note at this point that AWS security tools can only be used in the accounts you have access to. When VMware Cloud on AWS is deployed, i.e. the Elastic Compute Cloud (EC2) bare metal instances with associated VPC and subnets, routing table, etc. the customer does not have access to the underlying account and VPC. This is where the VMware logs outlined above are used to monitor the environment.

Penetration Testing & Audit

VMware has a comprehensive vulnerability management program. As a part of the vulnerability management program, penetration tests are performed at least annually.

Penetration test results are not provided externally.  VMware Cloud on AWS is subject to regular internal and external reviews and security assessments.  As a part of the VMware Cloud on AWS vulnerability management program, results are reviewed by the VMware security team(s) and remediation is performed based on the security team’s guidance.

With prior approval, Tenants are permitted to perform vulnerability assessments against their allocated service objects. Tenants are not permitted to perform vulnerability assessments against shared VMware assets.

VMware engages independent third-party auditors to perform reviews against industry standards. VMware will furnish audit reports under NDA.

Internal audits are performed at least annually under the VMware information security management system (ISMS) program. VMware utilizes internal/external audits as a way to measure the effectiveness of the controls applied to reduce risks associated with safeguarding information and to identify areas of improvement. Audits are essential to the VMware continuous improvement programs.

External audit reports will be provided to customers under NDA. Internal audit reports are classified as VMware confidential information and are not provided to tenants. Internal audit reports are reviewed by independent third-party auditors as a part of the VMware compliance program.

Risk assessments are performed at least annually, and results are disseminated to management.  Adjustments are made to policies, procedures, standards and controls where necessary to address risks and corrective action plans.

Source: Cloud Security Alliance (CSA) VMware Cloud on Amazon Web Services (AWS) self assessment

Data Access

VMware Cloud on AWS is built on the VMware Photon OS and VMware ESXi. The VMware Cloud on AWS Operations team disables unnecessary ports, protocols and services to harden the production environment. VMware applies security templates via Group Policy Object and we further harden servers through scripts. External communication in the production environment is restricted to ports 80 and 443 and all traffic passes through a firewall before reaching proxy servers in our DMZ. Managed interfaces are configured to deny-all communications traffic by default and allow network communications traffic by exception. – VMware Cloud on AWS deployment uses AWS CloudFormation, configuration and failure remediation is also scripted and automated to ensure a consistent and secure approach.

Customers maintain control of who has access to their VMware Cloud on AWS SDDC environment. VMware Cloud on AWS supports Identity Federation between vSphere and the customer’s identity provider using SAML standards for authentication. – Role-Based Access Control (RBAC) is used to assign permissions to both the vCenter Server and the Cloud Services Portal.

VMware Cloud on AWS natively supports Common Access Card Authentication and RSA SecurID Authentication to the vSphere client. Other multi-factor authentication systems can be supported via federation between vSphere and the customer’s Identity Provider.

Access control, separation of duties, and other policies define which individuals are allowed to have access to VMware Cloud on AWS management systems. Access to customer environments where customers data is stored, is limited to authorized VMware support engineers who must authenticate via two-factor authentication to an access control system in order to generate user-specific, time-limited credentials. Generation of these temporary credentials must be tied to an existing specific support incident ticket in the system.  All activity performed by the support engineers is logged while accessing customer SDDCs. – In general, automated runbooks will address previously encountered issues. Execution of automated runbooks is logged and can be traced back to specific support personnel. However in the event, an issue requires the VMware Cloud on AWS Site Reliability Engineering (SRE) team to access the SDDC; time-limited credentials are generated providing access to a specific SDDC for only 8 hours, and must be linked to a system-generated or customer-generated support ticket. All activities carried out are visible straight away to the customer via the vCenter logs.

Privileged access is logged and captured in a centralized log server.  VMware continuously collects and monitors services operation logs using SIEM technologies. The 24x7x365 VMware Security Operations Center uses the SIEM to correlate information with public and private threat feeds to identify suspicious and unusual activities.

Restricted, authorized personnel have access to the definitive central log servers for the VMware Cloud on AWS servers. Log aggregation sources and storage are protected and integrity of log data is preserved.  Security logs are stored for at least 1 year.

The Customer’s access logs are replicated to other systems where they can be viewed by customers and other individuals with appropriate approvals.

VMware has also deployed mechanisms to ensure that the log data has been properly copied, transported and securely stored to preserve the information as required to maintain full data integrity. Metadata about the environment including security logs are stored for at least 1 year.

VMware Cloud on AWS platform access controls are implemented via directory services group management.  All individuals who have access to the IT infrastructure and their level of access can be identified by enumerating the members of these dedicated groups.

VMware conducts criminal background checks, as permitted by applicable law, as part of pre-employment screening practices for employees commensurate with the employee’s position and level of access to the service.

In alignment with the ISO 27001 standard, all VMware personnel are required to complete annual security awareness training.  Personnel supporting VMware managed services receive additional role-based security training to perform their job functions in a secure manner.

All VMware personnel are required to sign confidentiality agreements as a part of onboarding. Additionally, upon hire, personnel are required to read and accept the Acceptable Use Policy and the VMware Business Conduct Guidelines.

Source: Cloud Security Alliance (CSA) VMware Cloud on Amazon Web Services (AWS) self assessment

Encryption

Key management policies and procedures are in place to guide personnel on proper encryption key management.  Access to cryptographic keys is restricted to limited operational personnel and all access is logged and monitored.  Cryptographic keys used by self-encrypting drives are managed by AWS.

All keys used in VMware Cloud on AWS are unique per tenant.  Tenant specific keys are programmatically generated by an independent and well-established certificate authority at the time of provisioning and are tied to the unique URLs created for each tenant.

All Customer Content imported to VMware Cloud on AWS is stored on dedicated physical NVMe storage hardware that is self-encrypting using XTS-AES-256. Encryption keys of the self-encrypting drives are generated in the physical SED controller and they never leave the storage device. The vSphere, (KEK), keys are encrypted with the AWS KMS (CMK), which is managed by the AWS KMS that uses FIPS 140-2 validated hardware security modules (HSM).  The DEK is encrypted using the local host KEK which are then used for encrypting and decrypting virtual machine files. The vSphere managed encryption keys can be managed/rotated by the customer at any time using the vSAN API or through the vSphere UI.

By default, all customer data at rest is also encrypted by vSAN XTS AES-256 cipher data-at-rest encryption, with two levels of keys: KEK (as the master key) and DEK (per-disk data key).

VMware personnel manage and secure the encryption certificates used to communicate with the VMware Cloud on AWS console and VMware has key management controls in place.  VMware Cloud on AWS operations have complete visibility into certificate information such as installed, expiring and revoked certificates through a certificate management dashboard.

Customers can provide their own keys for VPN connectivity and VMware Cloud on AWS fully supports the use of in-guest encryption of Customer Content which further enables customers to use additional encryption technologies of their choice as well as the key management products and processes to meet their security requirements. For customers who choose to implement in-guest encryption of their Customer Content, VMware does not manage the keys.

VMware utilizes an industry leading commercial solution to secure, store, and tightly control access to tokens, passwords, certificates, API keys, and other secrets.

Data in-transit (authentication, administrative access, customer information, etc.) is encrypted with standard encryption mechanisms (i.e. SSH, TLS). Encrypted vMotion is available at VMware Cloud on AWS between hosts inside the Cloud SDDC.

VMware provides customers with the ability to create IPSEC and SSL VPN tunnels from their environments which support the most common encryption methods including AES-256.

Source: Cloud Security Alliance (CSA) VMware Cloud on Amazon Web Services (AWS) self assessment

Whenever a host machine is removed from a cluster, the data encryption keys used by the self-encrypting drives are destroyed. This cryptographic erasure ensures that there is no customer content on the drives before returning the servers to the pool of available hardware. The use of self-encrypting drives protects customers from an individual with physical access to the data centre, being able to physically remove drives and access the contents of the drives.

As a further layer of security, the VMware vSAN implementation for VMware Cloud on AWS has encryption enabled by default for all clusters, along with de-duplication and compression. These features are defined when a cluster is provisioned and cannot be disabled. Also, vSAN provides customisable storage protection policies to ensure data is tolerant of the failure of one or more physical drives in a cluster.

The vSAN storage array encryption allows customers to rotate encryption keys on-demand to meet industry regulations, this can be done via vShere and API. All vSphere features such as vMotion, Distributed Resource Scheduler (DRS), and High Availability (HA) are supported with vSAN Encryption without impacting I/O performance.