Author Archives: esxsi

vSphere 7 with Kubernetes and Tanzu on VMware Cloud Foundation

vSphere 7 Cloud Infrastructure for Modern Applications Part 1

vSphere 7 Cloud Infrastructure for Modern Applications Part 2:

VMware Cloud Foundation 4

VMware Cloud Foundation (VCF) 4 delivers vSphere with Kubernetes at cloud scale, bringing together developers and IT operations by providing a full stack Hyper Converged Infrastructure (HCI) for Virtual Machines (VMs) and containers. By utilising software-defined infrastructure for compute, storage, network, and management IT operations can provide agility, flexibility, and security for modern applications. The automated provisioning and maintenance of Kubernetes clusters through vCenter Server means that developers can rapidly deploy new applications or micro-services with cloud agility, scale, and simplicity, while IT operations continue supporting the modern application framework by leveraging existing vSphere functionality and tooling.

VMware has always been an effective abstraction provider, VCF 4 with Tanzu Services View takes infrastructure abstraction to the next level. Within vSphere underlying infrastructure components are abstracted into a set of services exposed to APIs, allowing the developer to look down from the application layer to consume the hybrid infrastructure services. Meanwhile IT operations can build out policies and manage pods alongside VMs at scale using vSphere.

VCFwKubernetes

vSphere 7 with Kubernetes

Now and over the next 5 years we are seeing a shift in how applications are built and run. In 2019 Line of Business (LOB) IT, or shadow IT, spend exceeded Infrastructure and Operations IT spend for the first time*. Modern applications are distributed systems built across serverless functions or managed services, containers, and Virtual Machines (VMs), replacing typical monolithic VM application and database deployments. The VMware portfolio is expanding to meet the needs of customers building modern applications, with a portfolio of services from Pivotal, Wavefront, Cloud Health, bitnami, heptio, bitfusion, and more. In the container space VMware is strongly positioned to address modern application challenges for developers, business leaders, and infrastructure administrators.

Launched on March 10 2020, with expected April 2020 availability, vSphere 7 with Kubernetes is powering VMware Cloud Foundation 4. vSphere 7 with Kubernetes integration, the first product including capabilities announced as part of Project Pacific, provides real-time access to infrastructure in the Software Defined Data Centre (SDDC) through familiar Kubernetes APIs, delivering security and performance benefits even over bare-metal hardware. The Kubernetes integration enables the full SDDC stack to utilise the Hybrid Infrastructure Services from ESXi, vSAN, and NSX-T which provide the Storage Service, Registry Service, Network Service, and Container Service. Developers do not need to translate applications to infrastructure, instead leveraging existing APIs to provision micro-services, while infrastructure administrators use existing vCenter Server tooling to support Kubernetes workloads along side Virtual Machines.

You can read more about the workings of vSphere with Kubernetes in this Project Pacific Technical Overview for New Users by Ellen Mei. In addition Initial Placement of a vSphere Pod by Frank Denneman is another useful and recent article detailing the process behind the ESXi container runtime.

VMware Application Modernisation Portfolio with Tanzu

VMware Cloud Foundation services is the first manifestation of Project Pacific; now vSphere with Kubernetes, and provides consistent services managed by IT operations to developers, ultimately anywhere that is running the VMware Cloud platform.

In the past VMware were efficient at taking a number of Virtual Machines and running them across multiple hypervisors in a cluster, the challenge was consolidating multiple physical servers for cost saving, management, and efficiency gains. Today the challenge is that application deployments are groups of VMs, which presents a new challenge of consolidating distributed applications across multiple clouds. Project Pacific brings Kubernetes and Tanzu to the vSphere environment, making it operationally easier to get upstream Kubernetes running, but also to effortlessly inplace upgrade and maintain Kubernetes clusters. This functionality accelerates vSphere into a much more modern, API driven, self-service and fast provisioning interface backed by optimised ESXi for all workloads.

Tanzu Kubernetes Grid (TKG) is a Kubernetes runtime built into VMware Cloud Foundation Services; allowing installing and maintaining of multi-cluster Kubernetes environments across different infrastructure. Tanzu Kubernetes Grid also works for operational consistency across Amazon Web Services (AWS), Azure, and Google Compute Engine (GCE). This is different to public cloud managed Kubernetes services such as EKS, AKS, GKE, etc. as it integrates natively into the existing infrastructure; meeting the needs of organisations who require abstracted logging, events, governance policies, and admission policies. This capability is delivering not just Kubernetes but a set of management services to provision, deploy, upgrade and maintain Kubernetes clusters. By having this fine grain level of control over the underlying VMs or cloud environment customers are able to implement, monitor, or enforce their own security policies and governance.

Tanzu Mission Control provides operator consistency for deployment, configuration, security, and policy enforcement for Kubernetes across multiple clouds, simplifying the management of Kubernetes at scale. Tanzu Mission Control is a Software as a Service (SaaS) control plane offering allowing VMware administrators to standardise Identity Access Management (IAM), configuration and policy control, backup and recovery, ingress control, cost accounting, and more. The multi-cluster control plane supports the propagation of Kubernetes across vSphere, Pivotal Container Services (PKS), AWS, Azure, and Google Cloud Platform (GCP), all from a single point of control.

VMware have announced availability of Tanzu Kubernetes Grid, Tanzu Mission Control, and Tanzu Application Catalog (open source software catalog powered by Bitnami), providing a unified platform to build, run, and manage modern applications.

VMware Cloud Foundation Additional Updates

VMware Cloud Foundation (VCF) 4 is expected in April 2020, and includes fully updated software building blocks for the private cloud, including vCenter, ESXi, and vSAN 7.0, plus the addition of NSX-T.

VCF with NSX-T is made up of workload domain constructs,  and by default every architecture starts with a management domain, which hosts vCenter, private NSX managers, and the edge cluster. There are a couple of changes in VCF 4 that reduce the footprint of the management domain; NSX-T is being fully utilised for the first time, the NSX Edge cluster can be deployed at day X, NSX Manager and controllers are now integrated, and the Platform Services Controller (PSC) is also now using the embedded model with vCenter. Additionally we have the capability to use Application Virtual Networks (AVM) using BGP peering on deployment, or again as a day X action. Another side note is that Log Insight has been changed from a default deployment requirement, to an optional day X action.

Workload domains are built out to server vSphere with Kubernetes and expose the network services for developers to use. Workload domains can be built on new or existing NSX-T managers; offering the choice of a one-to-one or one-to-many relationship for NSX-T instances with VCF 4. This provides customers with the option of separating out NSX-T instances, whilst simultaneously protecting the management domain. Day X automation then can be used to place edge deployments in the appropriate cluster:

NSXWorkloadDomains

SDDC Manager and Lifecycle Manager (LCM) provide automated provisioning and upgrades. Lifecycle Manager enhances ease of upgrade and patching by providing automated lifecycle management; with update notifications, review and schedule options, and monitoring and reporting. In addition LCM can manage all inter-dependencies of versioning at a cluster level, from vSphere right through to the Kubernetes runtime. SDDC Manager is orchestrating and automating the provisioning of vSphere with Kubernetes workload domains, and crucially enabling the LCM functionality for maintaining upgrades for the entire software stack, and eliminating typical day 2 challenges for developers.

SDDCManager

Multi-Instance Management: multiple VCF instances can now be federated to provide a global view of workload domains without the installation of any additional components. Administrators can click-through to any VCF data centre to centrally view patching, upgrades, maintenance, and remediation operations.

New Security Enhancements: native Workspace ONE Identity Access integration for vRealize suite and NSX-T using AD or LDAP identity sources. Admin and Operator roles for API and UI, with the operator role providing all privileges minus password management, user management, and backup and restore. Token based authentication is also now enforced across all APIs.

You can find out more about the VMware Cloud Foundation 4 update at What’s New in VMware Cloud Foundation 4  and Delivering Kubernetes at Cloud Scale with VMware Cloud Foundation 4.

VMware vRealize Cloud Management Integration

The vRealize Cloud Management product suite has been comprehensively updated to include vSphere 7 with Kubernetes support. vRealize Operations (vROps) 8.1 is now available for the first time as a SaaS (Software as a Service) offering with an enhanced feature-set. Some of the key new functionality enables self-driving operations across multi-cloud, hybrid-cloud and data centre environments.

vROps 8.1 and Cloud now fully support integrations with GCP, native VMware Cloud on AWS as a cloud account (including additional vSAN, NSX-T, and Cloud Services Portal information with billing),  enhanced portfolio of AWS objects, Cloud Health, and vSphere Kubernetes constructs. With the latter crucially enabling Kubernetes cluster on-boarding, discovery, continuous performance optimisation, capacity and cost optimisation, monitoring and troubleshooting, and configuration and compliance management. Furthermore, new dashboards and topology views of workload management can be leveraged to display all Kubernetes objects visible from vCenter, for a complete end-to-end view of the infrastructure.

K8s_Dashboard

vRealize Operations 8.1 and Cloud integration for vSphere with Kubernetes:

  • Automatically discover new constructs of supervisor cluster, namespaces, pods, and Tanzu Kubernetes clusters.
  • New dashboards and summary pages for performance, capacity, utilisation, and configuration management of Kubernetes constructs, with full topology views from Kubernetes substrate to physical infrastructure.
  • Capacity forecasting detects utilisation and potential bottlenecks for supervisor clusters and pods, and shows time remaining projections for CPU, memory, and diskspace.
  • Out of the box reporting functionality for workload management, inventory, configuration, and capacity, with configurable alerting to operationalise the workload platform and provide complete visibility and control.
  • Container management pack extends visibility to monitor and visualise multiple Kubernetes clusters, map and co-relate virtual infrastructure to Kubernetes infrastructure, setup alerts and monitoring, and provide support for PKS.

You can find out more about what’s new in the vRealize suite at Delivering Modern Infrastructure for Modern Apps with vRealize Suite.

*LOB spend 51% to infrastructure operations spend 49% – source IDC WW Semiannual IT Spending Guide: Line of Business, 09 April 2018 (HW, SW and services; excludes Telecom)

vSphere 7 and vSAN 7 Headline New Features

vSphere 7 Cloud Infrastructure for Modern Applications Part 1:

vSphere 7 Cloud Infrastructure for Modern Applications Part 2

vSphere 7 with Kubernetes

Now and over the next 5 years we are seeing a shift in how applications are built and run. In 2019 Line of Business (LOB) IT, or shadow IT, spend exceeded Infrastructure and Operations IT spend for the first time*. Modern applications are distributed systems built across serverless functions or managed services, containers, and Virtual Machines (VMs), replacing typical monolithic VM application and database deployments. The VMware portfolio is expanding to meet the needs of customers building modern applications, with a portfolio of services from Pivotal, Wavefront, Cloud Health, bitnami, heptio, bitfusion, and more. In the container space VMware is strongly positioned to address modern application challenges for developers, business leaders, and infrastructure administrators.

Launched on March 10 2020, with expected April 2020 availability, vSphere 7 with Kubernetes is powering VMware Cloud Foundation 4. vSphere 7 with Kubernetes integration, the first product including capabilities announced as part of Project Pacific, provides real-time access to infrastructure in the Software Defined Data Centre (SDDC) through familiar Kubernetes APIs, delivering security and performance benefits even over bare-metal hardware. The Kubernetes integration enables the full SDDC stack to utilise the Hybrid Infrastructure Services from ESXi, vSAN, and NSX-T which provide the Storage Service, Registry Service, Network Service, and Container Service. Developers do not need to translate applications to infrastructure, instead leveraging existing APIs to provision micro-services, while infrastructure administrators use existing vCenter Server tooling to support Kubernetes workloads along side Virtual Machines.

At this point on-premises Kubernetes orchestration is available through VMware Cloud Foundation 4. You can read more about Kubernetes with vSphere 7 in vSphere 7 with Kubernetes and Tanzu on VMware Cloud Foundation. Continue reading this post to review the additional functionality introduced with vSphere 7 and vSAN 7 around lifecycle management, scalability, security and compliance, you can also review the full vSphere 7 introduction here.

vSphere 7 Headline New Features

vCenter Server Profiles

vCenter Server Profiles are introduced in vSphere 7; enabling consistent configuration across the vCenter Server estate. vCenter Server Profiles export management, network, authentication, and user configurations into JSON format. The configurations can be edited, validated, and imported or pushed to up to 100 vCenter Servers, providing version control and a consistent last-known good state. vCenter Server Profiles are accessible via 4 new REST APIs; list, export, validate, and import.  This also means they can be consumed with DCLI, PowerCLI, or other automation tools such as Ansible, Puppet, and Chef. Behind the scenes vCenter Server Profiles are known as Infrastructure Profiles, and can be found under infra-profiles in the vCenter Developer Center  API Explorer. Note that vCenter Server profiles do not replace file based backups for vCenter, the profile exports do not contain GUIDs etc. that would be required for a full and supported vCenter Server restore.

vCenter Server Profiles

vCenter Server Update Planner

The new vCenter Server Update Planner provides native tooling to help with discovering, planning, and upgrading vCenter Server and connected products successfully. VMware administrators can receive notifications in the vSphere client when an upgrade or update is available. VMware product interoperability is built-in and automatically detects installed products to provide monitoring and checks against the current vCenter Server version; showing compatible upgrades and removing guesswork and complicated interoperability questions for complex environments. To further validate upgrade paths ‘what-if’ workflows, and pre-update checks, can be run against the selected target vCenter. The vCenter Server Update Planner also links to applicable release notes and Knowledge Base (KB) articles. An extra benefit of the vCenter Server 7 upgrade process is the automation of the external Platform Services Controller (PSC) which is now built into the upgrade, more on this further down the post.

vCenter Server Cluster Image Management

Cluster Image is the new model for management of the ESXi lifecycle, providing consistency of ESXi hosts across a cluster. The cluster image comprises of specific firmware, drivers, or vendor software add-ons, to create a desired state model with multi-host remediation capabilities. The Cluster Image feature is exposed through the vSphere client, REST API, and also integrates with third party vendor management tools such as Dell OpenManage and HPE OneView. This means host firmware can now be managed and upgraded from within vSphere, removing the risk of unsupported drivers and firmware. To use this feature all hosts in a cluster must be the same hardware type and must all be running ESXi 7.0.

New vSphere DRS Improvements

Distributed Resource Scheduler (DRS) is evolving to meet Virtual Machine needs, and has undergone a number of new improvements. DRS now makes workload centric placement decisions based on VM data gathered every minute, as oppose to cluster centric decisions based on 5 minutes of data. Placement decisions are now based on the individual VM DRS scores and granted memory. This shifts the focus onto whether the workload is getting the required resource, rather than the balance of the whole cluster. The VM DRS score is calculated using CPU %RDY (Ready) time, memory swap, CPU cache behaviour, headroom for the workload to burst, and migration cost. VM DRS scores are grouped into buckets of 20% increments.

Improved DRS vSphere 7

Improved DRS in vSphere 7 now includes Scalable Shares, providing relative resource entitlements across Resource Pools. Setting a share level to ‘high’ now ensures prioritisation over lower share VM entitlements, whereas previously the higher share level did not guarantee a higher resource entitlement. Scalable Shares need to be enabled and can be configured on a cluster level and/or Resource Pool level. Share allocations are dynamically changed depending on the number of VMs in a Resource Pool. The only exception to this rule is vSphere with Kubernetes where a Resource Pool is used as a Namespace, in this instance Scalable Shares are used by default.

DRS placement now includes assignable hardware – support for hardware accelerators, for both DRS initial placement and vSphere High Availability (HA). When adding a new device dynamic DirectPath IO or NVIDIA GRID vGPU devices are supported. DRS works with the assignable hardware framework to find a host with an available PCIe device configured, or hardware profile, when making initial placement decisions. The functionality requires the new VM hardware version 17.

New vSphere vMotion Improvements

Increased workload resource consumption as applications change over-time has started presenting performance challenges during vMotion and stun times for large or monster VMs. To address these challenges vMotion has been refactored as part of vSphere 7, bringing back vMotion capabilities for worklaods like SAP HANA or Oracle.

During vMotion a page tracer is installed so vSphere can keep track of the memory pages that are overwritten by the guest OS while the VM is in a vMotion state. To install the page tracer the vCPU is stopped (for micro-seconds), allowing the monitoring of memory page overwrites. These overwrites as referred to as page fires, which are replicated to the destination ESXi host. The page tracer was previously installed on all vCPUs in a VM. In vSphere 7 only one vCPU is claimed and dedicated to all the page tracing work during a vMotion operation. This improves the efficiency of page tracing, and greatly reduces the performance impact on a workload. When all memory pages have been migrated the last memory bitmap is transferred, in previous versions the entire bitmap was transferred, in vSphere 7 the bitmap is compacted and only the last pages are sent, cutting down the switch over and stun time.

vMotion Improvements

Enhanced vMotion Capability (EVC) has been updated with new baselines for CPU packages: Intel Cascade Lake generation and AMD Zen2 generation (EPYC Rome).

New vSphere Security & Compliance Features

vSphere 7 now supports Intel Softguard Extensions (SGX) which allows applications to work within the underlying hardware to create a secure enclave that cannot be viewed by the guest OS or hypervisor. The application can store secrets or data in the enclave which is an important feature for risk management, although currently there is minimal hardware support. Intel Ice Lake CPUs will have dual socket implementations of SGX. If implementing SGX remember that you will lose certain features such as vMotion, snapshots, and so on if the hypervisor cannot see everything in the VM, this becomes very much an application design decision.

vSphere 7 introduces vSphere Trust Authority (vTA), providing trusted hosts and encryption key management. Previous trust models in vSphere had potential for running secure workload on untrusted hosts, with no repercussions for failing secure baselines. Attestation and key management was done by vCenter Server, which itself could not be encrypted. The dependencies on the vCenter Server itself made it difficult to implement the principle of least privilege. With vTA a hardware root of trust is created using a separate ESXi host cluster, this can also be your management cluster. The key manager only talks direct to trusted hosts, rather than the vCenter Server. Workloads running on the trusted cluster, now including vCenter Server, can be encrypted. A smaller number of administrators can be given access to the trusted hosts, with regular admins maintaining access to the workload hosts. Currently vTA is still foundational, so expect more functionality to be available in future releases. It is important to note that to use the trusted host model the physical server must have the TMP 2.0 chip, which is cryptographically bonded to the host.

vSphere 7 Trust Authority

Identity Federation is introduced in vSphere 7 to modernise vSphere Authentication utilising standards-based federated authentication with enterprise Identity Providers. Using Identity Federation organisations can benefit from reduced audit scope and administrative workload, as well as security enhancements such as Multi-Factor Authentication (MFA). Initial integration will be with Active Directory Federation Services (ADFS) / Azure Active Directory, which alongside MFA is great for compliance and security. Identity Federation will also work with the Supervisor Cluster for Kubernetes, which inherits a lot of the security and functional controls from vCenter to help bridge the gap between developing modern applications and existing processes and infrastructure.vSphere 7 Identity Federation

There are hundreds of improvements in vSphere 7 to drive consistency and trust into the environment. For example the default settings for the vSwitch now includes SecurebyDefault to enforce security settings, the Certificate Management UI has been consolidated and simplified, and so on. You can review the vSphere 7 release notes for full information.

Additional Noteworthy vSphere 7 Features

  • vCenter Server Content Library: a new interface provides vast improvements on template management. Virtual Machine templates are now checked out to edit and checked in to save, facilitating version control, quick historical view of edits, and ability to restore to previous versions. You can switch between the new view and classic view in the vSphere client. Additional features such as versioning is only available when the VM template is stored in a Content Library. Advanced Configuration now allows update of auto-sync frequency, and performance optimisation.
  • vCenter Server Multi-Homing: vCenter Server 7 now supports multiple network adaptors, the maximum supported vNIC limit is 4 per vCenter Server, with NIC1 reserved for vCenter HA.
  • vCenter Server SSO Domain Consolidation: vSphere SSO domain or external PSC consolidation has been simplified with new tooling commands for domain re-pointing or un-registering: cmsso-util unregister and domain-repoint.
  • vCenter Server External PSC Consolidation: the Upgrade and Migration setup no longer allows the deployment of an external PSC. Furthermore, the external PSC consolidation process is now automatically built into the upgrade, reducing administrative time and effort during the upgrade process. This means the vCenter Server Converge Tool has been removed from the ISO. The external PSC consolidation during upgrade is also a supported configuration in JSON format when upgrading using the CLI.
  • VM Hardware v17: the new VM hardware version features a virtual Watchdog Timer providing guest OS and application monitoring, especially important for clustered applications like databases and filesystems. Precision Time Protocol (PTP) now provides sub-millisecond timekeeping, helpful for financial and scientific applications. PTP requires both an in-guest device and the ESXi service to be enabled.
  • vCenter Server Configuration Maximums: further enhancements to vCenter Server scalability:
    • vCenter Server (standalone) number of hosts per vCenter Server: 2500, powered-on VMs per vCenter Server: 30,000
    • Linked Mode vCenter Servers (15 per SSO domain) hosts: 15,000, powered-on VMs: 150,000
    • vCenter Server latency requirements for vCenter Server to vCenter Server: 150ms, vCenter Server to ESXi Hosts: 150ms, vSphere Client to vCenter Server: 100ms

vCenter Server Config Maximums

You can read the full vSphere 7 release information at Introducing vSphere 7: Essential Services for the Modern Hybrid Cloud as well as the vSphere 7 Data Sheet and vSphere 7 Product Page.

vSAN 7 Headline New Features

A number of new features have been added to vSAN 7 alongside the vSphere 7 announcement, here are the key product enhancements:

Simplified Lifecycle Management

vSphere Lifecycle Manager (vLCM) is a new approach to unified software and firmware management increasing reliability and decreasing the number of update tools. vLCM is built around the desired state model and monitors and remediates compliance drift. Desired state and desired images are applied at cluster level and manage the server stack as a whole, across hypervisor, drivers, and firmware. Furthermore, the modular framework supports vendor firmware plugins such as Dell and HPE.

Unified Block and File Storage

Fully Integrated File Services provides a native file service built into the hypervisor through vSAN. Cluster capacity for vSAN can be provisioned into file shares with support for NFS v4.1 & v3, and file share quotas, unifying management of block and file storage. vSAN file shares are aimed at ease of use for both cloud native and traditional workloads running in the cluster, it is not necessarily a replacement for large scale filers.

vSANFile

Expanded Data Services

Continued Integration of Cloud Native Storage provides the control plan and storage service for vSphere with Kubernetes integration, and offers file-based persistent volumes easily accessible and managed within vCenter. This now includes support for vVols, persistent volume encryption, and snapshots, volume resizing, and a mixture of tooling such as application monitoring with Wavefront, next generation monitoring solutions like Prometheus, and infrastructure analytic solutions like vRealize Operations, providing an advanced level of visibility for vSphere administrators.

vSANCloudNative

Improved Efficiency and Operations

  • Enhancements for Stretched Cluster and 2-Node Topologies: such as support for overriding the default gateway used by vSAN hosts to simplify deployments and routed topologies, immediate repair operation after a witness host appliance is replaced.
  • Intelligent capacity management for stretched cluster topologies; when the cluster is in a capacity constrained state, for example due to host failure, objects in a critical state are marked by vSAN as absent, allowing I/O to be processed at another site. The degraded state of the object in terms of resilience still stands, but the VM uptime is improved by allowing continuation of read/write operations. The object is updated when the capacity strain condition is removed.
  • Stretched cluster awareness for DRS placement decisions; enables prioritisation of I/O read locality over VM site affinity rules, completion of vSAN resync before DRS migrations,  and a reduction in I/O across ISL in recovery conditions.
  • Improved accuracy in VM capacity reporting across vCenter UI and APIs when working with thin provisioned VMs, swap objects, and namespace objects; reducing confusion and inconsistency over provided and used space for a given VM.
  • A new vSAN memory metric has been added in the vSAN performance service to display memory consumption of vSAN operations such as hardware and software configuration changes. The additional vSAN memory metric displays time-based memory consumption per host and is available in the vCenter UI and API.
  • New vSphere Replication object identity types to easily identify objects created by or using vSphere Replication, replacing the previous unknown object type.
  • Additional support for larger storage devices; up to 32 TB physical drives, and up to 1 PB in logical capacity. This gives potential for improved deduplication ratios when using larger devices for the capacity tier and deduplication domain.
  • Native support for NVMe hot plug through vSAN and vSphere for selected OEM platforms. This feature reduces host restarts and administrative complexity when carrying out planned or unplanned maintenance.
  • Removal of Eager Zero Thick (EZT) requirement for vSAN shared disks, improving application consumption and flexibility.
  • The full vSAN announcement can be found here

vSphere 7 with Kubernetes and vSAN 7 are built into VMware Cloud Foundation 4, read more on the March 10 2020 announcement in vSphere 7 with Kubernetes and Tanzu on VMware Cloud Foundation.

*LOB spend 51% to infrastructure operations spend 49% – source IDC WW Semiannual IT Spending Guide: Line of Business, 09 April 2018 (HW, SW and services; excludes Telecom)

AWS Native Services Integration With VMware Cloud on AWS

This post lists some of the available options for connecting VMware Cloud on Amazon Web Services (AWS) with native AWS services for hybrid cloud deployments.

The most common way of consuming AWS services with VMware Cloud on AWS is to use the built in Elastic Network Interface (ENI) functionality. Each VMware Cloud Software Defined Data Center (SDDC) can be connected to another AWS Virtual Private Cloud (VPC) during the deployment phase. A VPC is Amazon’s logical separation of virtual networks. At scale, you may choose to have many VPCs and many accounts for different applications and environments. Multiple VPCs can be connected together using an AWS Transit Gateway (TGW). A further option we will look at is VPC Endpoints, enabling you to privately connect to supported AWS services and endpoints.

1. Connected VPC

The AWS bare metal hosts deployed for VMware Cloud on AWS use a redundant 25 Gbps physical interface or Elastic Network Adaptor (ENA). The physical interface uses a trunk port to carry multiple VLANs for services like management, vMotion, NSX, and connectivity to the AWS backbone network.

The cross-linked VPC architecture is provided by a series of ENIs. Each host in the vSphere cluster uses the network adaptor outlined above to provide an individual cross-VPC ENI per physical host; supporting high-bandwidth, low latency connectivity to native AWS services.

VMC_Host_Connectivity

VMware Cloud on AWS uses the AWS VPC  as an underlay for NSX-T. The NSX Edge (Tier 0) router is a virtual router acting as the uplink to the connected VPC. The active ENI in use is the physical ESXi host where the virtual router is running. The connected VPC is owned and managed by the customer, any native services deployed are billed separately by AWS. When deploying the SDDC the connected account and VPC is required along with a private subnet in each applicable Availability Zone (AZ). A static route is created for the defined subnets adding the connected VPC router as the next hop.

Traffic that traverses the ENI is not chargeable, however cross-AZ charges do need to be taken into consideration if a Stretched Cluster is in use. During provisioning of the SDDC, and connection of the customer managed AWS account, a CloudFormation template is deployed creating the necessary AWS Identity Access Management (IAM) roles and ENI configuration.

SDDC_AWS_1SDDC_AWS_2

Post-SDDC deployment you can view the connected account, VPC, ENI, and subnet details in the Connected VPC menu under the Networking & Security tab of the SDDC, from the VMware Cloud Services Portal.

Access to and from native AWS services can be controlled, and needs to be opened, using the NSX firewalls (gateway and distributed) and AWS Security Groups. To see an example configuration see the Connecting VMware Cloud on AWS to Amazon EC2 post, or the Access an EC2 Instance section of the VMware Cloud on AWS Docs page.

2. VPC Endpoints

VPC Endpoints allow private connectivity between your VPC and supported AWS services or custom applications. Network traffic traversing a VPC endpoint does not leave the AWS backbone network, and therefore does not require Internet Gateway, Direct Connect, or VPN.

The Access an S3 Bucket Using an S3 Endpoint section of the VMware Cloud on AWS Docs page details the process for configuring a Gateway VPC Endpoint to access AWS Simple Storage Services (S3) from VMware Cloud on AWS, without having to go out to the Internet. Furthermore, you can use Interface VPC Endpoints to connect to supported AWS services in another VPC, or VPC Endpoint Services (AWS PrivateLink) to connect to custom applications in another VPC. Here are some examples:

AWS_Integration_Examples

The general process for creating an endpoint is the same across these VPC Endpoint types. In the example below we are connecting to a VPC Endpoint Service for Splunk, fronted by a Network Load Balancer (NLB) in another VPC. The administrator of the VPC Endpoint Service needs  to grant IAM service consumer permissions and accept the incoming connection, as detailed in the AWS documentation here.

In the AWS console I log into the connected account and select the VPC service. I select Endpoints and Create Endpoint. To create a Gateway VPC Endpoint, e.g. for S3, or an Interface VPC Endpoint, e.g. for DynamoDB or other services, I would select the appropriate service from the AWS services service category. In this instance I use Find service by name and enter the endpoint private service name. Either way, the key point is that I select the connected VPC from the VPC drop-down, and the subnets that match up with those used for the ENI when deploying the VMware Cloud on AWS SDDC.

VPCEndpoint1VPCEndpoint1cont

By using the cross-VPC linked subnets the Virtual Machines in the SDDC will utilise the static route across the ENI outlined in the Connected VPC section above. AWS Security Groups can be used to limit this to certain source IP addresses from within the SDDC or the wider VPC if required. In this instance we are able to successfully test the connection over port 443 following the creation of the VPC Endpoint.

3. Additional VPC Connectivity

Traditionally VPC Peering has been used to provide one to one private network connectivity between VPCs, including VPCs in different accounts. VPC Peering cannot be configured in the SDDC as we do not have access to the underlying AWS account. VPN connections between additional VPCs and the SDDC router (Tier 0) can be configured from the VMware Cloud Services Portal, enabling VMware Cloud on AWS connectivity with other VPC environments. As the number of VPCs and accounts begins to scale the VPN approach becomes harder to manage.

This predicament is resolved with a relatively new addition to AWS; the Transit Gateway (TGW). The native AWS TGW is available now and acts as a transit network hub allowing you to connect multiple VPCs and on-premise networks (using Direct Connect or VPN attachments). A Managed Transit Gateway is being developed by VMware to assist with multi-SDDC and multi-VPC connectivity. You can review how the native AWS Transit Gateway fits into the VMware Cloud on AWS architecture on the VMware Network Virtualization blog: VMware Cloud on AWS with Transit Gateway Demo:

VMC_TGW_Examples

Image VMware Cloud on AWS with Transit Gateway Demo. Further Resources: