There will be more apps deployed in the next 5 years than in the last 40 years (source: Introducing Project Pacific: Transforming vSphere into the App Platform of the Future). In a shift towards application focus and to address application support complexities between development and operations teams VMware have announced Project Pacific. One of the drivers behind Project Pacific is to run Kubernetes components natively in vSphere. This post gives vSphere administrators an introduction on the technology and how it is expected to work.
Kubernetes is an open-source orchestration and management tool that provides a simple Application Programming Interface (API). Kubernetes enables containers to run and operate in a production-ready environment at enterprise scale by managing and automating resource utilisation, failure handling, availability, configuration, scale, and desired state. Micro-services can be rapidly published, maintained, and updated through self-service automation. Kubernetes managed containers and containers package applications and their dependencies into a distributed image that can run almost anywhere, often made up of micro-services. Kubernetes makes it easier to run applications across multiple cloud platforms, accelerates application development and deployment, increases agility, flexibility, and the ability to adapt to change.
Kubernetes uses a cluster of nodes to distribute container instances. The master node is the management plane containing the API server and scheduling capabilities. Worker nodes make up the control plane and act as compute resources for running workloads (known as pods). A pod consists of one or more running container instances, cluster data is stored in a key-value store called etcd. Kubelet is an agent that run on each cluster node ensuring containers are running in a pod. Kubernetes uses a controller to constantly monitor the state of containers and pods, in the event of an issue Kubernetes attempts to redeploy the pod. If the underlying node in a Kubernetes cluster has issues Kubernetes redeploys pods to another node. Availability is addressed by specifying multiple pod instances in a ReplicaSet. Kubernetes replica sets run all the pods active-active, in the event of a replica failing or node issue then Kubernetes self-heals by re-deploying the pod elsewhere. Kubernetes nodes can run multiple pods, each pod gets a routable IP address to communicate with other pods.
Kubernetes namespaces are commonly used to provide multi-tenancy across applications or users, and to manage resource quotas. Kubernetes namespaces segment resources for large teams working on a single Kubernetes cluster. Resources can have the same name as long as they belong to different namespaces, think of them as sub-domains and the Kubernetes cluster as the root domain that the namespace gets attached to. Pods are created on a network internal to the Kubernetes nodes. By default pods cannot talk to each other across the cluster of nodes unless a Service is created, this uses either the cluster network, the nodes network, or a load balancer to map an IP address and port of the cluster or node network to the pods IP address and port, thus allowing pods distributed across nodes to talk to each other if needed.
Kubernetes can be accessed through a GUI known as the Kubernetes dashboard, or through a command-line tool called kubectl. Both query the Kubernetes API server to get or manage the state of various resources like pods, deployments, services, ReplicaSets, etc. Labels assigned to pods can be used to look up pods belonging to the same application or tier. This helps with inventory management and with defining Services. A Service in Kubernetes allows a group of pods to be exposed by a common IP address, helping you defined network routing and load balancing policies without having to understand the IP addressing of individual pods. Persistent storage can be defined using Persistent Volume Claims in a YAML file, the storage provider then mounds a volume on the pod.
Native vSphere Integration
VMware have re-architected vSphere to include a Kubernetes control plane for managing workloads on ESXi hosts. The control plane is made up of a supervisor cluster using ESXi as the worker nodes, allowing workloads or pods to be deployed and run natively in the hypervisor. This functionality is provided by a new container runtime built into ESXi called CRX. CRX optimises the Linux kernel and hypervisor, and strips some of the traditional heavy config of a Virtual Machine enabling the binary image and executable code to be quickly loaded and booted. The container runtime is able to produce some of the performance benchmarks VMware have been publishing, such as improvements even over bare metal, in combination with ESXi’s powerful scheduler. In addition the role of the Kubelet agent is handled by a new ‘Spherelet’ within each ESXi host. Kubernetes uses Container Storage Interface (CSI) to integrate with vSAN and Container Network Interface (CNI) for NSX-T to handle network routing, firewall, load balancing, etc. Kubernetes namespaces are built into vSphere and backed using vSphere Resource Pools and Storage Policies.
Developers use Kubernetes APIs to access the Software Defined Data Centre (SDDC) and ultimately consume Kubernetes clusters as a service using the same application deployment tools they use currently. This service is delivered by Infrastructure Operations teams using existing vSphere tools, with the flexibility of running Kubernetes workloads and traditional Virtual Machine workloads side by side.
By applying application focused management Project Pacific allows application level control over policies, quota, and role-based access for Developers. Service features provided by vSphere such as High Availability (HA), Distributed Resource Scheduler (DRS) and vMotion can be applied at application level across Virtual Machines and containers. Unified visibility in vCenter for Kubernetes clusters, containers, and existing Virtual Machines is provided for a consistent view between Developers and Infrastructure Operations alike.
At the time of writing Project Pacific is in tech preview. This post will be updated when more information is released, continued reading can be found as follows:
- Project Pacific at Tech Field Day Extra at VMworld 2019
- Introducing Project Pacific
- Project Pacific Technical Overview
- Workload Evolution in vSphere – Project Pacific
- 5 Things to Know About Project Pacific
- Kubernetes on vSphere for Dummies
- Kubernetes Academy by VMware
- Deploying a Kubernetes Sandbox on VMware with Terraform
- How to Install Kubernetes MiniKube (Altaro)