Stateful Containers on Kubernetes using Persistent Volume and Amazon EBS

This blog will show how to create stateful containers in Kubernetes using Amazon EBS. Couchbase is a stateful container. This means that state of the container needs to be carried with it.In Kubernetes, the smallest atomic unit of running a container is a pod. So a Couchbase container will run as a pod. And by default, all data stored in Couchbase is stored on the same host.

This figure is originally explained in Kubernetes Cluster on Amazon and Expose Couchbase Service . In addition, thisfigure shows storage local to the host. Pods are ephemeral and may be restarted on a different host.A Kubernetes Volume outlives any containers that run within the pod, and data is preserved across container restarts. However the volume will cease to exist when a pod ceases to exist. This is solved by Persistent Volumes that provide persistent, cluster-scoped storage for applications that require long lived data.

Creating and using a persistent volume is a three step process:

Provision : Administrator provision a networked storage in the cluster, such asAWS ElasticBlockStore volumes. This is called as PersistentVolume . Request storage : User requests storage for pods by using claims . Claims can specify levels of resources (CPU and memory), specific sizes and access modes (e.g. can be mounted once read/write or many times write only). This is called as PersistentVolumeClaim . Use claim : Claims are mounted as volumes and used in pods for storage.

Specifically, this blog will show how to use anAWS ElasticBlockStore as PersistentVolume , create a PersistentVolumeClaim , and then claim it in a pod.

Complete source code for this blog is at: github.com/arun-gupta/couchbase-kubernetes .

Provision AWS Elastic Block Storage Following restrictions need to be met if Amazon ElasticBlockStorageis used as a PersistentVolume with Kubernetes: the nodes on which pods are running must be AWS EC2 instances those instances need to be in the same region and availability-zone as the EBS volume EBS only supports a single EC2 instance mounting a volume Create an AWS Elastic Block Storage: aws ec2 create-volume --region us-west-2 --availability-zone us-west-2a --size 5 --volume-type gp2 The region us-west-2 region and us-west-2a availability zone is used here. And so Kubernetes cluster need to start in the same region and availability zone as well. This shows the output as: { "AvailabilityZone": "us-west-2a", "Encrypted": false, "VolumeType": "gp2", "VolumeId": "vol-47f59cce", "State": "creating", "Iops": 100, "SnapshotId": "", "CreateTime": "2016-07-29T21:57:43.343Z", "Size": 5 } Check if the volume is available as:

aws --region us-west-2 ec2 describe-volumes --volume-id vol-47f59cce

It shows the output as:

{ "Volumes": [ { "AvailabilityZone": "us-west-2a", "Attachments": [], "Encrypted": false, "VolumeType": "gp2", "VolumeId": "vol-47f59cce", "State": "available", "Iops": 100, "SnapshotId": "", "CreateTime": "2016-07-29T21:57:43.343Z", "Size": 5 } ] }

Note the unique identifier for the volume in VolumeId attribute. You can also verify theEBS block in AWS Console:

Start Kubernetes Cluster Download Kubernetes 1.3.3 , untar it andstart the cluster on Amazon: export KUBERNETES_PROVIDER=aws KUBE_AWS_ZONE=us-west-2a NODE_SIZE=m3.large NUM_NODES=3 ./kubernetes/cluster/kube-up.sh Three points to note here: Zone in which the cluster is started is explicitly setto us-west-1a .This matches the zone where EBS storage volume was created. By default, each node size is m3.medium . Here is issetto m3.large . By default, 1 master and 4 worker nodes are created. Here only 3 worker nodes are created. This will show the output as: ... Starting cluster in us-west-2a using provider aws ... calling verify-prereqs ... calling kube-up Starting cluster using os distro: jessie Uploading to Amazon S3 +++ Staging server tars to S3 Storage: kubernetes-staging-0eaf81fbc51209dd47c13b6d8b424149/devel upload: ../../../../../var/folders/81/ttv4n16x7p390cttrm_675y00000gn/T/kubernetes.XXXXXX.ISohbaGM/s3/bootstrap-script to s3://kubernetes-staging-0eaf81fbc51209dd47c13b6d8b424149/devel/bootstrap-script Uploaded server tars: SERVER_BINARY_TAR_URL: https://s3.amazonaws.com/kubernetes-staging-0eaf81fbc51209dd47c13b6d8b424149/devel/kubernetes-server-linux-amd64.tar.gz SALT_TAR_URL: https://s3.amazonaws.com/kubernetes-staging-0eaf81fbc51209dd47c13b6d8b424149/devel/kubernetes-salt.tar.gz BOOTSTRAP_SCRIPT_URL: https://s3.amazonaws.com/kubernetes-staging-0eaf81fbc51209dd47c13b6d8b424149/devel/bootstrap-script INSTANCEPROFILE arn:aws:iam::598307997273:instance-profile/kubernetes-master 2016-07-29T15:13:35Z AIPAJF3XKLNKOXOTQOCTkubernetes-master / ROLES arn:aws:iam::598307997273:role/kubernetes-master 2016-07-29T15:13:33Z / AROAI3Q2KFBD5PCKRXCRM kubernetes-master ASSUMEROLEPOLICYDOCUMENT 2012-10-17 STATEMENT sts:AssumeRole Allow PRINCIPAL ec2.amazonaws.com INSTANCEPROFILE arn:aws:iam::598307997273:instance-profile/kubernetes-minion 2016-07-29T15:13:39Z AIPAIYSH5DJA4UPQIP4Bkubernetes-minion / ROLES arn:aws:iam::598307997273:role/kubernetes-minion 2016-07-29T15:13:37Z / AROAIQ57MPQYSHRPQCT2Q kubernetes-minion ASSUMEROLEPOLICYDOCUMENT 2012-10-17 STATEMENT sts:AssumeRole Allow PRINCIPAL ec2.amazonaws.com Using SSH key with (AWS) fingerprint: SHA256:dX/5wpWuUxYar2NFuGwiZuRiydiZCyx4DGoZ5/jL/j8 Creating vpc. Adding tag to vpc-fa3d6c9e: Name=kubernetes-vpc Adding tag to vpc-fa3d6c9e: KubernetesCluster=kubernetes Using VPC vpc-fa3d6c9e Adding tag to dopt-3aad625e: Name=kubernetes-dhcp-option-set Adding tag to dopt-3aad625e: KubernetesCluster=kubernetes Using DHCP option set dopt-3aad625e Creating subnet. Adding tag to subnet-e11f5985: KubernetesCluster=kubernetes Using subnet subnet-e11f5985 Creating Internet Gateway. Using Internet Gateway igw-5c748f38 Associating route table. Creating route table Adding tag to rtb-84fcf1e0: KubernetesCluster=kubernetes Associating route table rtb-84fcf1e0 to subnet subnet-e11f5985 Adding route to route table rtb-84fcf1e0 Using Route Table rtb-84fcf1e0 Creating master security group. Creating security group kubernetes-master-kubernetes. Adding tag to sg-91590bf7: KubernetesCluster=kubernetes Creating minion security group. Creating security group kubernetes-minion-kubernetes. Adding tag to sg-9d590bfb: KubernetesCluster=kubernetes Using master security group: kubernetes-master-kubernetes sg-91590bf7 Using minion security group: kubernetes-minion-kuber