Killing it with Kops — A detailed walkthrough for deploying Kubernetes cluster on AWS with KOPS

You have an application consisting of 5 different microservices working seamlessly locally, now time to deploy them somewhere. what are your options if you want to use AWS cloud? well not much, use AWS native Kubernetes solution EKS (Elastic Kubernetes Service) or deploy Kubernetes on a bunch of EC2 instances. declare at least one as master and at least 1 as Node and manually install dependencies on each EC2 instance. looks like too much labor work right? Well, it's true… but the Pros are it's Cheap when compared to EKS.

Hmmm.. did I just say EKS is expensive? let's calculate EKS costs 0.10$ per Hour. ( does not look expensive at all right, same thought initially) This will cost you 0.10 * 24 * 30 =72 $ a month. Mind you this does not include the actual cost of surrounding resources you deploy. Even if that resource cost is there for any other method but you can save on these 72 $. heyy, but it's better than manually installing Kubernetes on EC2s.

In comes kops. Kops is Kubernetes Operations. Its the best of both worlds. easy to deploy Kubernetes on EC2 and its FREEE !!!

It does all the heavy lifting for you of installing Kubernetes on EC2 instances and scales according to need as well. pretty cool, right?

I am using a macOS. All the commands should be good to run on a mac.

Requirements:

You would need two binaries installed. kops and kubectl.

kopbrew install kubectl

make sure you have enough rights to spin up resources on AWS and you have used aws configure command to add your secrets.

That's it. we are done with requirements.

The first thing you would need is an s3 bucket to store cluster definitions into. This is where kops will look into when you run kops commands.

aws s3api create-bucket --bucket SOME-NAME --region ap-south-1

now store this name as an environment variable so you won't have to suffix it for each command.

export KOPS_STATE_STORE=s3://SOME-NAME

The domain I have is kickinpants.com, make changes to following commands as required.

let's deploy some resources.

kops create cluster \
    --name kickinpants.com.k8s.local \
    --zones "ap-south-1a" \
    --master-size t3.medium      \
    --master-count 1 \
    --node-size t3.medium \
    --node-count 2 \
    --dry-run -oyaml > infra.yaml

This will store your resource definition in a file named infra.yaml

you can (and probably should) commit this in source control. Infrastructure as code, you know.

I won't go into details about each line in infra.yaml but quickly explain what's happening here. we are asking kops to generate a Kubernetes cluster called kickinpants.com.k8s.local deploy it in ap-south-1a what and how many EC2 instances to use for master and worker nodes. (TODO: rename this when terminology changes. :P )

kops create -f infra.yaml

This creates instance groups.

to deploy other resources, run :

kops update cluster kickinpants.com.k8s.local — yes

you’d need to create ssh public key if you ever want to SSH into Ec2.

kops create secret — name kickinpants.com.k8s.local sshpublickey admin -i ~/.ssh/id_rsa.pub

Finally, we can deploy

kops update cluster kickinpants.com.k8s.local — yes

wait for 5 minutes and verify your cluster is deployed.

kops validate cluster

You will see an output as follows:

the output of kops validate cluster

great this means your cluster is ready.

you can deploy all your services, pods ingresses like a normal kubernetes cluster here onwards.

Adding NGINX

If you want to access services running within your cluster on a domain, in my case www.kickinpants.com, you would need an ingress. there are multitudes of ingress controllers available. In this write up we will be using Nginx.

We will use a classic load balancer in front of our Kubernetes service.

so the flow will be as follows:

internet → rout 53 → classic load balancer → nginx controller → service

Download the file which can deploy nginx controller in your Kubernetes cluster

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/aws/deploy-tls-termination.yaml 

here you have to make two changes.

1. Go to the EC2 → Load balancer

You would see a load balancer created by kops to interact with cluster from anywhere.

Go to the description tab at the bottom click VPC in which this load balancer is deployed. It will look something like this.

Copy the IPv4 CIDR.

Go the deploy-tls-termination.yaml the file we just downloaded. and replace proxy-real-ip-cidr value with this CIDR value.

2. You also have to replace the value for service.beta.kubernetes.io/aws-load-balancer-ssl-cert in deploy-tls-termination.yaml file.

First, make sure you have a public hosted zone created in Route 53 for the domain you wish to use.

This is the ACM certificate ARN. to get this you will have to create an ACM certificate using the domain name you want your clusters services to be accessible on.

Go to

ACM → request a public certificate → request certificate

here add your domain and a wild card entry for subdomains.

Add other domains if you wish (make sure you have hosted zones created for them) . hit next.

select DNS validation hit next.

Add tags if you want and click review

confirm and request.

It will look like above. validation can happen later (usually take <5 minutes)

Click the buttons to create records in Route 53.

It is very critical that before proceeding forward you have a validation status set to Success and its not in pending validation state.
At the bottom, There is ARN for this certificate. COpy this ARN

replace the value for service.beta.kubernetes.io/aws-load-balancer-ssl-cert in deploy-tls-termination.yaml file.

Now you are ready to apply your Nginx controller.

kubectl apply -f deploy-tls-termination.yaml

Go to the EC2 → load balancer and you will see a new load balancer is created. Route 53 is supposed to talk to this load balancer to forward requests to our cluster through nginx controller.

testing it out

we will deploy a sample stateless server to see if things are working. we will use nginx example. not to be confused with nginx controller that we created above.

create a file called deployment.yaml

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

creates a file called service.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx" # use the one that corresponds to your ingress controller.
spec:
  rules:
  - host: kickinpants.com # CHANGE THIS
    http:
      paths:
        - backend:
            serviceName: nginx-service
            servicePort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  ports:
  - port: 80
    name: http
    targetPort: 8000
  selector:
    app: nginx
---

run following commands to apply these files.

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

use kubectl get all to check the status of running Kubernetes resources. if everything looks good move onto the next steps.

connect route 53 to service.

go to rout 53 → hosted zone → yourdomain.com

click create recordset

Name: <empty>

Type: A — IPv4 Address

Alias: Yes

Alias Target: a load balancer (not the API one, but the one that talks to nginx controller)

hit create

and that's it. Wait for some time and try accessing your domain.

You have successfully deployed a hello world Kubernetes app on your domain. notice you have an SSL certificate as well.

Couple of things to note:

There are some caveats that we learned over a period of time. These can save days.

1. Never Ever Change your certificate on load balancer.
2. we have used t3.medium ec2 instances. you might use t2.micro for learning. well it will work for a while and go unresponsive eventually as you start to run something useful in it. t2.micro is good only for nginx app.
3. If you ever see an error

unexpected error during validation: error listing nodes: Get “https://api-kickinpants-com--hllh7h-786232374.ap-south-1.elb.amazonaws.com/api/v1/nodes": EOF

It means kops and kubectl are unable to talk to your cluster. one of the reasons being load balancer thinks instances are unhealthy, because of the above two errors.

4. I f you have not done either of the first two points and still not able to talk to the cluster. Then go to the Load balancer go to healthcheck.

edit health check and replace ssl with tcp

These could be some of the ways you can fix the errors if the cluster is unreachable.

I hope this guide helps you.