Booting Kubernetes on Amazon Elastic Compute with kops¶

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides compute capacity in the cloud. This quickstart guide uses AWS EC2 to boot a Kubernetes cluster using kubernetes kops.

Installing kops¶

Download the latest version of kops

macOS¶

$ curl -sSL https://github.com/kubernetes/kops/releases/download/1.5.3/kops-darwin-amd64 -O
$ chmod +x kops-darwin-amd64
$ sudo mv kops-darwin-amd64 /usr/local/bin/kops

linux¶

$ curl -sSL https://github.com/kubernetes/kops/releases/download/1.5.3/kops-linux-amd64 -O
$ chmod +x kops-linux-amd64
$ sudo mv kops-linux-amd64 /usr/local/bin/kops

For more information see the official kops installation guide.

Validate kops is installed¶

$ kops version
Version 1.5.3

Install kubectl if you haven't done so yet¶

$ curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl
$ chmod +x kubectl
$ sudo mv kubectl /usr/local/bin/

Setup your AWS account¶

Install the awscli tool¶

The officially supported way of installing the tool is with pip as in

$ pip install awscli

You can also grab the tool with homebrew (for macOS users only), although this is not officially supported by AWS.

$ brew update && brew install awscli

Configure the awscli tool¶

The first thing you need to do is get valid AWS credentials out of the console. See the official documentation on how to find your SecretAccessKey and AccessKeyID.

Once you have those you can configure the awscli tool with

$ aws configure # Input your credentials here

Setting up IAM permission for kops¶

The recommended practice is to use a dedicated IAM user for kops. At a minimum kops will require the following IAM permissions to function properly.

• AmazonEC2FullAccess
  • This is used to deploy to instances in EC2
• AmazonRoute53FullAccess
  • This is used so kops can automatically create friendly DNS records for your cluster resources
• AmazonS3FullAccess
  • This is used to store meta configuration about your cluster. We will need read/write here to use S3 as a virtual filesystem in kops.
• IAMFullAccess
  • This is used because kops will create new IAM users for some of its resources. Those resources will have permissions managed securely by kops.
• AmazonVPCFullAccess
  • This used to create a VPC which serves as the foundation of all networking components in kops. Without a VPC, kops wouldn't be able to deploy any resources dependent on a network.

(Optional) Create a dedicated IAM user from the command line¶

Note: This can only be done AFTER you already have valid aws credentials in place. We will use the official kops provided convenience script to configure a new user with the following syntax: sh new-iam-user.sh $group $user

$ curl -O https://raw.githubusercontent.com/kubernetes/kops/master/hack/new-iam-user.sh
$ sh new-iam-user.sh kops-group kops-user

Note the SecretAccessKey and AccessKeyID so you can enter them in the following commands

$ aws configure # Input your credentials here
$ aws iam list-users

Configure DNS¶

In order to build a Kubernetes cluster with kops, we need to prepare somewhere to build the required DNS records. There are three scenarios below and you should choose the one that most closely matches your AWS situation.

Scenario 1a: A Domain purchased/hosted via AWS¶

If you bought your domain with AWS, then you should already have a hosted zone in Route53. If you plan to use this domain then no more work is needed.

In this example you own example.com and your records for Kubernetes would look like etcd-us-east-1c.internal.clustername.example.com

You can now skip to testing your DNS setup

Scenario 1b: A subdomain under a domain purchased/hosted via AWS¶

In this scenario you want to contain all kubernetes records under a subdomain of a domain you host in Route53. This requires creating a second hosted zone in route53, and then setting up route delegation to the new zone.

In this example you own example.com and your records for Kubernetes would look like etcd-us-east-1c.internal.clustername.kubernetes.example.com

This is copying the NS servers of your SUBDOMAIN up to the PARENT domain in Route53. To do this you should:

$ ID=$(uuidgen) && aws route53 create-hosted-zone --name subdomain.example.com --caller-reference $ID | jq .DelegationSet.NameServers

• Note your PARENT hosted zone ID

# Note: This example assumes you have jq installed locally.
aws route53 list-hosted-zones | jq '.HostedZones[] | select(.Name=="example.com.") | .Id'

• Create a new JSON file with your values (subdomain.json)

Note: The NS values here are for the SUBDOMAIN

{
  "Comment": "Create a subdomain NS record in the parent domain",
  "Changes": [
    {
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "subdomain.example.com",
        "Type": "NS",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "ns-1.awsdns-1.co.uk"
          },
          {
            "Value": "ns-2.awsdns-2.org"
          },
          {
            "Value": "ns-3.awsdns-3.com"
          },
          {
            "Value": "ns-4.awsdns-4.net"
          }
        ]
      }
    }
  ]
}

• Apply the SUBDOMAIN NS records to the PARENT hosted zone.

$ aws route53 change-resource-record-sets \
 --hosted-zone-id <parent-zone-id> \
 --change-batch file://subdomain.json

Now traffic to *.example.com will be routed to the correct subdomain hosted zone in Route53.

You can now skip to testing your DNS setup

Scenario 2: Setting up Route53 for a domain purchased with another registrar¶

If you bought your domain elsewhere, and would like to dedicate the entire domain to AWS you should follow the guide here

You can now skip to testing your DNS setup

Scenario 3: Subdomain for clusters in route53, leaving the domain at another registrar¶

If you bought your domain elsewhere, but only want to use a subdomain in AWS Route53 you must modify your registrar's NS (NameServer) records. We'll create a hosted zone in Route53, and then migrate the subdomain's NS records to your other registrar.

You might need to install jq for some of these instructions.

$ ID=$(uuidgen) && aws route53 create-hosted-zone --name subdomain.kubernetes.com --caller-reference $ID | jq .DelegationSet.NameServers

• You will now go to your registrar's page and log in. You will need to create a new SUBDOMAIN, and use the 4 NS records listed above for the new SUBDOMAIN. This MUST be done in order to use your cluster. Do NOT change your top level NS record, or you might take your site offline.

• Information on adding NS records with Godaddy.com

• Information on adding NS records with Google Cloud Platform

You can now skip to testing your DNS setup

Using Public/Private DNS (Kops 1.5+)¶

By default the assumption is that NS records are publically available. If you require private DNS records you should modify the commands we run later in this guide to include:

$ kops create cluster --dns private $NAME

Testing your DNS setup¶

You should now able to dig your domain (or subdomain) and see the AWS Name Servers on the other end.

$ dig ns subdomain.example.com

Should return something similar to:

;; ANSWER SECTION:
subdomain.example.com.        172800  IN  NS  ns-1.awsdns-1.net.
subdomain.example.com.        172800  IN  NS  ns-2.awsdns-2.org.
subdomain.example.com.        172800  IN  NS  ns-3.awsdns-3.com.
subdomain.example.com.        172800  IN  NS  ns-4.awsdns-4.co.uk.

This is a critical component of setting up clusters. If you are experiencing problems with the Kubernetes API not coming up, chances are something is wrong with the cluster's DNS.

Please DO NOT MOVE ON until you have validated your NS records!

Cluster State storage¶

In order to store the state of your cluster, and the representation of your cluster, we need to create a dedicated S3 bucket for kops to use. This bucket will become the source of truth for our cluster configuration. In this guide we'll call this bucket example-com-state-store, but you should add a custom prefix as bucket names need to be unique.

We recommend keeping the creation of this bucket confined to us-east-1, otherwise more work will be required.

$ aws s3api create-bucket --bucket prefix-example-com-state-store --region us-east-1

Note: We STRONGLY recommend versioning your S3 bucket in case you ever need to revert or recover a previous state store.

$ aws s3api put-bucket-versioning --bucket prefix-example-com-state-store  --versioning-configuration Status=Enabled

Creating your first cluster¶

Prepare local environment¶

We're ready to start creating our first cluster! Let's first setup a few environment variables to make this process easier.

$ export NAME=myfirstcluster.example.com
$ export KOPS_STATE_STORE=s3://prefix-example-com-state-store

Note: You don’t have to use environmental variables here. You can always define the values using the –name and –state flags later.

Create cluster configuration¶

We will need to note which availability zones are available to us. In this example we will be deploying our cluster to the us-west-2 region.

$ aws ec2 describe-availability-zones --region us-west-2

Below is a basic create cluster command. The below command will generate a cluster configuration, but not start building it.

$ kops create cluster \
    --zones us-west-2a \
    ${NAME}

All instances created by kops will be built within ASG (Auto Scaling Groups), which means each instance will be automatically monitored and rebuilt by AWS if it suffers any failure.

Customize Cluster Configuration¶

Now we have a cluster configuration, we can look at every aspect that defines our cluster by editing the description.

$ kops edit cluster ${NAME}

This opens your editor (as defined by $EDITOR) and allows you to edit the configuration. The configuration is loaded from the S3 bucket we created earlier, and automatically updated when we save and exit the editor.

We'll leave everything set to the defaults for now, but the rest of the kops documentation covers additional settings and configuration you can enable.

Build the Cluster¶

Now we take the final step of actually building the cluster. This'll take a while. Once it finishes you'll have to wait longer while the booted instances finish downloading Kubernetes components and reach a "ready" state.

$ kops update cluster ${NAME} --yes

Use the Cluster¶

Remember when you installed kubectl earlier? The configuration for your cluster was automatically generated and written to ~/.kube/config for you!

Optionally you can always pull the configuration with the following command:

$ kops export kubecfg --name ${NAME}

A simple Kubernetes API call can be used to check if the API is online and listening. Let's use kubectl to check the nodes.

$ kubectl get nodes

You will see a list of nodes that should match the --zones flag defined earlier. This is a great sign that your Kubernetes cluster is online and working.

Also kops ships with a handy validation tool that can be ran to ensure your cluster is working as expected.

$ kubectl cluster-info

You can look at all the system components with the following command.

$ kubectl -n kube-system get po

You are now ready to install Deis Workflow