Lab 13 - Custom Container

In this lab, you're going to create a custom container running a new Tiger Enterprises Web API and run it in your Kubernetes/Docker cluster.

Docker Host Cleanup

The custom containers you create in this lab will be run on the Docker host you created in a previous lab. Let's do some cleanup on that Docker host now, to avoid running out of disk space or other resources later.

First, ensure that you removed the MSSQL server that AWS installed on the Ubuntu image we've been using:

$ sudo apt -y purge mssql* msodbc*

Then, ensure that the Wordpress and Plone-related containers are been stopped, and their images deleted, from the previous lab. We don't need them any longer:

# Look at the containers that still exist
$ docker container ls --all

# Stop any containers you may have left running
$ docker stop WordPress MySQL ........

# Remove the containers that are no longer needed
$ docker rm WordPress MySQL ..........

# Look at the images that still exist
$ docker images

# Remove the image files that are no longer needed
$ docker rmi plone wordpress mysql

# Check the disk space available for the "/" (root) partition
$ df -h

Part 1 - Custom Container (Create Manually)

Web API Requirements, v1

Write a web API program - written in any programming language of your choice, using any libraries of your choice - that runs on a Linux Docker container. Your API should be reachable over port 8080 and provide the following interfaces that return data in JavaScript Object Notation (JSON):

Path	Description
/	Returns the string Tiger Enterprises Web API v1.0
/aws-ip	Returns the IPv4 address of the AWS EC2 node that the container is running on. This value should obtained when the web API is loaded and then saved for future accesses. You might use the aws cli to obtain this information, or via the link-local interface at http://169.254.169.254/latest/meta-data/public-ipv4/ Example: { "aws-ip":"10.11.12.13" }
/container-ip	Returns the IPv4 address of the container interface (if there are multiple IP addresses, return the IP address of the interface that is the default route to the Internet). This value should obtained when the web API is loaded and then saved for future accesses. Example: { "container-ip":"172.16.10.11" }
/container-hostname	Returns the hostname of the container. This value should be obtained when the web API is loaded and then saved for future accesses. Example: { "container-hostname":"containerXYZ" }
/all	Returns a JSON object with all of the above values Example: { "aws-ip":"10.11.12.13", "container-ip":"172.16.10.11", "container-hostname":"containerXYZ" }

Container Creation (Manually)

Your web API application should be packaged as a custom Docker image that can be run in the cluster. You should create this image by starting from an existing container, adding the application and associated configuration to it, and capturing the results as a new image.

References:

• Create a New Docker Image for My Application

To create a new image from an existing container, go to Docker Hub and find a clean Linux image to start with. For example, you might choose Ubuntu because its tools and available packages are familiar to you, or you might choose alpine because it is tiny (5MB!) and can allow you to make very lightweight images.

Pull your desired IMAGE. You may want to specify a TAG in order to get a specific version. You can see all available tags from Docker Hub (click on "View Available Tags")

$ docker pull IMAGE:TAG

Run an instance of the image you just downloaded. Go ahead and publish port 8080 externally and internally, so you have a way to access your web API once it is running.

$ docker run -dit -p 8080:8080 --name="tiger-api-dev" IMAGE:TAG
# -d            : Run in detached mode
# -i            : Run in interactive mode with a teletype
# -t            : Run with a teletype (for console)
# -p 8080:8080  : Publish port 8080 externally to port 8080 inside the container
# --name        : Specify a human-friendly name to identify the container. Otherwise, a UUID is assigned

Verify that the container is running:

$ docker ps
# Should see your "tiger-api-dev" container running here

Get a Bash shell and start working inside your container.

$ docker exec -it tiger-api-dev /bin/bash
# -i         : Interactive mode
# -t         : Allocate a TTY (for a shell)
# /bin/bash: : The command (program) to run inside the container 

When finished with your Bash shell, just type exit in the container to return to the host shell.

Test your API using Curl at the docker host command line (outside of the container):

$ curl http://localhost:8080/ 
$ curl http://localhost:8080/aws-ip
$ curl http://localhost:8080/container-ip
$ curl http://localhost:8080/container-hostname
$ curl http://localhost:8080/all

Test your API using the web browser on your own computer. First ensure that port 8080 is open through the AWS security group for the docker host, and then load http://docker.STUDENT-NAME.tigerenterprises.org:8080/all

Once you are completely happy with version 1 of your API, go ahead and capture the current running container as a new image. Here, container-name is the name of the currently running container (tiger-api-dev) and image name is a string of the format [username]/[imagename]:[tags]

# Login to Docker Hub 
# (create an account at https://hub.docker.com/ first)
$ docker login

# Capture the currently-running container
# docker commit -m "Message" -a "Author Name" [container-name] [image-name]
$ docker commit -m "TigerAPI v1" -a "STUDENT-NAME" tiger-api-dev your-docker-id/tiger-api:v1
# -m      : Human readable commit message
# -a      : Human readable Author Name
# [image name] takes the format your-docker-id/image-name:tag

Push that image to Docker Hub:

$ docker push your-docker-id/tiger-api:v1

Check if the image is available on the local system:

$ docker images

Stop the current running container:

$ docker stop tiger-api-dev

To verify that everything is 100% correct, start a new container based on the image you just captured:

$ docker run -dit -p 8080:8080 --name="tiger-api" your-docker-id/tiger-api:v1

# Verify container is running
$ docker ps
# Should see your-docker-id/tiger-api:v1  as a running image with name "tiger-api"

# Verify that your web API is functional. You may need to run your application within the container with `docker exec`. 
# (Use CURL or your web browser)
# $ curl http://localhost:8080/all
# { 
#     "aws-ip":"52.90.22.253",
#     "container-ip":"172.17.0.2",
#     "container-hostname":"5fc4a977983f"
# }

# Stop the container when finished with testing
$ docker stop tiger-api

Part 2 - Custom Container (Create with Dockerfile)

Web API Requirements, v2

Your application requirements have grown! Such is life. An additional API is now needed:

Path	Description
/local-count	Returns an incrementing counter that is locally unique to the particular instance of the API. The counter should start at 0 when an API server is launched. Example: { "local-count": 125 }
/all	Returns a JSON object with all of the API values. Note that the same incrementing counter should appear here as well. Example: { "aws-ip":"10.11.12.13", "container-ip":"172.16.10.11", "container-hostname":"containerXYZ", "local-count":125 }

Container Creation (Dockerfile)

For this new version of your web API, the custom Docker image should be created by way of a Dockerfile, which automates the process that you previously did manually.

First, create your custom Dockerfile.

$ nano Dockerfile

In the Dockerfile, you should:

• Start from a clean base image using FROM
• Install any libraries or language interpreters or any other programs you need using RUN
• Expose any network ports needed using EXPOSE
• Set any environment variables needed using ENV
• Copy in your program code using COPY
  • Tip: Place your custom API files in the same directory as Dockerfile or in a subdirectory, NOT higher in the directory tree
• Automatically run the web API server when the container is launched using CMD as the final line of the file

Reference:

• Dockerfile reference
• Best practices for writing Dockerfiles
• What is a Dockerfile?

Once your Dockerfile is ready, build it to create v2 of your API image. Don't omit the period at the end - that's the argument that tells docker build to use the current directory as its starting point.

$ docker build -f Dockerfile -t your-docker-id/tiger-api:v2 .

To verify that everything is 100% correct, start a new container based on the image you just captured:

$ docker run -dit -p 8080:8080 --name="tiger-api-v2" your-docker-ID/tiger-api:v2

# Verify container is running
$ docker ps
# Should see your-docker-ID/tiger-api:v2  as a running image with name "tiger-api-v2"

# Verify that your web API is functional. Use CURL or your web browser.
# $ curl http://localhost:8080/all
# { 
#     "aws-ip":"52.90.22.253",
#     "container-ip":"172.17.0.2",
#     "container-hostname":"5fc4a977983f",
#     "local-count": 5
# }

# Stop the container when finished with testing
$ docker stop tiger-api-v2

Push that new image to Docker Hub:

$ docker push your-docker-id/tiger-api:v2

Troubleshooting

Let's say that your Web API had a coding error that causes it to crash immediately when run in the container. Or your Dockerfile had an error. Perhaps instead of the line CMD ["flask", "run", "--host=0.0.0.0", "--port=8080"] (if using Python & Flask), you forgot the required commas between the input arguments. Or you set an environment variable incorrectly, such as ENV FLASK_APP=/typo/myapi.py, resulting in Flask being unable to locate your application code. Numerous minor issues can result in a nonfunctioning program.

What you'll observe in execution is:

# Run your image as a container
$ docker run -dit -p 8080:8080 --name="tiger-api-v2" your-docker-ID/tiger-api:v2

# See if it's active
$ docker ps
# [NOTHING APPEARS!?!?!]
# [Considers switching major to philosophy]

# See if the container ran BRIEFLY
$ docker ps --all
CONTAINER ID        IMAGE                         COMMAND                  CREATED              STATUS                          PORTS               NAMES
ea1506d49529        your-docker-ID/tiger-api:v2   "flask run --host=0.…"   About a minute ago   Exited (2) About a minute ago                       tiger-api-v2
# ... older containers

At this point, you see that the container did briefly exist, but it exited because the application it was trying to run exited. What went wrong in your container with your program? The instructor is unlikely to know.

You can try to restart your container using docker container start --attach --interactive tiger-api-v2. The -ai (attach, interactive) arguments will normally give you interactive shell access, but before you can get to the shell, the same (failing) command will run first and lead to the container exiting. You may, however, see some helpful debug messages printed before that happens. (e.g. "Unable to find python3", "ModuleNotFoundError: No Module named 'typo'", "Could not import /typo/myapi.py", etc.)

If you need interactive shell access to find out exactly what has gone wrong in your container, try this technique to override the default container behavior with an entrypoint you know is valid:

# Remove the failing container
docker rm tiger-api-v2

# Create a new container from the same BAD image.
# But this time with --entrypoint=/bin/bash -s   specified
# Which will run bash with NO ARGUMENTS afterwards
# This should give you an interactive shell.
# YES, the -s needs to be at the END, after the image ID.
docker run -it -p 8080:8080 --name="tiger-api-v2" --entrypoint /bin/bash your-docker-ID/tiger-api:v2 -s

# Now you should be inside your container at the shell.
# Look around and troubleshoot.

Failing all that, you can return to the Dockerfile and edit it so that, instead of running your custom API at launch, it instead runs /bin/bash. Rebuild and re-run your container. Then your container will start and stay running, you can attach to it, and interactively try to debug your program.

Part 3 - Deploy to Kubernetes

Deploy your new Tiger Web API to your Kubernetes cluster!

First, give Kubernetes your Docker Hub login information. This allows you to access your private repositories (if you set tiger-api to private), and gets you out of the rate limiting public tier restrictions. (A free account is still rate limited, just not as badly)

References:

• Pull an Image from a Private Registry
• How to pull image from dockerhub in kubernetes?

k8s-controller$ kubectl create secret docker-registry dockerhub-cred \
--docker-username=YOUR-USERNAME \
--docker-password=YOUR-PASSWORD \
--docker-email=YOUR-EMAIL \
-n default
# "Namespace" is the default (as opposed to kube-system, kubernetes-dashboard, or any custom namespace)...

Then, create a new YAML file that defines the tiger API deployment and replica set. You should have four replicas running across your existing cluster of 2 worker nodes.

k8s-controller$ nano tiger-api.yaml

Contents of YAML file. Be sure to update YOUR-DOCKER-ID. The file is similar to the one used in the Kubernetes lab, but with the addition of imagePullPolicy and imagePullSecrets

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: tiger-api
  name: tiger-api-deployment
spec:
  replicas: 4
  selector:
    matchLabels:
      app.kubernetes.io/name: tiger-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: tiger-api
    spec:
      containers:
      - image: YOUR-DOCKER-ID/tiger-api:v2
        name: tiger-api
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
      imagePullSecrets:
      - name: dockerhub-cred

Now, ensure that your cluster is running, both the controller node and the two worker nodes. You can confirm via kubectl top nodes. Once the cluster is running, apply that YAML file to the cluster:

k8s-controller$ kubectl apply -f tiger-api.yaml

Get information on the Deployment:

k8s-controller$ kubectl get deployments tiger-api-deployment

# Example Output:
# NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
# tiger-api-deployment   4/4     4            4           20s

# Troubleshooting:  Deployment never becomes ready?  Check the logs
# $ kubectl logs deployment/tiger-api-deployment

Create a Service object that "exposes" the Deployment.

k8s-controller$ kubectl expose deployment tiger-api-deployment --type=LoadBalancer --name=tiger-api-service

Display information about the Service:

k8s-controller$ kubectl get services tiger-api-service
# Example Output:
# NAME                TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
# tiger-api-service   LoadBalancer   10.99.1.100   <pending>     8080:31298/TCP   7s

Verify that your load balancer is connecting to your API by loading it repeatedly via curl:

# From the Controller:
k8s-controller$ CLUSTER_IP=AAA.BBB.CCC.DDD    # Replace with your CLUSTER-IP from "kubectl get services"
k8s-controller$ curl http://$CLUSTER_IP:8080/all
k8s-controller$ for i in {1..10}; do curl http://$CLUSTER_IP:8080/all; done;
# Do you see the TWO aws-ip values?
# Do the AWS IP addresses match the public IPs for K8s-Worker-01 and -02 shown in the AWS web console?
# Is the local-count incrementing as expected? (within each container)

When finished, to clean up, first delete the Service:

k8s-controller$ kubectl delete services tiger-api-service

Then delete the Deployment, ReplicaSet, and the Pods that are running the Tiger API application:

k8s-controller$ kubectl delete deployment tiger-api-deployment

Troubleshooting

To verify that your Docker credential is correctly loaded into Kubernetes, use the following debugging commands:

# Check to see if "dockerhub-cred" exists
k8s-controller$ kubectl get secret
NAME                  TYPE                                  DATA   AGE
default-token-rkz4l   kubernetes.io/service-account-token   3      24d
dockerhub-cred        kubernetes.io/dockerconfigjson        1      24d

# Get the credential info that is set for "dockerhub-cred"
# Verify that your username, password, and email is correct
k8s-controller$ kubectl get secret dockerhub-cred --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode
{"auths":{"https://index.docker.io/v1/":{"username":"YOUR-USERNAME","password":"YOUR-PASSWORD","email":"YOUR-EMAIL","auth":"c2hhZmVyamE6VWpOakFBZ1ZQN3paQng2RXg4TnI5S3N1cW1pcDRw"}}}

Lab Deliverables

Upload to the Lab 13 Canvas assignment all the lab deliverables to demonstrate your work:

• Part 1 - Custom Container (Manually)
  • Submit a screenshot of Docker Hub, showing that the your-docker-id/tiger-api repository exists
  • Submit a screenshot of your web browser, on your home computer, accessing http://docker.STUDENT-NAME.tigerenterprises.org:8080/all with the Web API output visible
• Part 2 - Custom Container (Dockerfile)
  • Submit a screenshot of Docker Hub, showing that the your-docker-id/tiger-api repository now contains two tags: v1 and v2
  • Submit a screenshot of Docker Hub -> Repositories -> your-docker-id/tiger-api -> Tags -> v2, showing the image layers that correspond to the commands in your Dockerfile
  • Submit a screenshot of your web browser, on your home computer, accessing http://docker.STUDENT-NAME.tigerenterprises.org:8080/all with the Web API output visible (including the new local-count API)
  • Submit the source code and configuration files (if any) for your final v2 Web API program
  • Submit the Dockerfile
• Part 3 - Deploy to Kubernetes
  • Submit a screenshot of kubectl get services tiger-api-service
  • Submit a screenshot of your repeated curl attempts demonstrating that your API is running on multiple containers (all accessible via the same load balancer IP)
  • Submit a screenshot of your Kubernetes dashboard showing your running deployment and running service.