April 2021
External Resources
-Jira -Risk Assessment -
Deployment of a simple Flask application, built in Python, that makes use of a microservice architecture comprising of 2 separate services and utilises a CI Pipeline.
- Jira board created with full expansion on tasks, providing a record of any issues or risks that were faced.
- Deploy the application using containerisation (Docker) and orchestration (Kubernetes) tools.
- Test application through the CI pipeline with Jenkins.
- Make use of two managed Database Servers: 1 for Testing and 1 for Production with AWS RDS.
- Use Github webhooks so that Jenkins recreates and redeploys the application when a change to the code base has been made.
- Configure infrastructure using an infrastructure management tool (Terraform).
- Create an Ansible Playbook that will provision the environment that the CI Server needs to run.
- Use a reverse proxy (NGINX) to make the application accessible to the user.
- Deployment on AWS EC2 instances.
- Fully integrate into a VCS Git.
Project planning was done in Jira. The Jira board can be found above in the external resources section. Tasks were categorised into epics and put into respective sprints. The Kanban board was used to track progress. Below is an example Sprint Kanban board as the project was nearing completion:
Below is the risk assessment that was conducted for the project:
Below is the architecture diagram for the project:
- The developer pushes to GitHub.
- This then triggers the Jenkins CI through an SCM webhook.
- The CI server runs through the stages necessary for the build.
- Removes Old Repo and Clones New.
- Swap backend Dockerfile with Secret Vars Dockerfile.
- Clean Docker - Remove all previous images/containers.
- Build App Docker Images.
- Run Docker Images.
- Run Pytest.
- Export Test Coverage
- Push images to Dockerhub.
- If tests are successful, pull from Dockerhub and deploy the app.
- The management VM contains the tools Terraform used to provision the VMs and RDSs, and Ansible using to install the software onto the previsioned VMs. This was the initial stage of the project and the section below goes into more depth on this.
This project utilised AWS cloud resources. A network diagram was created to get a clear idea of how the resources should be provisioned. This was then coded in terraform (mentioned below).
Terraform is an infrastructure management tool that allows the provision of AWS (and many other) cloud resources through code - as opposed to running commands on the command line.
In this project, Terraform was extensively used to provision the 2 EC2s, 2 RDSs and the networking that's required.
Terraform uses modules, they are templates for multiple resources that are used together.
Below is a snippet of the Terraform plan output:
Ansible is another infrastructure as code tool that enables software installations as code. Instead of the command line, Ansible allows installations via the playbook.
The playbook specifies the roles which contain the tasks that Ansible would perform in the host.
For this project, 3 roles were used, common - to run tasks on both hosts, ci_vm - to run tasks that are specific to the Jenkins server, and the test_vm role - for tasks that would need to run only on the Test VM.
Here is a list of the tasks that are inside the roles. The tasks shown below are running a set of scripts for installations.
These are the scripts that are run on the VMs to install the software.
CI means Continuous Integration and CD means Continuous Delivery and Continuous Deployment. This is the principle of automation that, for this project, has been achieved with the tool Jenkins.
Along with the code, a Jenkinsfile needs to be pushed to the repository.
When Jenkins receives a webhook trigger from a push to Github, the build starts.
The Jenkinsfile is run and the works through the stages.
In this project, there were 9 total stages of the CI/CD pipeline.
- Removes Old Repo and Clones New.
- Swap backend Dockerfile with Secret Vars Dockerfile.
- Clean Docker - Remove all previous images/containers.
- Build App Docker Images.
- Run Docker Images.
- Run Pytest.
- Export Test Coverage
- Push images to Dockerhub.
- If tests are successful, pull from Dockerhub and deploy the app.
Different stages access different scripts that are run on the respective VMs. In the case below, Jenkins is providing the script for the Test VM to run, in order to build the image.
For the Jenkins pipeline, the app is not deployed in the Kubernetes cluster so that phase of the build has not passed. This shows that CI was successful, but it has not deployed yet.
From the second stage, Docker was used to containerise the application ready for testing. This is explained further in the section below.
For this project, Docker was used to containerise the flask application. The app consisted of both the front-end and back-end and both parts needed to be put in a docker container to mitigate "dependency hell". It allows devs to send code to other teams inside of a container with all the required dependencies, which in theory means the software that runs on your computer will be the software that runs on another person’s computer.
The provided application needed to be containerised. This was done through a Dockerfile. The Dockerfile for the front end is shown below. The app is installed in a python environment along with all the requirements for the apps to run.
Kubernetes is a container orchestration tool that deploys Docker images automatically, running the containers in a cluster. Kubernetes also allows for features like auto-scaling and rolling updates.
Below is the Kubernetes file structure used for this project.
As with the tools above, Kubernetes resources are created in a declarative way.
For an app to be deployed in Kubernetes, there has to be a deployment YAML file that determines the number of replicas, the image that's being pulled from Dockerhub, as well as ports and namespaces.
There also needs to be a file that has information about the service. The service is an abstraction that describes a group of pods of an app.
The containers will run with 3 replicas (pods), and those replicas are replaceable (when one goes down another starts up), therefore you would require a service that will have a non-changing DNS name for connections which are then routed to the pods automatically.
NGINX was also used in this project to act as the load-balancer and reverse-proxy for end-user access/interaction with the app. This was also run as a pod with multiple replicas.
The tests were run as part of the CI pipeline. It was essential that the tests were run before the image was pushed to Dockerhub, as you wouldn't want to push the wrong image. Below is an output of the pytest that was run, for both front and backend. Both are run with 100% coverage and passed tests, so they are ready for pushing to Dockerhub.
Frontend
Backend
Manish Reddy
I would like to acknowledge:
- The trainers at QA for supporting me through this project.
- My team for their moral support.