Personal Infrastructure — CI/CD Servers

Photo by Science in HD on Unsplash

Before anything else, CI/CD is Continuous Integration and Contentious Deployment, meaning a constantly running set of actions that will run and end up deploying our application somewhere.

If you ask any engineer who has built a system that is highly complex and included a number of different services all working together, they will tell you they need one thing. An automated pipeline for building, testing, and deploying.

How to Pick the Right Solution

That’s why I’m going to walk you through setting up a CI/CD pipeline for your personal projects and personal infrastructure. There’s a lot of different tools you can use, some of the most common being:

• Jenkins — The leading open source automation server, Jenkins provides hundreds of plugins to support building, deploying and automating any project. (From my experience, Jenkins is difficult to manage long-term and not worth the trouble)
• Github Actions — GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub. Make code reviews, branch management, and issue triaging work the way you want. (Love the integration with Github, very similar to Drone and Gitlab Runners. Requires the use of Github.)
• Drone CI/CD — Drone is a self-service Continuous Integration platform for busy development teams. (High Scale, Isolated, and really easy to setup for any team. Drone is my preferred solution, and can be run anywhere with anything.)
• Gitlab Runners — Gitlab Runner is the open source project that is used to run your CI/CD jobs and send the results back to Gitlab. (Love the integration with Gitlab. Again Similar to Drone, Github Actions. This requires a bit more setup on your part though and tends to be a bit more complex than Drone. Requires use of Gitlab.)

Given these options lets add one to our Gogs instance as a CI/CD pipeline. If this is you first time setting up a CI/CD solution I recommended using Github Actions and Github as a Source Code Management Server, this keeps things together and you don’t have to manage things.

Personal Infrastructure

Since we’re using Gogs as our Source Code Management Server, I’m going to setup the CI/CD pipeline with my preferred tool, Drone CI/CD.

Setup your own Drone Server

Using the same server or workstation that you installed docker on previously, we’re going to start up a few containers that will host our Drone CI/CD Pipeline.

From your installed instance of Gogs, we’ll need to produce an Application Token:

Using the “Your Settings” page, create a application token

Copy and store the application token somewhere for use later:

Copy your token from the message blue box and store it for later

There are two main components to Drone, first a management server which will handle authentication, your repository pipelines, and secrets management. The second component of Drone is the Runners which can be setup for a couple different tools, we’ll be using Docker runners. To start these two components run these commands with the parameters that match your setup:

$ docker run \
  --volume=/var/lib/drone:/data \
  --env=DRONE_AGENTS_ENABLED=true \
  --env=DRONE_GOGS_SERVER="http(s)://YOUR_GOGS_SERVER/" \
  --env=DRONE_RPC_SECRET="GENERATED_APPLICATION_TOKEN" \
  --env=DRONE_SERVER_HOST="SERVER_IP_ADDRESS/DNS_NAME" \
  --env=DRONE_SERVER_PROTO="Pick http/https" \
  --publish=80:80 \
  --publish=443:443 \
  --restart=always \
  --detach=true \
  --name=drone \
  drone/drone:1
$ docker run -d \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -e DRONE_RPC_PROTO="Pick http/https" \
  -e DRONE_RPC_HOST="SERVER_IP_ADDRESS/DNS_NAME" \
  -e DRONE_RPC_SECRET="GENERATED_APPLICATION_TOKEN" \
  -e DRONE_RUNNER_CAPACITY=2 \
  -e DRONE_RUNNER_NAME=${HOSTNAME} \
  -p 3000:3000 \
  --restart always \
  --name runner \
  drone/drone-runner-docker:1

This will create a new server that will connect into your application, from here you’ll need to enable the webhook that connects Gogs to Drone CI/CD. Once the containers are running, go to the port that you’ve started your Drone management server on --publish=80:80 which should have you login. After logging in you’ll have the option to activate your repo with drone.

Select Activate and Activate the Repository

Once this is done, you’ll be prompted with settings for the pipeline, leave them as is and select “Activity Feed” on the top left of the drone page.

Create a Drone.yml pipeline

Next you’ll need to create a basic pipeline:

kind: pipeline
name: default
steps:
- name: build
  image: golang
  commands:
  - go mod download
  - go test ./...
  - go build ./...

There’s a lot of different things you can do with a Drone pipeline, but I’ll let you explore that on your own.

Using the above Drone pipeline commit that as a new file to your project:

Create a new Drone Pipeline config file

After you commit you should see pipelines running for your builds, and you’ll be able to track the status of your build from the drone Feed page.

Find your build history in the Activity Feed

All new commits to your project should now start a build, you can configure a lot of different things related to how you build, what secrets or tokens you use, and even if you need to manually approve the builds. You’ve now setup your own build pipeline with a Source Code Management server and a containerized Continuous Integration and Contentious Deployment server.

This completes the tutorial for setting up a Continuous Integration and Continuous Deployment server for you, I hope you found it interesting and helpful. We’ll create our own artifact repository next, so I hope you are as excited as I am.