How to use drone for continuous integration

What is Drone

Drone is a modern continuous integration and continuous delivery platform that enables busy teams to automatically build, test, and publish workflows. Teams using drone release software more frequently with fewer bug s. 1

Human words: Drone is a lightweight jenkins, which can simply realize the pipelined testing, compilation and deployment of software. And it can be easily combined with gitlab github gogs gitea.

premise

This article makes the following assumptions for readers:

  • Rich experience in git
  • Can skillfully operate some git service platforms, such as gogs, github, gitlab This paper takes gogs as an example
  • Have certain knowledge reserve and operation experience of linux and docker
  • You may also use docker compose
  • Maybe you know something about k8s

Environment configuration:

  • This article uses Gogs as an example
  • drone is the latest version: 1.8 X

Engineering documents involved in this paper: https://github.com/alex-techs/drone-tutorial

Link to this article: https://println.org/url/ODQ=

principle

Personal opinion, for reference only

Participating roles

role function
user Gogs
Drone Server Drone main service, providing Web interface
Drone Runner I understand it as an adapter that implements various operations, such as ssh, docker, and k8s operations
Drone Agent Agent that operates the Docker API of the host machine
Docker Server Doker program of host machine

install

The whole Drone system is based on docker, so there is no need to install it. Just pull down a few images. Of course, it can also be processed automatically when running.

Here we just list these images and explain their functions

  • Main service
docker pull drone/drone
  • docker operation agent
docker pull drone/agent
  • ssh runner

with ssh runner take as an example

docker pull drone/drone-runner-ssh

allocation

There is no configuration, just a few docker startup parameters.

server

parameter effect
DRONE_GOGS_SERVER GOGS server address to connect
DRONE_GIT_ALWAYS_AUTH Still authenticate when cloning public repo
DRONE_RPC_SECRET The key of the external interface of the DRONE main service, which is required for calling all interfaces
DRONE_SERVER_HOST The address that the Dragon main service listens to when starting, similar to the server_ Concept of name
DRONE_SERVER_PROTO The protocol when the Dragon main service is started, http\ \ https, not required
DRONE_DEBUG The default is false. Whether to output the debug log is optional
DRONE_PROVIDER Service provider, declared as gogs, not required
DRONE_DATABASE_DATASOURCE Declare the data source used by the main service. It is not required
DRONE_DATABASE_DRIVER Declare the database driver used by the main service. It is not required
DRONE_GOGS_SKIP_VERIFY Whether to force the use of TLS to link with gogs. The default is false. It is not required

agent

parameter effect
DRONE_RPC_SERVER DRONE_SERVER_HOST
DRONE_RPC_SECRET DRONE_RPC_SECRET
DRONE_DEBUG The default is false. Whether to output the debug log is optional
DOCKER_HOST The json api of the host docker listens to the address by default
DRONE_RPC_PROTO The protocol when the Dragon main service is started, http\ \ https

ssh-runner

parameter effect
DRONE_RPC_PROTO DRONE_SERVER_PROTO
DRONE_RPC_HOST DRONE_SERVER_HOST
DRONE_RPC_SECRET DRONE_RPC_SECRET

Other parameters: https://docs.drone.io/server/reference/

How to make docker listen to TCP port 2375

DOCKER_HOST requires this value

Edit /usr/lib/systemd/system/docker Service file

Found:

 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Amend to read:

 ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H fd:// --containerd=/run/containerd/containerd.sock

Reload service cache

systemctl daemon-reload

Restart docker

systemctl restart docker

start-up

environment variable

export GOGS_URL="http://1.1.2.3:2048"
export DRONE_HOST="0.0.0.0"
export DRONE_SECRET="xh1HJLO2yfandlwjeHdsL3Kklwheour89"
export DOCKER_HOST="tcp://`docker network inspect --format='{{range .IPAM.Config}}{{.Gateway}}{{end}}' bridge`:2375"

server

docker run -d \
    --volume=/var/lib/drone:/data \
    --env=DRONE_GOGS_SERVER=${GOGS_URL} \
    --env=DRONE_RPC_SECRET=${DRONE_SECRET} \
    --env=DRONE_SERVER_HOST=${DRONE_HOST} \
    --env=DRONE_SERVER_PROTO=http \
    --publish=3005:80 \
    --restart=always \
    --name=drone \
    drone/drone

agent

docker run -d \
    --env=DOCKER_HOST=${DOCKER_HOST} \
    --env=DRONE_RPC_SERVER=http://drone-server \
    --env=DRONE_RPC_SECRET=${DRONE_SECRET} \
    --restart=always \
    --name=drone-agent \
    --link drone:drone-server \
    drone/agent

ssh-runner

docker run -d \
  -e DRONE_RPC_HOST=drone-server \
  -e DRONE_RPC_SECRET=${DRONE_SECRET} \
  --restart always \
  --name runner \
  --link drone:drone-server \
  drone/drone-runner-ssh

Manage builds using the Web interface

If you have correctly started the above images, you can open the browser and enter IP:3005 to enter the login interface of the web management interface of the Dragon main service. The account password is dragon_ Gogs_ The account and password of the gogs server set by server.

Note: the firewall must open port 3005, or you will not be able to access Drone

Main interface:

When your Gogs has added a new project, you can use the SYNC button to synchronize it to Drone

Start building for new projects

Update: after setting public, you can view the project construction status without logging in. Private must be logged in

pit

After saving, Gogs WebHook will be automatically generated, but the generated address is not converted, and it is directly dragon_ Host splices /hook, which will cause some problems

  1. We configure the address as 0.0.0.0 so that the public network can access it. Where should we send a callback to generate such an address. We need to manually change the address that can be accessed in the browser, for example: http://xxx.org:3005/hook .

    No matter what address you configure, you should personally come to Gogs to see if it is an accessible address

  2. If we configure the Drone_ If the host is 127.0.0.1/ then it will generate such an address. If you don't pay attention to the port for the time being, there will only be multiple /. Generally speaking, one more / will not affect the effect of the request, but the Drone will be affected. It is necessary to manually remove / and change it to the correct address. For example: http://xxx.org:3005/hook .

  3. After the configuration is completed, you need to click the pencil symbol and pull it to the bottom. There is a Test Delivery. Click it to see if it can be pushed successfully.

    If so, you need to recheck your Webhook domain name configuration.

    If it is green, your configuration is correct. And drone will start the first build, but because we don't Drone YML file, so the build must fail

Project details interface

Writing Drone YML configuration file

Hello World

In your project root directory, create a new Drone YML and write the following

kind: pipeline
name: default
steps:
    -   name: Hello World
        image: centos
        commands:
            - echo Hello World
Key meaning
kind The default is pipeline
type It is not used here to mark the runner type used in this build. The default is docker, that is, Docker Runner is used by default
name Because drone supports writing multiple construction tasks at the same time, you need to give this task a name
steps Array structure, pipeline construction steps
steps.images In which docker image this step runs, the image must exist in the docker hub
steps.commands Commands that will be executed in turn during the construction process. If the command exit code is not 0, the construction of this construction will be terminated

The json version is provided to assist in understanding the data structure. Of course, you can also use json directly, because json is a legal yml format

{
    "kind": "pipeline",
    "name": "default",
    "steps": [
        {
            "name": "Hello World",
            "image": "centos",
            "commands": [
                "echo Hello World"
            ]
        }
    ]
}

In addition to yml and json, the Drone configuration file can also use jsonnet. The official example is: https://docs.drone.io/pipeline/scripting/jsonnet/

A bold attempt

cat >.drone.yml<<EOF
kind: pipeline
name: default
steps:
    -   name: Hello World
        image: centos
        commands:
            - echo Hello World
EOF

git add .drone.yml
git commit -am "first commit"
git push origin master

Gogs

Drone

Note: if your build has been waiting to start, that is, if the gear turns unsuccessfully, it will not fail. You need to check whether the agent container is started correctly and whether the main drone service can be correctly linked to the agent container. Generally speaking, the main service is not linked and is waiting for the agent.

About secrets

The role of secrecy has been slightly explained in the section "starting construction for new projects". This section describes how to use Secrets

New secret

Why should we define a value called log path instead of a real secret such as password. You'll know in a minute

Add complete

In Drone Using secrets in YML

Created when modifying Hello World Drone YML, using from_secret to get the secret value

cat >.drone.yml<<EOF
kind: pipeline
name: default
steps:
    -   name: Hello World
        image: centos
        environment:
                VARPATH: /var
            LOGPATH:
                from_secret: log-path
        commands:
            - echo $${VARPATH}
            - echo $${LOGPATH}
            - ls $${LOGPATH}
EOF

git add .drone.yml
git commit -am "first commit"
git push origin master

We have added the environment group, which is used to preset some environment variables in the running container. And several environment variables are established.

Important: VARPATH is direct plaintext, while LOGPATH uses from_secret and enter the secret key we just created on the web interface.

So you can simply understand from_secret is to read a key from the secret_ In addition to environment, secret can also be applied to any field.

Note: general shell commands use a $to obtain variables, while commands use $$because yml will parse $, and replace it with the value of the yml variable. Using two $$escapes can keep the original text.

validate

Note: because the original text cannot be seen, the example of password cannot be used to verify whether it is correct. Using a directory to ls can determine whether the secret value has been read

Runner

Pipelines help you automate steps in the software delivery process, such as starting code generation, running automated tests, and deploying to staging or production environments. Pipeline execution is triggered by the source code repository. Code changes trigger the Webhook of the Drone running the corresponding pipeline. Other common triggers include automatically scheduled or user initiated workflows. Configure the pipe by placing the file in the root directory of the git repository. The yaml syntax is designed to be easy to read and express so that anyone viewing the repository can understand the workflow.

This example uses the ssh runner just started. Please refer to the official example for the use of other runners https://docs.drone.io/pipeline/overview/

npm init
npm install helloworld

cat >.drone.yml<<EOF
kind: pipeline
type: ssh
name: default

server:
    host: xxxx.org
    user: root
    password:
        from_secret: password
  # ssh_key:
  #         from_secret: ssh_key
    
steps:
    -   name: list
        commands:
            - ls /var
    -   name: npm
        environment:
            PATH: "$$PATH:/opt/node-v14.3.0-linux-x64/bin/"
        commands:
            - pwd
            - node --version
            - npm version
            - npm install
EOF

git add .
git commit -am "first commit"
git push origin master

Note:

  1. The password we use here is obtained from the secret
  2. You can also use ssh_key, but we did not map the key file when we started ssh runner, so we used a password to demonstrate.
  3. npm init is used to convert a project into an npm project to demonstrate commands such as npm. It is not necessary
  4. The PATH setting value is my node installation directory, so that npm in commands can be executed correctly. You can change to your own node installation PATH
  5. This time, multiple step s are configured, and drone will execute these commands in turn in the host to which ssh is linked.

results of enforcement

The execution is correct and the build is complete.

Composite build task

As we mentioned in Hello World, multiple build tasks can be written. The principle is that you can use ---... To splice multiple files by using yml, which can be simply understood as Drone yml can be multiple Drone yml.

Example: put our Hello World and ssh runner tasks together and execute them at the same time.

---
kind: pipeline
name: Hello World
steps:
    -   name: Hello World
        image: centos
        environment:
            VARPATH: /var
            LOGPATH:
                from_secret: log-path
        commands:
            - echo $${VARPATH}
            - echo $${LOGPATH}
            - ls $${LOGPATH}

---

kind: pipeline
type: ssh
name: Ssh-Runner

server:
    host: xxxxx.org
    user: root
    password:
        from_secret: password

steps:
    -   name: list
        commands:
            - ls /var
    -   name: npm
        environment:
            PATH: "$$PATH:/opt/node-v14.3.0-linux-x64/bin/"
        commands:
            - pwd
            - node --version
            - npm version
            - npm install
...

Each pair, in the middle, is a construction task Indicates the end and can be omitted.

effect

Using docker compose to compose containers

This article does not explain docker compose. You can go to the official website to learn.

This file has the same effect as the ssh command in the start section. Among them, DOCKER_HOST needs to be replaced according to the actual value

cat >docker-compose.yml<<EOF
version: '3'
services:
    drone-server:
        image: drone/drone
        ports:
            - 3005:80
        volumes:
            - /var/lib/drone:/data
        environment:
            - DRONE_GOGS_SERVER=http://1.1.2.3:2048
            - DRONE_RPC_SECRET=xh1HJLO2yfandlwjeHdsL3Kklwheour89
            - DRONE_SERVER_HOST=0.0.0.0
            - DRONE_SERVER_PROTO=http

    drone-agent:
        image: drone/agent
        depends_on:
            - drone-server
        environment:
            - DRONE_RPC_SERVER=http://drone-server
            - DRONE_RPC_SECRET=xh1HJLO2yfandlwjeHdsL3Kklwheour89
            - DOCKER_HOST=tcp://172.17.0.1:2375
        links:
            - drone-server

    drone-ssh:
        image: drone/drone-runner-ssh
        depends_on:
            - drone-server
        environment:
            - DRONE_RPC_PROTO=http
            - DRONE_RPC_HOST=drone-server
            - DRONE_RPC_SECRET=xh1HJLO2yfandlwjeHdsL3Kklwheour89
        ports:
            - 3000:3000
        links:
            - drone-server
EOF

docker-compose up -d

Plug in

There are quite a lot of plug-ins available for Drone, for example: https://docs.drone.io/plugins/overview/

Unlike other software that provides api for plug-in development, the principle of the drone plug-in is to run the docker image. Automatically load environment variables or commands written in step when the container starts. You can treat the plug-in as a step.

The principle is simple and will not be repeated.

Our final code structure

.
├── .git
│   └── ...
├── node_modules
│   └── helloworld
│       └── ...
├── .drone.yml
├── package-lock.json
└── package.json

summary

We have learned the installation, configuration and basic use of drone. We have also learned how to use drone to automatically build and troubleshoot problems. We have also learned how to use "secrets". We have also learned how to use an ssh runner and perform multiple build tasks.

Now we can automatically operate the remote host after code update, update, test, compile, deploy, and realize basic continuous integration.

But Drone's ability goes far beyond that.

Tags: Docker ci docker compose

Posted by GarDouth on Mon, 30 May 2022 18:36:23 +0530