preface
After the review of product requirements, separate tasks, cut a release branch from the master branch, and cut the development branch of update or feature according to their respective tasks;
When developing, debugging or testing, push the code to the remote branch, and raise the merge request (hereinafter referred to as mr) to the test branch. GitLab CI automatically builds and deploys the project code to the test environment;
After the test, mr is raised to the release branch. After the development branches of the requirements are reviewed and merged, mr is raised from the release branch to the pre branch. GitLab CI automatically builds and deploys the project code to the pre production environment, and then carries out regression testing. If there is a problem, cut out the development branch from the release branch for modification, and repeat the previous process.
After the pre production environment is OK, move mr from the release branch to the master branch, and then click tag to go online. GitLab CI automatically builds and deploys the project code to the production environment, and then performs regression testing. If there is a problem, it will be released again.
At this point, the complete development process of one-time requirements has come to an end. GitLab CI helped us complete some repetitive work such as construction / deployment. I have been curious about this. Next, we will try to build a project using GitLab CI.
Build a new project
If you use GitLab CI in an existing project, you can skip this step and start from Start here
You can build the warehouse step by step according to the following steps, or you can clone the warehouse directly: gitlab-ci-example
Initialize project
New project folder
mkdir gitlab-ci-example cd gitlab-ci-example
Initialize git and npm
git init npm init -y
New project file
mkdir src build
New gitignore file
gitlab-ci-example/.gitignore
dist node_modules
New editorconfig file
gitlab-ci-example/.editorconfig
# editorconfig.org root = true [*] indent_style = space indent_size = 2 end_of_line = lf charset = utf-8 trim_trailing_whitespace = true insert_final_newline = true [*.md] trim_trailing_whitespace = false
New index HTML file
gitlab-ci-example/src/index.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Document</title>
</head>
<body>
<div id="app">
<h1>Learn Gitlab CI</h1>
</div>
</body>
</html>New main JS file
gitlab-ci-example/src/main.js
function appendElementToAPP({ tag = "div", content = "" }) {
const appEle = document.getElementById("app");
const newEle = document.createElement(tag);
newEle.innerHTML = content;
appEle.append(newEle);
}
appendElementToAPP({
tag: "div",
content: `append content by js on ${new Date().toUTCString()}`,
});Create a new webpack Dev.js file
gitlab-ci-example/build/webpack.dev.js
"use strict";
const path = require("path");
const HtmlWebpackPlugin = require("html-webpack-plugin");
const resolve = (dir) => path.resolve(__dirname, "../", dir);
module.exports = {
mode: "development",
entry: {
app: "./src/main.js",
},
output: {
path: resolve("dist"),
filename: "[name].[hash].js",
},
resolve: {
extensions: [".js"],
},
devServer: {
port: 8090,
contentBase: resolve("dist"),
},
plugins: [
new HtmlWebpackPlugin({
filename: resolve("dist/index.html"),
template: "src/index.html",
}),
],
};Create a new webpack Prod.js file
gitlab-ci-example/build/webpack.prod.js
"use strict";
const path = require("path");
const { CleanWebpackPlugin } = require("clean-webpack-plugin");
const HtmlWebpackPlugin = require("html-webpack-plugin");
const resolve = (dir) => path.resolve(__dirname, "../", dir);
module.exports = {
mode: "production",
entry: {
app: "./src/main.js",
},
output: {
path: resolve("dist"),
filename: "[name].[hash].js",
},
plugins: [
new CleanWebpackPlugin(),
new HtmlWebpackPlugin({
filename: resolve("dist/index.html"),
template: "src/index.html",
}),
],
};Create an append element JS file
gitlab-ci-example/build/append-element.js.sh
const path = require("path");
const fs = require("fs");
const cheerio = require("cheerio");
const htmlFilePath = path.resolve(__dirname, "../dist/index.html");
fs.readFile(htmlFilePath, (err, data) => {
if (err) {
return;
}
const $ = cheerio.load(data);
$("#app").append(
`<div style="color: red;">append content by build on ${new Date().toUTCString()}</div>`
);
fs.writeFileSync(htmlFilePath, $.html());
});Create a new deploy test SH file
gitlab-ci-example/build/deploy-test.sh
cp -rf dist/* /www/test/gitlab-ci-example
Modify package JSON file
gitlab-ci-example/package.json
{
"name": "gitlab-ci-example",
"version": "0.0.1",
"description": "",
"main": "index.js",
"scripts": {
"deploy-test": "build/deploy-test.sh",
"dev": "webpack-dev-server --config build/webpack.dev.js",
"build": "webpack --config build/webpack.prod.js && node build/append-element.js"
},
"keywords": [],
"author": "",
"license": "ISC"
}Install project dependencies
npm i -D cheerio webpack webpack-cli webpack-dev-server clean-webpack-plugin html-webpack-plugin
Running items
npm run dev
Open link in browser: http://localhost : 8090/, you should see:
Package project
npm run build
Open index HTML file, you should see:
At this point, the basic functions of the project have been set up. Next, start using GitLab CI in the project.
Using gitlab CIS in projects
Before using GitLab CI, you should prepare:
- One ECS
- A GitLab warehouse
Set up GitLab Runner
On the warehouse home page, click the Sidebar - Settings - CI / CD, jump to the CI / CD Settings page, expand the Runners option, and manually set the GitLab Runner according to the steps:
Installing GitLab Runner
According to the system architecture, download and install the corresponding software package, View details
# Download (package for amd64) curl -LJO https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_amd64.deb # If the download is too slow, it is recommended to copy to the remote through the scp command after the local download, such as # scp ~/gitlab-runner_amd64.deb yourUserName@yourIPAddress:/home/yourUserName # install sudo dpkg -i gitlab-runner_amd64.deb # output Selecting previously unselected package gitlab-runner. (Reading database ... 67015 files and directories currently installed.) Preparing to unpack gitlab-runner_amd64.deb ... Unpacking gitlab-runner (13.0.1) ... Setting up gitlab-runner (13.0.1) ... GitLab Runner: detected user gitlab-runner Runtime platform arch=amd64 os=linux pid=28968 revision=21cb397c version=13.0.1 gitlab-runner: Service is not installed. Runtime platform arch=amd64 os=linux pid=28975 revision=21cb397c version=13.0.1 gitlab-ci-multi-runner: Service is not installed. Runtime platform arch=amd64 os=linux pid=28993 revision=21cb397c version=13.0.1 Runtime platform arch=amd64 os=linux pid=29039 revision=21cb397c version=13.0.1 # If you receive an error like the above, run the following command. If you can output information, it means normal sudo gitlab-runner status # output Runtime platform arch=amd64 os=linux pid=29971 revision=21cb397c version=13.0.1 gitlab-runner: Service is running!
For the error information above, you can see this gitlab issue
Register gitlab runner
Start to register. Here is an example of Linux, See other systems here
# register sudo gitlab-runner register # output Runtime platform arch=amd64 os=linux pid=31237 revision=21cb397c version=13.0.1 Running in system-mode. # Specify GitLab instance URL Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/): https://gitlab.com/ # Enter the registration token (copy from project settings CI/CD settings Runners) Please enter the gitlab-ci token for this runner: JhXh7o********yDXATd # Enter description Please enter the gitlab-ci description for this runner: [hostname]: runner-001 # Enter associated label Please enter the gitlab-ci tags for this runner (comma separated): runner-001-tag # output Registering runner... succeeded runner=JhXh7oEx # Select the execution environment. The shell is selected here Please enter the executor: virtualbox, docker-ssh+machine, kubernetes, parallels, shell, ssh, docker+machine, custom, docker, docker-ssh: shell # output Runner registered successfully. Feel free to start it, but if it\'s running already the config should be automatically reloaded!
After downloading, installing and registering the Runner, return to the CI / CD Settings page. You should now be able to see the Runner associated with the project
Configuring gitlab CIS
After setting the GitLab Runner, we can start configuring gitlab CIS and creating new ones GitLab CI YML file
gitlab-ci-example/.gitlab-ci.yml
# Job name
job-test:
# stage
stage: test
# Trigger condition: when the test branch is updated
only:
- test
# Assign work to Runners with specific tags
tags:
- runner-001-tag
# script
script:
- npm install
- npm run build
- npm run deploy-testBy default, GitLab Runner will not run work without tags, so here we specify the tag set when registering GitLab Runner: runner-001-tag, View more GitLab CI/CD configuration options
If you do not want to set tags, you can modify the configuration of GitLab Runner, and check Run untagged jobs to allow GitLab Runner to run tasks without tags.
Save Gitlab ci After the YML file, push the changes to the remote warehouse
Trigger GitLab CI
After the configuration file is available, we need to trigger it. We need to cut out a test branch from the branch containing the above changes and submit it to the remote to trigger the GitLab CI (new submission and merging of test branches will trigger the CI). Of course, it is also possible to create a test branch through the graphical interface
git checkout test git push -u origin test
On the warehouse home page, click the Sidebar - CI / CD - Pipelines to view all CI records of the current warehouse, similar to the following:
Problems encountered
1. mkdir: cannot create directory '/home/gitlab-runner/builds/3-1Hb5zy': Permission denied
Running with gitlab-runner 13.0.1 (21cb397c) on runner-001 3-1Hb5zy Preparing the "shell" executor 00:00 Using Shell executor... Preparing environment 00:00 Running on xx-ubuntu... Getting source from Git repository 00:00 mkdir: cannot create directory '/home/gitlab-runner/builds/3-1Hb5zy': Permission denied Uploading artifacts for failed job 00:00 mkdir: cannot create directory '/home/gitlab-runner/builds/3-1Hb5zy': Permission denied ERROR: Job failed: exit status 1
Reason:
After the code is push ed to the remote, the above error is reported during the construction. GitLab Runner uses the GitLab Runner user during the construction. When creating the directory, you will be prompted that the permission is insufficient. Try to view the directory information:
# View file and directory information ls -alF /home/gitlab-runner # drwxr-xr-x 4 root root 4096 Jun 2 17:45 builds/
The permissions and permission groups of the current directory are root. The gitlab runner user is not under the root permission group, so he has no permission to operate.
Think about it 🤔 Something is wrong. GitLab Runner is used to build GitLab Runner, but why is the builds directory under the root permission group? Recall the operations related to the root user that have been done before. After confirmation and consulting the data, it is found that the root user (--user root) was specified during the manual installation of the service (GitLab Runner install) before this build:
# Install the service, specify the working directory, and specify that the user running the task is the root user sudo gitlab-runner install --working-directory /home/gitlab-runner --user root
How to solve:
Delete the builds directory, uninstall and reinstall the gitlab runner service, and point the user associated with the service back to the gitlab runner user
# stop service sudo gitlab-runner stop # Uninstall service sudo gitlab-runner uninstall # Reinstall service, specify working directory and user sudo gitlab-runner install --working-directory /home/gitlab-runner --user gitlab-runner # Full configuration # sudo gitlab-runner install --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner # check sudo gitlab-runner verify # Start service sudo gitlab-runner start # View status sudo gitlab-runner status
# View file and directory information again ls -alF /home/gitlab-runner # drwxrwxr-x 3 gitlab-runner gitlab-runner 4096 Jun 3 16:21 builds/
Now the permissions of the builds directory are returned to the gitlab runner user. Find the retry button associated with this work on the Pipelines or Jobs page of the gitlab warehouse, and click the button to try to run the build again
2. bash: line 92: npm: command not found
Running with gitlab-runner 13.0.1 (21cb397c) on runner-001 3-1Hb5zy Preparing the "shell" executor 00:00 Using Shell executor... Preparing environment 00:00 Running on VM-0-5-ubuntu... Getting source from Git repository 00:03 Fetching changes with git depth set to 50... Reinitialized existing Git repository in /home/gitlab-runner/builds/3-1Hb5zy/0/Lsnsh/gitlab-ci-example/.git/ Checking out 4e716630 as test... Skipping Git submodules setup Restoring cache 00:00 Downloading artifacts 00:00 Running before_script and script 00:00 $ npm install bash: line 92: npm: command not found Running after_script 00:00 Uploading artifacts for failed job 00:00 ERROR: Job failed: exit status 1
Reason:
After the service is reinstalled and the retry is built, the above error is reported because the environment of gitlab runner user is not installed with node (by default, nodes installed on root or other users cannot be accessed in the environment of gitlab runner user)
How to solve:
Log in to the server, switch to the gitlab runner user, install nvm, and then install node
# Switch to root sudo su # Log in to gitlab runner user su -l gitlab-runner # Install NVM( https://github.com/nvm-sh/nvm ), if you access script 443, try to install NVM in other ways or install node directly curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash # View nvm nvm ls # Install node (up to now, the latest LTS version is 12.18.0, and you can choose the version to install by yourself) nvm install 12.16.2 # View node and npm versions node -v npm -v
Now the gitlab runner user's environment has installed node. After happily trying to retry, he found that it was still bash: line 92: npm: command not found. For a time, he thought it was an illusion that the service did not detect the existence of node. After trying to reinstall and restart the gitlab runner service, retry still failed.
After calming down and looking up a large number of similar cases, it is preliminarily judged that the environment variables are not loaded, that is, the nvm is not loaded. Try to manually load ~/ bashrc file:
before_script: - source ~/.bashrc
After retry ing again, it still fails. Finally, it still fails at ~/ profile and ~/ Some new ideas are found in the notes in the headers of the two bashrc configuration files:
~/.profile
# ~/.profile: executed by the command interpreter for login shells. # ...
Literal translation: ~ / profile: executed by the command interpreter for the login shell.
~/.bashrc
# ~/.bashrc: executed by bash(1) for non-login shells. # ...
Literal translation: ~ / bashrc: executed by bash (1) on a non login shell.
In the above information, the login and non login states are mentioned. The configuration file will only be executed in the corresponding state. It is found by adding debugging information that ~/. will not be loaded when gitlab runner performs task construction bashrc file, only ~/ Profile file; When you log in to the server through ssh, both files will be loaded. Are you confused 🤔， This is because ~/ At the beginning of the profile file, the environment (bash) determines whether to load ~/ bashrc file, the specific code is as follows:
# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login
# exists.
# see /usr/share/doc/bash/examples/startup-files for examples.
# the files are located in the bash-doc package.
# the default umask is set in /etc/profile; for setting the umask
# for ssh logins, install and configure the libpam-umask package.
#umask 022
# if running bash
if [ -n "$BASH_VERSION" ]; then
# include .bashrc if it exists
if [ -f "$HOME/.bashrc" ]; then
. "$HOME/.bashrc"
fi
fiTo solve the problem that the npm command cannot find this problem, you need to Add the code to load nvm on the profile Profile:
# Edit profile
vi ~/.profile
# allocation nvm Load, add the following code to the configuration file( https://github.com/nvm-sh/nvm#installing-and-updating)
export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm
# After saving the configuration file, reload the configuration file
source ~/.profile3. sh: 1: build/deploy-test.sh: Permission denied
$ npm run deploy-test > gitlab-ci-example@1.0.0 deploy-test /home/ubuntu/builds/3-1Hb5zy/0/Lsnsh/gitlab-ci-example > build/deploy-test.sh sh: 1: build/deploy-test.sh: Permission denied npm ERR! code ELIFECYCLE npm ERR! errno 126 npm ERR! gitlab-ci-example@1.0.0 deploy-test: `build/deploy-test.sh` npm ERR! Exit status 126 npm ERR! npm ERR! Failed at the gitlab-ci-example@1.0.0 deploy-test script. # ...
Reason:
# In the project build directory (similar to: /home/ubuntu/builds/3-1Hb5zy/0/Lsnsh/gitlab-ci-example), view the permission information of the deployment script ls -alF ./build/deploy-test.sh # -rw-rw-r-- 1 ubuntu ubuntu 42 Jun 2 19:40 deploy-test.sh
Discover and search through the above commands Related issues , found deploy test The SH file cannot be executed because it does not have executable permissions
How to solve:
- Directly change the deploy test SH file permissions
# Indicates deploy test SH file is executable git update-index --chmod=+x ./build/deploy-test.sh # Changes will be directly entered into the staging area. The git state of the editor may indicate that there are new changes. After ignoring and directly submitting this change, the git state status will be updated git commit -m 'Make build.sh executable' # Submit to remote, master, test or other branches git push
- Execute deploy-test sh file
# package.json - "deploy-test": "build/deploy-test.sh", + "deploy-test": "sh build/deploy-test.sh",
4. /www/test/gitlab-ci-example: No such file or directory
$ npm run deploy-test > gitlab-ci-example@1.0.0 deploy-test /home/gitlab-runner/builds/3-1Hb5zy/0/Lsnsh/gitlab-ci-example > build/deploy-test.sh /www/test/gitlab-ci-example: No such file or directory Please create this directory and then assign the directory permissions to the gitlab-runner user. You can execute the following command as root: mkdir /www/test/gitlab-ci-example chown gitlab-runner /www/test/gitlab-ci-example
Reason:
Build/deploy-test The SH script will copy the built code to the /www/test/gitlab-ci-example directory. Therefore, you need to create this directory before building
How to solve:
Refer to build/deploy-test For the prompt information printed in the SH script, create a new directory and assign directory permissions:
# Create a new directory (use root or other gitlab runner users) mkdir /www/test/gitlab-ci-example # Assign gitlab runner user folder permissions chown gitlab-runner /www/test/gitlab-ci-example
summary
At this point, the CI can finally run through. The content of the deployed page is as follows, Click to view:
Adopted Gitlab ci YML configuration file, you can handle it in various stages of construction, for example, you can_ Script and after_ The script phase calls the nailing robot interface to synchronize the deployment status to individuals / groups in time:
before_script: # Nail notification nail group - curl -X POST 'https://oapi.dingtalk.com/robot/send?access_token=xxx&xxxx'
The notification is similar to the following:
More about Gitlab ci For the configuration information of the YML file, see Official documents
Making good use of CI / CD will greatly improve team collaboration and development efficiency. Everything is difficult at the beginning. There must be resistance at first. After crossing this barrier, there is another barrier waiting for you [manual dog head]
The warehouse link of the sample project is as follows. Welcome to star 🌟:
github repository (template): https://github.com/Lsnsh/gitl...
gitlab warehouse: https://gitlab.com/Lsnsh/gitl...
Command summary
# View gitlab runner related processes ps aux|grep gitlab-runner # Register gitlab runner sudo gitlab-runner register # Reinstall gitlab runner service # stop service sudo gitlab-runner stop # Uninstall service sudo gitlab-runner uninstall # Reinstall service, specify working directory and user sudo gitlab-runner install --working-directory /home/gitlab-runner --user gitlab-runner # Full configuration # sudo gitlab-runner install --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner # check sudo gitlab-runner verify # Start service sudo gitlab-runner start # View status sudo gitlab-runner status # Copy files to remote host # scp ~/gitlab-runner_amd64.deb yourUserName@yourIPAddress:/home/yourUserName # eg: (copy the file ~/gitlab-runner_amd64.deb to the root directory of the remote host with the public IP of 110.120.130) scp ~/gitlab-runner_amd64.deb root@110.120.130:/home/root # View file and directory information # eg: (view the /home/gitlab-runner directory) ls -alF /home/gitlab-runner # Switch to root sudo su # root user logs in to another user # eg: (log in to gitlab runner user) su -l gitlab-runner # Indicates that the file is executable # Git update index --chmod=+x file path # Indicates that the file is not executable # Git update index --chmod=-x file path # eg: (indicates that the./build/deploy-test.sh file is executable) git update-index --chmod=+x ./build/deploy-test.sh # Assign file or folder permissions to users # chown username file or folder path # eg: (assign gitlab runner user /www/test/gitlab CI example folder permission) chown gitlab-runner /www/test/gitlab-ci-example