More Concise Pipeline Definition and Cross-Project Reuse¶

Gems and Jewels to Collect¶

In this episode we will discuss ways to take a CI pipeline apart and extract parts of the pipeline into separate YAML files that can be included into the main .gitlab-ci.yml YAML file or even reused in other projects in the same or different GitLab instances. Finally, we will show you how to benefit from existing GitLab CI/CD Templates and include these in your pipeline.

Introduction¶

With time your CI pipeline might become large with several hundred lines of code and the pipeline might lack some structuring. You may split it up into multiple YAML files to give it a structure that is easier to handle and to arrange it more clearly. These YAML files can also be reused in different places, i.e. projects in the same or even in different GitLab instances.

`include` YAML files¶

The include keyword used here is the keyword of your choice if you want to include YAML files into another YAML file. Imagine you have defined a YAML file that appears to be useful for many Python projects: An example could be a YAML file lint.gitlab-ci.yml that is linting Python code and is put into directory .gitlab/ci/ of your Python project:

.prepare_linting:
  image: python:3.11
  stage: lint
  before_script:
    - pip install --upgrade pip
    - pip install poetry
    - poetry install

license-compliance:
  extends: .prepare_linting
  script:
    - poetry run reuse lint

codestyle:
  extends: .prepare_linting
  script:
    - poetry run black --check --diff .
    - poetry run isort --check --diff .

It can be included into the main .gitlab-ci.yml YAML file as follows:

stages:
  - lint

include:
  - local: '.gitlab/ci/lint.gitlab-ci.yml'

As you can see the keyword include has got sub-keys namely local, file, remote and template which we will explore in the next sections.

Include Files Inside the Same Project with `include:local`¶

As demonstrated above, the simplest way to include YAML files in another one is to put the file to include somewhere in the same project and reference it with the include:local sub-key in the main YAML file .gitlab-ci.yml by using the relative path from the project root to the YAML file to include.

Include Files from the Same GitLab Instance with `include:file`¶

The extracted YAML file might not be contained in the same project but in a different project in the same GitLab instance. The corresponding reusable YAML given above can be referenced with the include:file sub-key like this:

stages:
  - lint

include:
  - project: 'my-group/my-project'
    ref: 'main'
    file: '.gitlab/ci/lint.gitlab-ci.yml'

Include Files from Other GitLab Instances with `include:remote`¶

If the reusable YAML file is not even located in the same GitLab instance, we could make use of the YAML with the include:remote sub-key:

stages:
  - lint

include:
  - remote: 'https://gitlab.com/my-group/my-project/-/raw/main/.gitlab/ci/lint.gitlab-ci.yml'

Include GitLab CI/CD Templates with `include:template`¶

If your use-case is a common one, you could also use one of the GitLab CI/CD Templates of the GitLab project on GitLab.com with the include:template sub-key:

include:
  - template: 'Python.gitlab-ci.yml'

At the time being the above-mentioned Python.gitlab-ci.yml template looks like this:

# To contribute improvements to CI/CD templates, please follow the Development guide at:
# https://docs.gitlab.com/ee/development/cicd/templates.html
# This specific template is located at:
# https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Python.gitlab-ci.yml

# Official language image. Look for the different tagged releases at:
# https://hub.docker.com/r/library/python/tags/
image: python:latest

# Change pip's cache directory to be inside the project directory since we can
# only cache local items.
variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

# https://pip.pypa.io/en/stable/topics/caching/
cache:
  paths:
    - .cache/pip

before_script:
  - python --version ; pip --version  # For debugging
  - pip install virtualenv
  - virtualenv venv
  - source venv/bin/activate

test:
  script:
    - pip install ruff tox  # you can also use tox
    - pip install --editable ".[test]"
    - tox -e py,ruff

run:
  script:
    - pip install .
    # run the command here
  artifacts:
    paths:
      - build/*

pages:
  script:
    - pip install sphinx sphinx-rtd-theme
    - cd doc
    - make html
    - mv build/html/ ../public/
  artifacts:
    paths:
      - public
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy:
  stage: deploy
  script: echo "Define your deployment script!"
  environment: production

Split the Pipeline Definitions into Multiple Files¶

Now we are going to split our rather long .gitlab-ci.yml YAML file into multiple YAML files to:

increase readability and maintainability,
reduce duplication of the same configuration in multiple places,
reuse complete YAML files in different projects of the same kind.

Let us first split it into three files, one for each stage:

File .gitlab/ci/lint.gitlab-ci.yml defines all CI jobs in stage lint:

license-compliance:
  stage: lint
  script:
    - poetry run reuse lint

codestyle:
  stage: lint
  script:
    - poetry run black --check --diff .
    - poetry run isort --check --diff .

File .gitlab/ci/test.gitlab-ci.yml defines all CI jobs in stage test:

test:python:
  image: python:${VERSION}
  stage: test
  script:
    - poetry run pytest tests/
  parallel:
    matrix:
      - VERSION: ["3.10", "3.11", "3.12"]
  needs: ["codestyle"]

File .gitlab/ci/run.gitlab-ci.yml defines all CI jobs in stage run:

running:
  stage: run
  script:
    - poetry run python -m astronaut_analysis
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  artifacts:
    paths:
      - results/
  needs: ["test:python: [3.11]"]

Now, to include all those YAML files in file .gitlab-ci.yml, you can list all files to include with the include:local sub-key:

stages:
  - lint
  - test
  - run

default:
  image: python:3.11
  before_script:
    - pip install --upgrade pip
    - pip install poetry
    - poetry install

include:
  - local: '.gitlab/ci/lint.gitlab-ci.yml'
  - local: '.gitlab/ci/test.gitlab-ci.yml'
  - local: '.gitlab/ci/run.gitlab-ci.yml'

This way your main CI-file stays clearly arranged, and it is easier to orient and get a good first overview. If you need to look deeper into the structure you will very easily spot the file in which you need to look into.

Make Use of GitLab CI/CD Templates¶

The GitLab CI/CD Templates of the GitLab project on GitLab.com offer a huge library of sophisticated reference templates. You can use an original or a modified version of a template and create a YAML file to be included, e.g. with the include:local sub-key. You can also include the file as is into your CI pipeline with the include:template sub-key if it already works out of the box for your project. Be aware that these templates might be subject to change over the course of time if you consider to include templates from GitLab.

Exercise

Exercise 1 - Restructure the CI Pipeline of the Exercise Project¶

In this final episode we used the include keyword to split up a CI pipeline into multiple files. Now try to apply this way of restructuring on the current version of our exercise CI pipeline.

Take Home Messages

Our last episode about mastering GitLab CI pipelines was about how to structure the .gitlab-ci.yml file even more by splitting it up into several files that can be included in the main .gitlab-ci.yml file.

How to Continue?¶

Your path to become a master in GitLab CI pipelines has been laid out. These resources will help you when you are about to develop CI pipelines in your own projects. In the next session of this workshop we will focus on your code projects and help you to get started with your CI projects. After that you will be able to dive deeper into the topic on your own by working on your specific requirements of your CI pipelines and reading more in the official GitLab documentation about concepts, keywords, and templates.