Skip to content

Continuous Integration (CI)

Please note

Currently, we do not enforce a timeout of jobs on runner side. The runners are built to best support your scientific projects. It would be a pity if the generally available offer had to be limited due to misuse of the platform.
So let us all use the resources for the best!

The Helmholtz GitLab is by default equipped with the possibility to use Continuous Integration for your projects. Several SaaS runners are provided for your projects.

Getting familiar with GitLab CI

In order to get started and familiar with GitLab CI we heavily suggest to have a look into any of these resources.

Linux shared runners

General Purpose

By default, without specifying a tag, all runners are configured with these properties:

Executor Operating System privileged (run Docker-in-Docker )
Docker Linux False

Multiple runners are available to execute your jobs. If you don’t provide a tag your job will run on any of the machines listed below. Except for hifis-runner-manager-1 and hifis-runner-manager-large-1 all concurrent jobs share the available resources.

Name Base OS # CPU cores Memory Hard disk # Concurrent Jobs Executor Tags privileged Default image
hifis-runner-manager-1 Flatcar Linux 2 4 GiB 40 GiB 30 docker+machine hifis, docker, dind True ubuntu:latest
hifis-runner-manager-large-1 Flatcar Linux 8 16 GiB 40 GiB 10 docker+machine hifis-linux-large-amd64, docker, dind, performance True ubuntu:latest
shared-hzdr-1 Ubuntu 2 8 GiB 80 GiB 2 docker webterminal, 2-core False alpine:latest
shared-hzdr-2 Ubuntu 2 8 GiB 80 GiB 2 docker webterminal, 2-core False alpine:latest

If you require additional isolation of your jobs, hifis-runner-manager-1 is the way to go. On this runner all your jobs run on a fresh VM in autoscaling mode. Each instance is used only for one job. This way it is guaranteed that no jobs affect each other.

What does privileged refer to?

Usually, Docker Containers are run with privileged set to false. For security reasons, it is in general a bad idea to use the privileged mode. A container in privileged mode is granted extended privileges. It allows the container nearly all the same access to the host as processes running outside containers on the host. Nevertheless, this privileged mode is a prerequisite to run a Docker-in-Docker setup. In our setup this is not a security issue: every job is executed in a fresh VM thrown away after the associated job exited. Find more information in the Docker blog.

What is autoscaling?

In this mode jobs are executed on machines created on demand. Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each job. This isolation of VMs created on-demand also maximizes security as a nice secondary effect, because jobs are not able to interfere with other jobs.

Specific Runners

Please note

The specific runners are a limited resource which is generally available. If you do not require the additional features, please do not use them. It would have a negative impact on the speed of your pipeline. The runners described above are available in a highly scalable, low latency way.
Help us to ensure that the resources remain accessible without restrictions.

For certain projects it is useful to test on different hardware or make use of accelerators, e.g. GPUs. This is why we enable you to run your jobs on a set of special purpose runners. We cover four different CPU architectures and two GPU vendors. Below table gives you an information about the available hardware and the tags to use.

Name Base OS Tags # CPU cores Memory Accelerator CPU architecture CPU type
ci-nvidia-hzdr Ubuntu intel, x86_64, cuda, p5000 8 20 GiB Quadro P5000 x86_64 Intel(R) Xeon(R) Silver 4110
ci-amd-rocm-hzdr Ubuntu amd, rocm, rx-vega, amd-epyc-7351, x86_64 8 12 GiB Radeon RX Vega 64 & Radeon R9 FURY x86_64 AMD EPYC 7351
ci-arm64-hzdr Ubuntu aarch64, arm64, cavium-thunderx-88xx 12 12 GiB - aarch64 Cavium ThunderX 88XX
ci-ppc64le-cuda Ubuntu ppc64le, p100, cuda, power8nvl 32 50 GiB 2 × Tesla P100 SXM2 ppc64le Power8NVL

Example

Below you find an example of a .gitlab-ci.yml file using some of the runners mentioned above. By adding tags to your ob definition you let GitLab know on which runner it shall schedule your job.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
test:
  image: nvidia/cuda
  stage: test
  script:
    - nvidia-smi
  tags:
    - cuda  # Job will run either on ci-nvidia-hzdr or ci-ppc64le-cuda

test:cuda:intel:
  image: nvidia/cuda
  stage: test
  script:
    - nvidia-smi
  tags:
    - cuda
    - x86_64  # Job will run on ci-nvidia-hzdr

MacOS shared runners (Beta)

The macOS-runner provides an on-demand macOS build environment. It allows you to build, test and deploy applications in the Apple ecosystem. Intel x86-64 runners were deprecated in favor of Apple silicon and shall no longer be used.

Available Machine Types

Runner Tag # vCPUs Memory Hard disk # Concurrent Jobs
macos-medium-m1 4 8 Gib 40 GiB 2

Supported Images

In contrast to our hosted Linux runners, where you can run any Docker image, we provide a predefined set of VM images for the MacOS shared runners.

Use one of the images listed below.

VM Image Description
hcr.helmholtz.cloud/ci/macos/ventura-base:latest Image has Homebrew pre-installed
hcr.helmholtz.cloud/ci/macos/ventura-xcode:14 Image is based on ventura-base and has Xcode 14 with Flutter pre-installed

Example .gitlab-ci.yml file

The following sample .gitlab-ci.yml file demonstrates how to use the runner. Therefore, it shows how to test against multiple Python versions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
test:
    image: "$IMAGE"
    parallel:
        matrix:
            - PYTHON_VERSION: ["3.9", "3.10", "3.11"]
              IMAGE:
                - "hcr.helmholtz.cloud/ci/macos/ventura-base:latest"
    before_script:
        - brew install python@${PYTHON_VERSION}
        - brew link --overwrite python@${PYTHON_VERSION}
        - pip3 install --upgrade pip poetry
        - poetry env use ${PYTHON_VERSION} && poetry install
    script:
        - poetry run python --version
    tags:
        - macos-medium-m1

Windows shared runners

Currently, we do not offer shared Windows runner. As soon as we also support the Windows platform we will let you know.

Feel free to register your own runners for Windows.

Last update: August 29, 2023
Created: August 20, 2021