Skip to content

HIFIS Technical Platform - Developer Manual

Warning

Note that this documentation is outdated.

Table of Contents

Document History

Current Version: 0.1

Current Version Date: 2020-08-21

Scope

This document contains all information neccessary to participate in the development of the technical platform (TP) of the HIFIS project.

Development Setup

In order to participate in the development, the following pre-requisites are necessary:

  • OpenJDK 11
  • Maven >= 3.6
  • Node >= 12.x (LTS)

For the IDE we strongly recommend using IntelliJ IDEA 2020 in order to have the same code formatting as all other team members and in order to facilitate the setup of the development environment.

You will need to have access to the following resources:

  • GitHub repositories in https://github.com/helmholtz-marketplace
  • Documentation https://gitlab.hzdr.de/hifis/hifis-cloud-documentation
  • Stories and Requirements https://gitlab.hzdr.de/hifis/cloud-technical-platform

Conventions

In order to maintain code readability and to avoid merge conflicts caused by different formatting, the project makes use of the standard code formatter of IntelliJ for the languages used. If we have to change formatting rules, this change has to be documented in this developer manual. The developers decide together whether it is necessary to provide the formatter configuration as a file or if it is sufficient to descibe the necessary configuration options in this manual.

Necessary changes in the IntelliJ preferences:

  • disable package/wildcard imports
    • https://www.jetbrains.com/help/idea/creating-and-optimizing-imports.html#disable-wildcard-imports)
    • Change Class count to use import with '*': to 999
    • Change Names count to use static import with '*': to 999

All code is indented with 4 spaces. We do not use tabs.

Documentation

The general system architecture is being described in the whitepaper. The source code should be documented sufficiently without describing boilerplate code (e.g. getters and setters).

Development Infrastructure

VCS

We are using git as VCS in the project. All source code is maintained in the GitLab repository from HZDR. The repositories are located in the group https://gitlab.hzdr.de/hifis-technical-platform

Development occurs strictly on branches in private repositories and code is only integrated in master after a successful check during the merge request.

Since the GitLab instance from HZDR does not support, there are “personal” subgroups in the main group which provide the namespace for repositories for personal use. If a new developer joins the development, a neew subgroup with his/her name should be created and he/she should be made owner of that subgroup.

CI/CD

In order to always have a compilable and executable code base, we use the CI features from GitHub. After all pushes and for all merge requests, a build pipeline is being executed, which prevents erroneous code from being integrated in the code base.

Additionally, we have an autodeployment setup, which deploys the new artifacts after a successful build to our CD infrastructure. In order to have an easily maintainable and reproducible environment, we are using a Docker deployment infrastructure for this:

  • Successful build triggers build of a Docker image
  • Docker image is being pushed to Docker Hub (https://hub.docker.com/)
  • Watchdog on CD server detects a new image and updates the container

CD Infrastructure

The CD uses a Docker setup on the DESY Openstack infrastructure. In order to minimize the administration effort, the build infrastructure is mostly set up automatically via a heat template (https://eosc-pan-git.desy.de/hifis/tp-dev-deployment-stack). The basic setup consists of the following containers:

  • Docker-compose setup from https://github.com/evertramos/docker-compose-letsencrypt-nginx-proxy-companion
    • nginx-web - reverse proxy
    • nginx-gen - Handles the generation of the reverse proxy configuration
    • nginx-letsencrypt - Handles the generation of Let’s Encrypt certificates
  • Watchtower (https://containrrr.dev/watchtower/) - Update running containers if their base image has been updated
  • one container for every project that has to be deployed continuously
    • Helmholtz Cerebrum
    • Helmholtz Marketplace (server + web app)

Hosts/Virtual Machines in Use

The DNS names are either from DESY Auto-DNS (os-234-tp-deployment.desy.de), DESY QIP (hifis-tp-dev.desy.de) or from the Dynamic DNS service for EGI Federated cloud (https://nsupdate.fedcloud.eu/).

The DynDNS-Entries are continuously being updated via crontab. Every DynDNS entry has an own line in the crontab of root:

1
24 */2 * * * curl -s https://helmholtz-cerebrum-dev.test.fedcloud.eu:cQCtwmXYaf@nsupdate.fedcloud.eu/nic/update > /opt/var/log/dyndns-helmholtz-cerebrum-dev.test.fedcloud.eu.log 2>&1
  • os-234-tp-deployment.desy.de
    • Virtual Machine on Open Stack@DESY
    • Login with SSH: ubuntu
  • helmholtz-cerebrum-dev.test.fedcloud.eu
    • Host - os-234-tp-deployment.desy.de
    • Docker container
    • Helmholtz Cerebrum
  • helmholtz-marketplace.test.fedcloud.eu
    • Host - os-234-tp-deployment.desy.de
    • Not Configured
  • helmholtz-cloud-cd.test.fedcloud.eu
    • Host - os-234-tp-deployment.desy.de
    • Not Configured
  • helmholtz-marketplace-dev.test.fedcloud.eu
    • Host - os-234-tp-deployment.desy.de
    • Docker container
    • Helmholtz Marketplace Server
  • hifis-tp-dev.desy.de
    • Virtual Machine on Open Stack@DESY
    • Login with SSH: ubuntu
    • Designated for deployment of the public test versions
    • atm hosting for HighRes mockups

Quality

Code Reviews

All code that gets merged into master has to be reviewed by another developer. For this purpose we use the code review system which comes with GitHub.

SonarQube

For the automatic monitoring of the code quality, a SonarQube instance is running automated checks on the code. The goal is to keep the count of problems at zero.

The following projects are configured in SonarQube:

  • Helmholtz Cerebrum - de.helmholtz.marketplace.cerebrum.helmholtz-cerebrum
  • Helmholtz Marketplace Server - de.helmholtz.marketplace.helmholtz-marketplace-server
  • Helmholtz Marketplace Web App - de.helmholtz.marketplace.hifis-marketplace

The SonarQube instance is installed at https://sonar.desy.de You can login with the Helmholtz AAI. For using the SonarQube rules already during development, you can include them via the SonarLint plugin (https://www.sonarlint.org/intellij/) in IntelliJ.

  1. Configure the Access Token in Sonar

    • Login with Helmholtz AAI Choose OpenID login
    • Choose your provider Choose provider
    • Enter your credentials Enter your credentials
    • Review information and consent
    • Go to “My Account” Sonar Account Details
    • Enter a name for a new token (e.g. intellij) Sonar Account Details
    • Copy the token value (optionally save it for later reuse) Copy token value
  2. install the plugin from the marketplace

    • Open the project in IntelliJ (this example uses helmholtz-cerebrum) and go to plugins (Ctrl+Shift+A/+Shift+A and enter “Plugins”) Open plugin settings in IntelliJ
    • Find the SonarLint plugin in the marketplace and install it Find and install SonarLint plugin
  3. configure this token in the plugin

    • Open the SonarLint view (Step 1)
    • Open the configuration dialog (Step 2) Open the SonarLint view
    • Check “Bind project to SonarQube / SonarCloud” and click on configure the connection Configure the connection
    • Add a new connection with the “+” sign Add a new connection
    • Choose SoanrQube and enter the URL of our server (https://sonar.desy.de ) Choose SonarQube and enter the URL of our server
    • Choose token authentication and enter the token you generated in the first step Choose token authentication and enter the token
  4. choose the project mapping(s) (see above)

    • Choose the correct project mapping - 1 Choose the correct project mapping - 1
    • Choose the correct project mapping - 2 Choose the correct project mapping - 2
  5. You have instantaneous feedback to recognized code problems in your currently opened files! Direct feedback in your opened files

Back to top