Skip to content

Helmholtz Cloud Agent

In the Helmholtz Cloud, various heterogenous services are connected to the central cloud portal. A lot of services can be used directly by users just by logging in through the Helmholtz AAI. Other services need additional steps to make accounts or resources available to a user or VOs. This usually is a manual process where users have send their requests directly to the service providers, e.g., via email, and the service administrators will then allocate the resource. To automate this process the Helmholtz Cloud Agent has been developed, which is running directly at the service and automates the work of the service administrator. The Cloud Portal is used to send requests to all available Cloud Agents.

Architecture and Implementation

The architecture is built around three main components: The Cloud Portal, a central RabbitMQ broker and several Cloud Agents running directly at the service providers. The Cloud Portal is the main point of interaction for the users to their resources. The RabbitMQ broker provides a communication channel between the Cloud Portal and the different Cloud Agents. The Cloud Agents do the service specific work to book the resources.

Cloud Portal

The Cloud Portal keeps track of all booked resources by the Cloud Agent. Logged in users can see all their resources, book new or update / delete existing resources. Whenever users request new resources the Cloud Portal will send a message to the corresponding service using RabbitMQ. The Cloud Agent will then respond back to the Cloud Portal with with another message that the resource was created or, in case of error, with additional details.

Central RabbitMQ

The RabbitMQ server is hosted at DESY and publicly available on the internet. The connections are secured using TLS. There are separate accounts for the Cloud Portal and the different Cloud Agents identified by username / password. Each service gets its own virtual organization on the server. In a VO there are always two queues to_hca and to_portal. to_hca is used by the Cloud Portal to send messages to the Cloud Agents and to_portal is used by the Cloud Agent to reply back to the Portal. The Cloud Portal has access to all VOs and the Cloud Agent can only access its service-specific VO.

Cloud Agents

The Cloud Agent itself is just a small program that runs close to the service and can receive and send messages from RabbitMQ. It is using a plugin approach where service admins extend the Cloud Agent with their own scripts to do the necessary work to (de-)allocate or update resources at their service.

Repositories

Exemplary Use Case: nubes

This sections describes in detail the first implemented use-case of the Helmholtz Cloud Agent for nubes at HZB. In this case users can ask for group folders on nubes that can be accessed by a provided list of users or a specific VO.

HCA_nubes_case

Cloud Portal

The implementation on the Cloud Portal side is split in two separate parts: the frontend that interacts with the users and a HCA service that communicates with the cloud agents using RabbitMQ.

Frontend

When users are logged in additional menu items become available in a dropdown menu to request and list resources:

logging_in_dropdown

When they want to request the Cloud Portal first checks their permissions. The permissions are based on entitlements that are applied per service and resource type. In the selection menu, users will only see services and resource types for which they have the appropriate permission. If they have no permissions at all they will see an error message.

request_interface

After a request has been submitted users can check the status of all their resources on an overview page:

resource_overview

They can also get detailed information for each service:

resource_details

Backend

When users submit requests in the frontend a new request object will be created in the database. A continuously running service in the backend will periodically check new requests and will send them to RabbitMQ. This service keeps a mapping of all services and their corresponding virtual hosts to send the request to the correct queue. After the request has been handled by the service the backend will receive the response from RabbitMQ and update the resource in the database accordingly.

The message that the Cloud Portal sends is ResourceAllocateV1 and the type is GroupStorageResourceSpecV1. Depending if the folder is created for a single user or a list of VOs the targetEntity is either UserSpecV1 or VOListSpecV1.

Cloud Agent @ nubes

At nubes a Nextcloud plugin is used to manage the requests coming from the Cloud Portal. The Cloud Agent gets the resource request from RabbitMQ and injects it into the Nextcloud database. The plugin then provides a table in the Nextcloud admin interface with all open requests. The service admins can then approve or deny the request. If the request is approved the folder will be created in nubes with the requested access permission. The Cloud Agent then sends a ResourceCreatedV1 response to the Cloud Portal. If the request has been denied an ErrorV1 response will be send with a description text.

Questions? Comment?

Don’t hesitate to contact us at support@hifis.net if you have any questions or comments on Helmholtz Cloud Agent.

Last update: May 2, 2023
Created: October 21, 2021