How to Build your own Helmholtz Cloud Agent

Here, we guide you through the first steps of writing and connecting your Helmholtz Cloud Agent (HCA) to the central Helmholtz Cloud resource management. A Helmholtz Cloud Agent is a small Python tool that runs close to a service and communicates with the Helmholtz Cloud. It receives resource requests and sends back information about the allocated resource. The communication with Helmholtz Cloud is handled automatically by a provided library so that you can focus on integrating your service.

Developing your first Helmholtz Cloud Agent

We provide a template repository that you can use as a starting point for your development. It contains a devcontainer setup with a local RabbitMQ message broker and a Helmholtz Cloud Agent development container. It also includes a sample Dockerfile that can be used to deploy your Helmholtz Cloud Agent.

Fork the template repository to start developing your own agent. If you are using Visual Studio Code with devcontainers you can simply check out the repository and open it. It will ask you to “Reopen in Container” which will launch the development environment. If you are not using VSCode you can just manually start the environment with the provided Docker Compose file in the dev folder.

Communicating with RabbitMQ

The template comes with a sample Helmholtz Cloud Agent implementation that we will discuss later. For now, we will just start the Helmholtz Cloud Agent and test if everything is set up correctly. To start the Helmholtz Cloud Agent run this in a shell in the hca_dev container:

poetry run python src/message.py

This will start the Helmholtz Cloud Agent and connect to RabbitMQ. RabbitMQ comes with a management console that you can access at http://localhost:15672. You can find the credentials (HCA_USERNAME and HCA_PASSWORD) to log in to the environment list in the docker-compose.yml file. Once you are logged in, you can access information about the RabbitMQ broker. For Helmholtz Cloud Agent development we are only interested in the “Queues and Streams” tab. In the table you will find two queues that are already set up: to_hca and to_portal. to_hca is used to send messages from Helmholtz Cloud to the Helmholtz Cloud Agent and to_portal to send messages in the other direction. You can send and receive messages directly from this management console which can be useful for debugging. Additionally, the plugin-template repository comes with a script that mocks the interaction of Helmholtz Cloud with the Helmholtz Cloud Agent.

To test the setup, we can start by sending a simple PingV1 message. Helmholtz Cloud may send this message at any time to check whether the Helmholtz Cloud Agent is still running. Your Helmholtz Cloud Agent will then automatically send a PingV1 response back to Helmholtz Cloud. While keeping the Helmholtz Cloud Agent running, open another shell and run:

poetry run python dev/mock_portal.py send-ping

You should see an output with the message “still alive”. If yes, your development environment is set up correctly, and we can now proceed with resource requests.

Send your first Resource Request

In the same way that we sent the PingV1 message we can also send resource requests to the Helmholtz Cloud Agent. The message type is ResourceAllocateV1 and the actual content depends on your resource type and your policy definition in Plony as described in integration guide. However, the main structure is always the same three fields:

• type: A string representing the title of the resource type JSON schema.
• target_entity: A dictionary containing either the group URN or the user ID, depending on the selected policy.
• specification: A dictionary containing the resource specification as defined in the JSON Schema properties of the resource type.

Using the Script

The sample Helmholtz Cloud Agent in the template accepts two different resource types (ChatTeamSpecV1 and ComputeResourceSpecV1). For testing, it randomly returns a ResourceCreatedV1 or ErrorV1 message without actually doing anything. Here, we will start by sending a ChatTeamSpecV1 to our Helmholtz Cloud Agent using the mock_portal.py scripts:

poetry run python dev/mock_portal.py send-resource-allocate payloads/ChatTeamSpecV1.json

The mock_portal.py expects a JSON file with a payload that it will send to the Helmholtz Cloud Agent. Two sample payloads are already provided in the payloads folder. You can adapt and add payloads as you need for developing your Helmholtz Cloud Agent. When you run the command you should either see an output with a ResourceCreatedV1 or ErrorV1 message.

Using the RabbitMQ Console

If you do not want to use the script you can also send messages directly to the Helmholtz Cloud Agent using the RabbitMQ console. On the RabbitMQ console go to the “Queues and Streams” tab, open the to_hca queue and go to the “Publish message” section. First, we need to fill in some headers and properties. In the production environment these are handled transparently by Helmholtz Cloud and the Helmholtz Cloud Agent, but here we need to set them manually:

In the headers you need to set the service_id. This is the ID given to your service in Helmholtz Cloud. In production, you need to set the HCA_SERVICE_ID environment variable correctly, but here it is set to a mock value that you can find in the docker-compose.yml file.

The type property needs to be set to ResourceAllocateV1 for requesting resources. The correlation_id is used so that Helmholtz Cloud can match the response from the Helmholtz Cloud Agent to a resource request. This can be set to any value for testing purposes.

The payload is a JSON dictionary that matches the schema defined in the resource type. It is the same as in the PAYLOAD variable in the mock_portal.py script. For this test we take a ChatTeamSpecV1 schema. The type is defined in the json_schema’s title attribute of the ResourceType. This can be customized by you when you define your own ResourceType later on. You are not limited to the example names. The target_entity is set to a user_id_target and the specification consists of a team_name and invite_only:

{
    "type": "ChatTeamSpecV1",
    "target_entity": {
        "group_urn_target": null, // (1)!
        "user_id_target": "bd2dc138-e63a-46d4-a042-3c6b7d313b8f" // (2)!
    },
    "specification": { // (3)!
        "team_name": "My Mega Team",
        "invite_only": true
    }
}

1. If this was a group resource, this would contain the group’s URN.
2. This is a personal resource. If this was a group resource, this would be null.
3. This structure is specified by you in the ResourceType.

Send a resource request to the Helmholtz Cloud Agent using the RabbitMQ console.

When you send the request you should immediately see some log output from the Helmholtz Cloud Agent. The example script randomly replies with either a successful or failure message, which you can again receive from the to_portal message in the RabbitMQ console.

Receive the Helmholtz Cloud Agent response using the RabbitMQ console.

Writing a Message Handler

So far, we have successfully sent and received messages from the sample Helmholtz Cloud Agent, but we have not seen how message handling is implemented. The Helmholtz Cloud Agent library provides an application that handles the actual communication with RabbitMQ. You can register handler functions that will be called whenever a message of a certain type is received. This is then the entry point for your development, from where you can then make the necessary calls to provision the resource at your service.

A very simple Helmholtz Cloud Agent would look like this:

...
from helmholtz_cloud_agent.core.main import HCAApplication
from helmholtz_cloud_agent.messages import ResourceAllocateV1, ResourceCreatedV1
...

app = HCAApplication()

@app.handle(message_type="ResourceAllocateV1")
def resource_allocate_v1(
    correlation_id: str, payload: ResourceAllocateV1
) -> ResourceCreatedV1:
    logger.info(f"Received a message payload of type '{payload.type}'")
    return ResourceCreatedV1(id=str(uuid.uuid4()))

app.run()

We start by importing the HCAApplication that comes with the Helmholtz Cloud Agent library along with the Pydantic structs for the ResourceAllocateV1 and ResourceCreatedV1 messages. Then, we create the actual app and register a function that will handle the messages we receive with the app.handle decorator. A different handler needs to be defined for each message type, and we need to specify it with the message_type parameter. The function itself will always get the correlation ID and the payload. The type will always be the same as the message type.

That’s it. Now you can take the payload and write the code that does the actual resource provisioning.

When the resource is created, you need to send a ResourceCreatedV1 message back to Helmholtz Cloud. The only thing you need to provide is an ID. This ID will be stored by Helmholtz Cloud and when the resource should be deprovisioned, Helmholtz Cloud will send you a deprovisioning request with this ID. Therefor, the ID should uniquely identify the resource at your service.

If for some reason the resource cannot be provisioned, you can send a message back to Helmholtz Cloud that is displayed to the user. To do this, you can define exception classes that can be raised in the message handler. The Helmholtz Cloud Agent application will handle these and send the appropriate message to Helmholtz Cloud.

...
class ChatTeamAlreadyExistsError(Exception):
    def __init__(
        self: "ChatTeamAlreadyExistsError",
        folder_name: str,
        message: str = "Chat team already exists",
    ) -> None:
        self.folder_name = folder_name
        self.message = message
        super().__init__(self.message)

    def __str__(self: "ChatTeamAlreadyExistsError") -> str:
        return f"{self.message}: {self.folder_name}"
...
@app.handle(message_type="ResourceAllocateV1")
def resource_allocate_v1(
    correlation_id: str, payload: ResourceAllocateV1
) -> ResourceCreatedV1:
...
    raise ChatTeamAlreadyExistsError(spec.desired_name)
...

Deploying your Helmholtz Cloud Agent

When you are ready to deploy your Helmholtz Cloud Agent, the template repository provides a sample Dockerfile that you can use to package your code. It installs a Python environment with all the dependencies from pyproject.toml, takes everything from the src folder and puts it in the /opt/hca folder in the container. Then, you just have to adjust the ENTRYPOINT to invoke your application.

To start the Helmholtz Cloud Agent, you need to set some environment variables:

• HCA_USERNAME: The RabbitMQ username. You will get this from Helmholtz Cloud developers.
• HCA_PASSWORD: The RabbitMQ password. You will get this from Helmholtz Cloud developers.
• HCA_SERVICE: The ID of your service in the Helmholtz Cloud. You will get this from Helmholtz Cloud developers.
• HCA_RABBITMQ_USE_SSL: Set to True for production.
• HCA_RABBITMQ_HOSTNAME: Set to hifis-rabbitmq.desy.de for production.
• HCA_RABBITMQ_PORT: Set to 5671 for production.