What is it about?
The HIFIS Transfer Service (HTS) is envisioned to be a core service in the HIFIS Backbone enabling Helmholtz members to easily share large data sets between Helmholtz centres.
Get to know it! (Video)
Have a look at this brief Introduction Video on how to use webFTS!
What has been done
HTS began as a pilot service in mid-2020 and CERN’s FTS3 was chosen as transfer management middleware together with the graphical, web based frontend WebFTS. In order to enable Helmholtz centres’ participation in HTS, a storage endpoint capable of being controlled by FTS was conceptualized and implemented by the end of 2020.
The ubiquitous Apache webserver httpd was chosen as an endpoint due to it’s modular design and familiarity for many IT administrators. The endpoint can be deployed easily and equipped with backend storage of the administrator’s choice. The configuration of the Apache enables it to interface with FTS3.
Transfer performance
First tests of HTS showed that transfers between connected centres are very feasible and the systems interoperate seamlessly with the Helmholtz AAI. The transfers could be executed with a bandwidth of 80 - 120 MB/s, thus making it possible to transfer 1 TB worth of data in less than four hours without the need for further user interaction.
For more information on HTS performance, you may also refer to the presentations given in recent ISGC and EGI conferences.
HTS for Helmholtz AI
Colleagues from the HZDR and DESY were among the first to use HTS for the transfer and sharing of datasets in the context of the Helmholtz AI platform.
Other centres have already shown interest in establishing endpoints for the participation in HTS and will thus enable more and more researchers to easily share data sets.
How-to
For submitting transfers between known endpoints, you can visit webFTS, log in with Helmholtz AAI and submit your transfers via the “Submit a transfer”. Simply enter the URLs of both endpoints, select the files you want to transfer and click on the arrow for the corresponding transfer direction. You can then monitor your jobs under “My jobs” and see when they are completed. We will also supply a small video demonstration for using HTS with webFTS in the very near future.
Technical instructions for setting up an endpoint for HTS can be found here in our Technical Documentation.
- A pre-configured version that can be deployed easily is available as a Docker image for either standalone or Kubernetes deployments in this gitlab repository.
- In case an administrator wants to set up the endpoint on their own, installation and configuration instructions are available in another repository.
The road ahead
Currently, implementation details of the webserver are refined and some new features are added by KIT and DESY in a joint effort to make the experience even smoother. We also plan on adding another data management layer that makes policy-driven tranfers possible in order to share and/or synchronize datasets automatically between designated participants in cross-centre collaborations. This will make HTS a complete solution for federated data management.
Interested? Need help? Comments and Suggestions?
If you are interested in the HTS itself, want to set up an endpoint at your centre or need support for submitting transfers, please do not hesitate to contact us:
Changelog
- 2021-10-20 – Added link to introduction video