HIFIS Conference
The first conference on HIFIS took place at DESY Hamburg.
The first conference on HIFIS took place at DESY Hamburg.
The platform software.hifis.net (meanwhile moved to hifis.net) is launched as the future home for software development services within Helmholtz and HIFIS. The static site is built with Jekyll. Its sources are available on GitLab. Feel free to build upon it.
In preparation of providing Helmholtz-wide training events 12 Carpentries instructors are trained as part of the membership of the HZDR and GFZ in the Carpentries.
The first Helmholtz-wide training events are conducted. Read the announcement post for information on how the initial course portfolio will look like.
The first steps, progress and ongoing works of HIFIS Cloud, Backbone, and Software Clusters have been reported in the first annual report of HIFIS. This report has been improved and approved by the ICT Federation Board and further served as a basis for feedback by the Scientific Advisory Board in April.
The HIFIS Scientific Advisory Board gathered in a video conference to discuss the progress of HIFIS and provide advice. Comments and suggestions were compiled in a comprehensive report.
From June 2019 to May 2020, a service survey throughout all 19 Helmholtz centres has been conducted. The target of this survey was to determine which services are desired to be offered as services in HIFIS. A second target was to identify which centres are willing to act as service providers for the services to be offered in the Helmholtz Cloud.
In preparation of the launch of the Consulting Services the HIFIS Software helpdesk has been launched basing on Zammad. Therefore, a reusable Ansible role has been implemented.
The blueprint for the new official Helmholtz-wide Authentication and Authorization Infrastructure (Helmholtz AAI), derived from former HDF AAI, is online and available for services and identity providers. It is drafted by FZJ and KIT, and is compatible to the AARC blueprint.
HIFIS fostered a service agreement with the German National Research and Education Network (DFN) on operating a a Helmholtz-wide Virtual Private Network (Helmholtz Backbone), covering the majority of Helmholtz centres. By September, the contract is expected to be signed from the participating partners.
A common contact point for all queries and support requests shall be introduced. This will be realized using a ticketing system, allowing to distribute tasks related to the different HIFIS clusters, especially Cloud Services and Software Services.
HIFIS Software offers free-of-charge consulting as a service to research groups within the Helmholtz umbrella. We can help you deal with specific licensing issues, migrating code between different environments and languages, setting up new projects, and other problems that you need to solve.
After multiple iterations, the first set of selected federated services, forming the initial Helmholtz Cloud Service Portfolio, will be announced and started to be integrated. This set will be continuously enhanced and the performance of the integrated services will be monitored.
A meeting of all HIFIS members is planned to foster collaboration between the HIFIS clusters. Non-HIFIS Helmholtz-centres as well as all incubator platforms shall be incorporated.
The future software management platform will be based on reusable Ansible recipes. This allows other research centers to reuse the implemented solution. The open-source recipes are publicly available on gitlab.com/hifis/ansible.
Having finished the list of the initial service portfolio, the integration of the services will start. At first, the exact details of each service and its tasks for service integration will be examined. This goes hand in hand with the first integration steps. It will also result in a specific roadmap to be established by the end of the year.
The software management platform is planned to be integrated into the Helmholtz Cloud as soon as the technical platform is available. Continuous Integration will be part of the software management platform from the very beginning.
One important element of the software management platform is a solution for continuous integration (CI). Reusable Ansible recipes for a GitLab CI infrastructure will be provided. The Ansible role is developed here.
First Virtual Private Network (Helmholtz Backbone) connections have been set up and are in productive use.
We are happy to announce that the group of the 10 most mature services has reached a maturity level of more than 80% - more than sufficient for them to be integrated into the Cloud Portal in the coming weeks. This top group includes services like GitLab (HZDR, FZJ), Nextcloud (HZB, KIT), Mattermost (HZDR), Rocket.Chat (FZJ) but also OpenStack (FZJ, KIT), JupyterHub (FZJ), Zammad (HZDR) and B2Share (FZJ).
Find more information of Initial Service Portfolio here.
The community work package publishes further details for the Helmholtz Hacky Hour for the first half of 2021 on the events page.
The GitLab service will be deployed to a new Ansible-managed infrastructure providing an enhanced maintainability and scalability. Access to this Helmholtz-wide GitLab will be granted to all people affiliated with the Helmholtz Association and partners to foster cross-center collaboration. If users encounter any issues, they can report them via the HIFIS helpdesk.
The education and training work package publishes the offered training events for the first half of 2021 on HIFIS Software Events.
After the ramp-up phase during the first year, the initial Cloud, Backbone and Software Services are expected to be in production by early 2021. We will report on the integration progress and impact of these services, especially with respect to user acceptance and added value for the scientific community.
The Helmholtz Cloud Portal allows harmonized access to all Helmholtz Cloud services. It contains all necessary information to access the available services. The alpha version of the portal is planned to be available by end 2020 and the publicly accessible beta version is planned for February 2021.
While the integration of the initial service portfolio will continue, the evaluation of further services will start. This process will presumably involve the review of postponed services from the initial service selection as well as service selection criteria, focusing on service usage and optimal user experience.
Starting data collection from Helmholtz employees about their used IT-related workflows and software tools. The data will be analysed to further improve our services. Results will be published later in the year.
HIFIS Community evaluates the reusability of the Netherlands RSD as RSD for all Helmholtz centers.
The education and training work package publishes the offered training events for the second half of 2021 on HIFIS Software Events.
Deprovisioning of users will be supported by the Helmholtz AAI. This includes a framework to query deprovisioning information from IdPs to allow automatic deprovisioning in Helmholtz AAI. In case of failure, user shall be contacted to confirm their account manually.
The success of the services is essentially reflected in their use and application. From the middle of the year, key figures on the use of the individual services are therefore determined and collected.
The automatic deprovisioning of users is documented and technically implemented for at least one service and at least one service using centre.
The HIFIS data transfer service will be improved such that any combination of interested centres can participate by only needing to install a lightweight client that can manage third party copy.
Developed concept and alpha implementation for automated communication with services, for example using Local Agent or via Cloud Portal. This requires cooperation with the Service Integration working group of the Cloud Cluster.
Services, that were postponed during the initial service selection, as well as newly applied services were transferred into the onboarding process for new Helmholtz Cloud services.
We consider Continuous Integration (CI) an important aspect in a modern software engineering workflow. Easy-to-use CI resources should be available for all Helmholtz scientists. Beside the Helmholtz-wide GitLab many Helmholtz centers run their own GitLab instances on dedicated infrastructure in their center. In order to bundle resources and enable research software projects to easily use CI for their projects, CI will be offered as a service. In this milestone, we will set up CI as a Service and test it first with few pilot centers, in order to make it available to all Helmholtz communities.
Please note: The milestone was originally set for the end of the second quarter. Due to the underlying compute resources not being available in time, this milestone needed to be rescheduled to the third quarter of 2021.
The integration of several services of the initial service portfolio is completed and they can be used according to their description and service level. Therefore, these services immediately become fully integrated services when the legal framework is signed.
Data collection and primary analysis of the HIFIS survey 2021 has been finished. First results will be presented in respective meetings (e.g., HIFIS conference, Incubator workshop).
The production version of the Helmholtz Cloud Portal will be made available. The new search function and filter options help to find a suitable service. The availability of the service is tested automatically and is visible in the service card. The service description contains all main information such as support, service level and limitations. User comments are welcome in the new free text field and will help to continuously improve the usability of the portal.
The Process Framework for Helmholtz Cloud Service Portfolio is published in version 1.0. It focuses on the explanation of the processes regarding the Service Portfolio Management for Helmholtz Cloud, giving an overview of which processes exist, how they interconnect, which roles are involved in each process and what is included in each process step.
Obtained deprovisioning information shall be provided to services. This requires cooperation with the Service Integration working group of the Cloud Cluster.
The legal framework describes the necessary coordination and responsibilities so that the Helmholtz Cloud can perform its task for and within the Helmholtz community. It covers the relevant aspects, such as data sovereignty, intended use, GDPR or IT security.
At least one of the prominent services (such as Nextcloud, Open Stack or Jupyter) or a combination of those is extended to a meta service: the meta service, which is provided by several Helmholtz Centers, is working as a federation or cooperation instead of separated local solutions.
Allowing scientists or reviewers to easily interact with computational environments can be considered a key aspect for reproducible and easy-to-use research software. Integrating tools like Binder or ShinyR into the Helmholtz-wide GitLab will allow scientists to easily make their research software product available to others without the need for installing and configuring tools locally. In this milestone, a concept will be elaborated on how this integration may be realized.
Publish results of HIFIS survey 2021.
The primary goal is to check whether the services in and the processes around the Service Portfolio continue to fulfill defined requirements. Consequently, the review may lead to changes in the Service Portfolio itself as well as the Process Framework. The results will enter the annual HIFIS report and further Service Portfolio Maintenance.
In Cloud Portal, logged-in users shall be able to have a specific view on the services they are allowed to use, as well as service usage conditions. Also, a user can see and manage their relevant service notifications.
(This work package was postponed from September 2021 to January 2022, as no user-specific information will be available by then.)
The education and training work package publishes the offered training events for the first half of 2022 on events.hifis.net.
Service Providers can now provide all service information asked for during the Onboarding process in Plony, our data management system based on Plone.
The Assembly of Members of the Helmholtz Association have received a draft for the definition of technical, administrative and data protection regulations to simplify the use of the Helmholtz Cloud and its Services, and to ensure that applicable conditions are met.
An initial course on using GitLab CI will be held. At this time also the initial course material will be published.
Improve the consulting handbook in all aspects, based on past experience. and publish it.
The education and training work package publishes the offered training events for the second half of 2022 on events.hifis.net.
Virtual Organisations (VOs) have been introduced in Helmholtz AAI during 2020. Requests to create new top-level VOs are currently being processed on end user request, using the HIFIS Helpdesk and considerable manual interaction for VO creation and documentation. In 2022, a transparent process shall be set up and implemented to allow VO handling in larger scales than so far.
The basis for the software catalogue will be the Research Software Directory developed at Netherlands eScience Center. In close collaboration with the original maintainers, an adapted version for Helmholtz will be developed. The development of the HIFIS Research Software Directory will be coordinated at GFZ in collaboration with DLR and UFZ, and aided by the HIFIS Software Technology work package. In this milestone a basic prototype with the initial functionality will be deployed, marking the beginning of the test phase with selected test users.
Until summer, HIFIS will give all stakeholders the opportunity to strengthen our community and to contribute to the further development of HIFIS, especially in light of the upcoming review. As a part of this, HIFIS will organize an All-HIFIS conference in Berlin, possibly as presence meeting, if circumstances allow.
Service categories and keywords categorise the Helmholtz Cloud Services and simplify the selection of a suitable service.
The hifis.net homepage will be relaunched, with improved and aligned design. Based on the progress that HIFIS meanwhile made, having implemented and distributed numerous services to the scientific communities, we will increasingly concentrate on the presentation of scientific use cases.
All steps of the Onboarding Process for new Cloud Services are completely integrated into Plony, the Plone-based database. This enables Service Providers and HIFIS administrators to maintain the Service Catalogue online and collaboratively. The Service Description is automatically transferred to Cloud Portal.
After a bit more than three years of initial phase, the progress and further perspectives of HIFIS will be reviewed. Earlier in 2022, potential perspectives will be proposed and formulated in a pre-review report. End of September, the main review meeting will take place at DESY.
The GitLab CI as a Service offer is planned to be expanded to at least 3 more partners within the Helmholtz association.
During 2021, the framework to automatically request user deprovisioning information from IdPs via attribute query has been principally set up for the central AAI (development) instance and a small set of Helmholtz IdPs. During 2022, the task is to remove remaining inconsistencies and set up the productive deprovisioning process for all IdPs, at least with the already implemented fallback of manual user based confirmation of active accounts.
In close cooperation with the data protection officers of the Helmholtz Association, we have developed a joint controller agreement that covers the Helmholtz Cloud core components that operate the cloud.
For 2022, a set of policy rules, regulating the basic participation conditions of the Helmholtz Backbone network, shall be worked out and put into action for the participants.
So far the Helmholtz Codebase service does not support the Windows operating system for executing jobs. As part of this milestone the Helmholtz Codebase service will support the Windows operating system to run CI jobs.
As part of this milestone it should be possible to
In 2021 the proposed goal of introducing a post-consulting survey for KPI-measurement was achieved by the Consulting work package. Therefore, this year the work package will focus on improving the KPIs by analysing software projects of clients, measure potential improvements and the long-term effects after a HIFIS consultation.
After ramp-up phase, and going towards operational phase, the components of Helmholtz AAI will undergo a regular updating and housekeeping process. Amongst these, the owners of registered services as well as the managers of registered Virtual Organisations (VO) will be contacted and requested for information confirmation or updates. Further updates and regular review processes are to be defined in the upcoming Policy Review Process.
For over one year, HIFIS already has been an observing member of the AARC Engagement Group for Infrastructures (AEGIS), which facilitates activities for the adoption of harmonised federation solutions and thus the design and adoption of AARC guidelines.
Based on the many practical use cases in the context of Helmholtz AAI and Helmholtz Cloud, as well as the continuous increase of user numbers and participating groups, HIFIS was asked to step up as a full member. It is planned to do so by February; we look forward to a fruitful cooperation on further developing harmonised federated access to digital resources for science.
Following the overall very positive HIFIS evaluation results received end of 2022 and in line with our ongoing annual reporting, we will define the short- and mid-term steps to focus on during 2023 and parts of 2024.
Using the Helmholtz-AAI as a starting point, the NFDI IAM proposal was submitted to support the 26 NFDI Consortia. The participating HIFIS members are tasked to maintaining full compatibility with the Helmholtz-AAI and hence the AARC blueprint, and to establish a two-way communication channel amongst the participants.
With the second regular Service Portfolio Review (Nov 2022 - Mar 2023) we incorporate first feedback from the recent HIFIS evaluation, adapted processes and improved the web forms built in Plony. The fact that Plony stores all service information paired with the review functionality (available since November 2022) simplified the process.
After six months of testing and preparation, HIFIS finally opens the Helmholtz Research Software Directory (RSD) for all Helmholtz members. If you have an account at a Helmholtz Institution, you can now login to the RSD via the Helmholtz AAI, and start adding your Research Software.
HIFIS Backbone members are participating in the EU project proposal for the third AARC project, which will update the globally accepted AARC Blueprint Architectures, Policies, and Guidelines. The participation of HIFIS partners will intensify useful feedback in both directions between the global AAI community and HIFIS.
The Helmholtz Cloud Portal has been migrated towards a new technical foundation, allowing for easier and more secure maintenance, functionality updates and integration. Furthermore, new user-oriented work flows can be integrated in coordination with the Service DB (Plony). The presentation of the services changes slightly. Based on the new back-end, user interface improvements and facelifts will be integrated more efficiently.
It is planned to enforce MFA in administrative endpoints of the central community AAI (Unity IdM), i.e., the /console and /oauth-home (userhome of oauth clients) endpoints. It will be made optional on normal user’s /home /upman /oauth2 /saml-idp endpoints and recommended for VO admins to use it. Corresponding documentation will be provided and signalling the MFA status will be prepared.
In spring and summer 2023, scientists will have the opportunity to be taught the combined HIFIS course portfolio in a joint RSE Bootcamp organized with HIDA.
Users from numerous non-Helmholtz organisations can in principle access Helmholtz AAI and Cloud services by logging in via their home IdP. However, lack of standard conformity, local policies and technical inconsistencies frequently prevent successful authentication, frustrating the users and also putting high load on the HIFIS support. A frequently used workaround, authenticating via social IdPs (ORCID, Github, Google) allows users to access our services with caveats. Using AAI statistics of user’s originating organisations, we will identify organisations with significant numbers of users, and non-working IdP based authentication, so that we can concentrate efforts on actively integrating these organisations systematically. The identification and workflow to do so is planned to be established by late spring 2023, with follow-up work being continuous.
HIFIS provides numerous channels that already allow users and user groups to funnel their feedback on HIFIS services. The tools include the single point-of-contact Helpdesk, direct feedback in the Helmholtz Cloud Portal, surveys, the Helmholtz Cloud Service Operation KPI, and many more. Until summer, we will streamline the processes to follow-up on user queries, including selection and priorisation. We will decide on potential additional or altered formats to systematically obtain user feedback. Further, the reporting (KPI) will be adapted, moving a bit away from reflecting the build-up of services, towards the representation of user acceptance and user satisfaction.
Again, a special focus will be put on Continuous Integration (CI). For this purpose, the team offers a container workshop or GitLab-CI workshop once per quarter. In addition, general CI support for Windows will be added to the Helmholtz Codebase.
The Helmholtz Software Spotlights found their new home in the Helmholtz RSD. The Spotlights previously presented at hifis.net will be migrated to the Helmholtz RSD and presented there from now on.
After filling the vacancies, the consulting team will put dedicated efforts to increase outreach and awareness of the consulting service in Helmholtz.
In close cooperation with the Helmholtz Data protection officers, a Joint Controller Agreement is finalized that is regulating the processing of personal data in the Helmholtz Cloud Core Components. The Core Components are needed to run the Helmholtz Cloud: Helmholtz AAI, Helmholtz Cloud Portal, Service Database and Helpdesk.
The HIFIS Backbone cluster, together with Cloud cluster, will define first coordinating steps to align with Helmholtz KoDa’s activities in fostering Operational Cybersecurity.
In the Helmholtz Incubator Workshop to be held in July, HIFIS will present the envisioned future developments, especially in the light of the advises given during HIFIS evaluation.
The technical tooling of asking IdPs to report the status of a specific user via Attribute Query, is supported by the Helmholtz Community AAI. For summer, we plan to put the automated deprovisioning process into action for users that have not logged in to Helmholtz AAI by more than a defined grace time, either by a) querying IdPs which support this until then, or b) querying users via email to log in again. The deprovisioning information is then forwarded to connected cloud services.
The Helmholtz Cloud Portal allows to book resources within specific Helmholtz Cloud Services. In order to know who is allowed to book which type of resource in which capacity, an access list is programmed in the service database Plony. This brings together the information about the resource booking in the Cloud Portal and the group membership in the Helmholtz AAI.
Any completely transparent interconnection of Helmholtz AAI with other federations, such as EGI Checkin, ELIXIR/Lifescience, or Indigo IAM, is notoriously problematic due to many degrees of freedom of such implementations (and thus potential conflicts) in terms of user lifecycle management, authorisation management, group memberships, applying policies and more. In coordination with specific user groups using infrastructures from other federations and communities, we will define specific technical and administrative procedures to enable the negotiated use cases while minimising the abovementioned problems.
Following the first presentation of envisioned future developments of HIFIS in the Helmholtz Incubator Workshop in July, further details are planned to be worked out until fall 2023.
The second iteration of the Incubator summer Academy will take place in September. As last year, the event, which is jointly organized by all Incubator platforms, will offer a broad portfolio around the topics of Research Software Engineering, Data Science or Artificial Intelligence.
The draft rules comes to an accepted solution for all touched diciplines: general legal, VAT, and state aid topics. Stakeholders such as data protection officers and works councils are informed. The Assembly of Members of the Helmholtz Association have received the document for signature, so that the use of technical resources within the context of the Helmholtz Cloud are regulated.
For the user to easily find up-to-date documents related to the service (e. g. AVV templates), we will set up a download area in Plony. Logged in users find all documents related to the service in one central place. In order easily find the download area, every service card will include a counter, thus giving the user the information how many documents can be found in the download area (including a link to get there).
Having worked out the future plans for HIFIS and possibly with a first feedback from the Helmholtz Assembly of Members to be held in September, we will gather in an all-hands meeting in or nearby Dresden / HZDR. HIFIS stakeholders such as SAB and Federation Board will be invited.
Group Accounts and Robot Accounts are not yet fully supported by DFN AAI and thus Helmholtz AAI. As such accounts are envisioned to play an increasing role in the Helmholtz AAI and Helmholtz Cloud, HIFIS will foster to establish an AARC guideline to fully support such accounts. In parallel, HIFIS will provide first conceptually compatible implementations so that ongoing use cases can be supported soon.
For the first time, an award for high-quality research software will be granted in Helmholtz.
With experiences made during the annual housekeeping, user deprovisioning, adoption of new use cases (for example, including robot accounts), and updates in the upstream AARC guidelines (SIRTFI-v2, REFEDS Assurance Framework V2), the AAI policies may need regular updates. This and yet to be defined elements will be part of the policy review process that is due to be defined. As this strongly related to similar review activities of the HIFIS Cloud cluster for the Helmholtz Cloud services, there will be a close collaboration between clusters on this.
Best practice workflows are to be defined on the basis of the guidelines for the sustainable handling of research software that are currently being established at Helmholtz. These can be used to check compliance with certain criteria as automatically as possible and to award software quality badges for research software. Initial work in this area is expected to be ready for presentation towards the end of 2023.
New functionalities will be enabled by a closer connection of the services Helmholtz Codebase and Mattermost with the Helmholtz Cloud Portal. This will, for instance, add support for VOs or allow the automatic deprovisioning of users.
During winter, the “Incubator Framework 2.0” will be finalized. Future development plans will be carved out and prepared for presentation to the April’s Helmholtz Assembly of Members. This is a follow-up to the preceding decision from October 2023.
Two important agreements of the Helmholtz Cloud are finalized: The Helmholtz Cloud Ruleset regulates the roles, duties and tasks for providing and using the services. It is flanked by the Joint Controller Agreement of the Core Components that covers the GDPR requirements. Find more information here.
By taking over his role as DESY Head of IT at the beginning of May 2024, he also became the spokesperson for the HIFIS platform. Read more…
Based on a review of the overall process and policies involved in VO registration, implementation of the process in plony database, and improving interconnection with AAI community proxy (Unity IdM), the VO registration will be streamlined and made more transparent to users.
To ensure that cloud services meet high quality standards above what is absolutely required during the Service Onboarding, the HIFIS Cloud team evaluates criteria to measure service quality during operation and identified five key fields, read more.
A first version of the envisioned SCIM API for querying information on federated groups (user group memberships) by services will be implemented. The basic development was mainly completed in 2023, ongoing works include configuration and extensive testing.
As a logged-in user one can book, manage and delete resources without writing emails to the help desk. The entire process is handled in the Cloud Portal and the Helmholtz Cloud Agent, and transmitted to the service provider without any manual steps.
The annual all-hands meeting will take place in September, visiting our HIFIS team member DLR in Cologne. Further details will be published on the events page. HIFIS stakeholders such as SAB and Federation Board will be invited.
Service providers can now configure all aspects around resource booking in Plony. This includes the content of the resource booking form displayed in Cloud Portal, how many resources are offered as well as access control to them. The modeling of the corresponding process as well as the implementation in Plony was subject to a bachelor thesis written at HZB.
The five Helmholtz Information & Data Science platforms - Helmholtz AI, Helmholtz Imaging, HIFIS, HIDA, and HMC - are crafting a two-week program to cater to participants with diverse backgrounds and interests. We offer a wide array of course packages, ensuring that you find the perfect fit regardless of your level of expertise.
The first productive cloud portal integration will allow users to to create teams in Mattermost via the Helmholtz Cloud portal. This integration is being carried out in close cooperation with the cloud cluster and provides the technical basis for further future integration.
An increasing number of IdPs supports MFA for federated login. To improve user experience, i.e. avoiding to bother users with multiple MFA entering, honoring MFA in the login chain is being implemented. Corresponding documentation will be provided and signalling the MFA status will be prepared.
In cooperation with Nextcloud and initiated by HIFIS, Nextcloud has now rolled out an update that enables federation: Several Nextcloud instances can join together so that the user sees them all together in one view.
A short survey reveals a ranking of which collected services ideas are most relevant to users.
As a blueprint, Plony Database, one of the cloud core components, is going to be the first service in where users are automatically deprovisioned based on the Helmholtz-AAI information about deleted or deactivated accounts.
HIFIS is developing a draft process to simplify the conclusion of the necessary data processing agreements (DPAs) within the Helmholtz Cloud. The DPAs supplements the existing Joint Controller Agreement and must be must be concluded between the service providers and using centres. This draft process is submitted to the Helmholtz Data Protection Officers (AK Datenschutz) for their review and feedback.
With experiences made during the last 2-3 years (annual housekeeping, user deprovisioning, adoption of new use cases) the AAI policies will be reviewed and potentially updated in close collaboration with all other clusters and AAI participants (e.g. NFDI)
The documented baseline of defined criteria makes the quality of the service portfolio as a whole measurable and comparable.
The fourth iteration of “Talking about Education Across Communities at Helmholtz” (TEACH) conference - is being organized together with HIDA, HMC and Forschungszentrum Jülich. For the first time as a face-to-face event.
To this end, technical options are evaluated and a strategy for determining user satiscation is developed.