HIFIS Technical Platform - Whitepaper¶
Note that this documentation is outdated.
Table of Contents¶
- Document History
- Development Methodology
- Source Code and Build system
- Operational Concerns
Current Version: 0.3
Current Version Date: 2020-07-20
This document is a basic description of the technical design for the technical platform (TP) within the Helmholtz Infrastructure for Federated ICT Services (HIFIS).
The TP is a central part of the Helmholtz Cloud. Its purpose is to be the main entry point for users looking for a service which could be used for their purpose. In this position, the TP has to interact with almost all components of the system. The schema in Figure 1 shows the designated architecture for the Helmholtz Cloud from the view of the TP. For all components outside of the TP, there are more detailed architecture schemas. For the sake of clarity about the architecture of the TP these parts are excluded here.
Evaluation of existing Solutions¶
In order to avoid duplicate work, existing solutions for the functionality of the desired functionality of the technical platform have been reviewed in order to gain insight about the functionality provided and in order to determine, if they are reusable for the purpose of HIFIS.
EOSC Whitelabel Marketplace (WLMP)¶
The EOSC Whitelabel Marketplace is currently in development. There is already a deployed version available (https://marketplace.eosc-portal.eu) which provides the existing functionality.
- Ruby 2.6
- Database: PostgreSQL
The EOSC Whitelabel Marketplace solution offers a wide range of functionality also expected as part of the technical platform for HIFIS:
- Login via AAI (EGI AAI)
- Search services
- Hierarchical navigation to services
- Detail page with different flavors
- Request access to services
Although the WLMP seems to offer quite a lot of functionality which is also needed for HIFIS, there are some major disadvantages:
- tight integration with Jira
- tight integration with xGUS (helpdesk system, based upon BMC Remedy Action Request system (commercial)
- usage of Google ReCaptcha (mandatory)
- usage of Google Analytics (optional)
On top of that, the existing documentation is insufficient in order to setup a running system in a production environment. First attempts to setup a test instance on Ubuntu 20.04 with nginx as reverse proxy have not resulted in a usable system.
EUDAT Data Project Management Tool - EUDAT (DPMT)¶
DPMT is in production for the use of the EUDAT network. It serves as the administrative backend for all information related to the service portfolio management. The actual user-facing part of the service catalog is much simpler and seems to be another application (https://www.eudat.eu/catalogue).
Onknown, no license file in GitHub repository
- Current version
- Python 2
- Plone 4
- Upcoming version
- Python 3
- Plone 5
DPMT offers a very complex and detailed object model.
Although there are a lot of requirements formulated in the HIFIS proposal, very few of them are so detailed, that they can be implemented without further discussions and without further evaluation of possible implementation ways. During the discussions in the various groups there are also new requirements being created, which have to be considered and can be added to the list of already existing requirements.
This situation is difficult in a classic development process, where all requirements should be listed, prioritized, specified and then implemented in a determined order. In order to address such uncertainties, software development is more and more often organized in a process following the Scrum model. Scrum defines ways how development teams can work in situations, where not all requirements have already been defined and where even new requirements can come up during the development process.
Based on those properties, the development of the TP will follow the Scrum process, which allows for the rapid development of a first version and ensures that there will always be a working version at the end of a development cycle.
At the moment of writing, the possible technologies for the implementation of the TP are still being evaluated. This whitepaper is also intended to give an overview about the proposed technologies.
System Architecture HIFIS TP¶
From a high level there are the following components which are an integral part of the TP:
Aside from that, there are at a minimum the components
Authentication and Authorization Infrastructure (AAI)
The service catalogue has a special place, since at the time of writing, it is still to be decided, how this part will be implemented. It is possible, that HIFIS will use existing software for the implementation of it, so that the TP itself will only act as a proxy and keep a cached copy of the service metadata which are relevant for the TP.
One question which concerns the backend as well as the frontend is the internationalization. At the time of writing, it has been agreed upon that the language for the user interface in the first public version will be English. At later stages it will probably be necessary to add more languages for the user interface.
Separated from the internationalization of the usere interface is the question of the internationalization of the contents. At the time of writing, it is assumed, that the content of the service catalogue will not be multilingual.
The backend will contain at least the following components:
ReST API for the user interface (UI) frontend and for use by 3rd-party applications which may at a later time want to use the TP functionality without the use of the web UI
Service Catalogue Cache/Proxy
Support for Service Orchestration/Meta-Services
While there are some components which will be in a minimal version necessary already for the first version (ReST API, Service Catalogue, Monitoring, Authentication/Authorization), there are other components which will be implemented at a later stage. Nonetheless, it is important to include them at least as components without further details about the implementation, so that their role and position in the HIFIS architecture can be determined.
The backend will be build on top of the Spring framework, which is an established dependency injection (DI) framework that is very mature and stable and it is used widely for building robust and maintainable server side applications. On top of providing the foundation for the implementation of the TP, Spring also provides existing modules for different tasks. In our case, this means for example that the integration with the Helmholtz AAI can be done very easy with the use of the Spring Security framework.
For the persistence layer of the components in the backend a database system will be used. One possible database is Neo4J [Neo4J].
The build management will be done with Apache Maven [Maven].
As a testing framework Robot [Robot] will be used. Robot is a versatile framework which provides an easy to use API to different underlying frameworks. The test cases can be written using in a syntax resembling natural language which allows adding more test cases very easily also by people without extended developer skills.
For the implementation of the UI frontend, a stack of modern technologies will be used. In order to build on a solid base for the generation of the UI, a framework for WebComponents, most likely LitElement [LitElement] or Polymer 3 [PolymerLibrary] will be used.
From the perspective of the system architecture is has yet to be decided, if there will be a separate server component for the frontend which then communicates with the backend ReST API.
As a testing framework the before mentioned Robot [Robot] will be used. This is especially useful for UI testing, as it allows for writing UI tests with a very simple syntax which is then translated to Selenium [Selenium]
In the implementation of the frontend, aspects like responsive design and accessibility will have to be considered and taken into account.
Although not a separate part of the TP, it is necessary to give some consideration to the management of users and user data on the TP. In HIFIS, the main component for the managament of users is the AAI infrastructure. This means, that there will be no local user management within the technical platform. All authentication/authorization is being done with the external AAI. Nonetheless, there is the need, to handle data from users on the TP. This can be
information about requests to use certain services
in the future information about the personal dashboard
From the view of the TP, the user-specific data handled on the platform should be kept at an absolute minimum. It has always to be considered if additional data could not be better handled by means of the AAI.
Source Code and Build system¶
For the management of the source code, git will be used. In order to support the workflows around the version control, Gitlab will be used in order to provide additional tooling.
As mentioned in the sections TP Backend and TP Frontend, Maven will be used as the primary build management tool. For the management of the UI libraries, the package manager npm from Node.js will be used. Both tools are very mature and are widely used for software development projects.
CI/CD will be handled through Gitlab pipelines. This provides a very tight integration with the VCS, so that every commit automatically triggers a build and errors are visible immediately.
Successful builds will trigger an automated deployment, either to an existing server with a combination of reverse proxy and application server or using a containerízed infrastructure. The method of deployment is still being evaluated at the time of writing.
The software for the TP is meant to be deployed on a central server for the whole Helmholtz community. Since it is impossible to determine the average/maximum system load at this time, there are no explicit developments planned for high availability of the platform. The platform is however designed, so that it will be possible to operate it in an environment with load balancing and other means to ensure HA.
Load balancing for the provided services is considered, but is not targeted for in the first versions of the TP. In order to allow some kind of load balancing for similar services, it will be necessary to determine, which preconditions have to be fulfilled so that similar services are considered for a load balancing approach.
[LitElement] LitElement. A simple base class for creating fast, lightweight web components. URL: https://lit-element.polymer-project.org/ (2020-04-24)
[Maven] Apache Maven. URL: https://maven.apache.org/ (2020-04-24)
[Neo4J] Neo4J. URL: https://neo4j.com/product/ (2020-04-24)
[npm] Node Packet Manager. URL: https://www.npmjs.com/ (2020-04-24)
[PolymerLibrary] Polymer Library. URL: https://polymer-library.polymer-project.org/ (2020-04-24)
[Robot] Robot Framework. URL: https://robotframework.org/ (2020-04-24)
[Selenium] Selenium UI Testing Framework. URL: https://www.selenium.dev/ (2020-04-24)
|AAI||Authentication and Authorization Infrastructure|
|API||Application Programming Interface|
|HIFIS||Helmholtz Infrastructure for Federated ICT Services|
|VCS||Version Control System|