How HIFIS helped in the aftermath of HZB’s cyberattack

Helmholtz-Zentrum Berlin (HZB) became a victim of a cyber attack on 15.06.2023. To protect against it, we have shut down all IT systems. The research centre cannot be reached via the website, email or telephone at the moment. We ask for your understanding.

This was the message on hzbblog.de the day after HZB discovered it had been the victim of a cyber-attack. Six months later, our colleagues finally got their accounts back. It’s time to take a look at what happened and understand how HIFIS could help. Andreas tells us the story.

Could you first provide a brief overview of the cyberattack HZB experienced last year? What were the initial signs of compromise, and how did the attack unfold?

I would like to share this story here because many research centres can learn from our experience. Actually, I was one of the first to be affected when the cyber-attack hit us. I was on call that week of the 15th of June [2023]. On a Thursday morning at 6am I got a call from campus security that some of their systems were not working, so I just assumed that something had broken down. I tried to connect remotely via the remote desktop protocol¹ and already noticed that the remote desktop terminal server was not responding. So, I tried to go to the documentation page and saw that I could not access any document on the SharePoint. I had to ride to the office at 6 something am. When I arrived, I still had no clue what the issue actually was. I just had a bit of a weird feeling.

When I got to the office, still tired, I saw a colleague, who’s responsible for SharePoint, run by my office. He’s not usually in that early in the morning, so I went to see him. He told me that he also received emails from SharePoint, monitoring emails, about files being encrypted. So, I left it to him to do further analysis. He found out that the encryption had been going on since some hours. We had already called the IT security officer, and when another colleague arrived, we very quickly called DFN to shut down our internet connection. Which was a good call, at around 7am or so. That’s when the distribution of ransomware stopped because the software lost contact to its control server.

But by then, a lot of damage had been done, and we shut down all of our systems within a couple of hours. We couldn’t shut everything down immediately because the synchrotron has to be shut down very carefully and so on. It was half a day of shutting things down.

What were the immediate challenges researchers and staff faced in the aftermath of the cyberattack?

We quickly realised that we were going to be offline for quite some time. People started to revert to their private email accounts very quickly, as did I for example.

I contacted DESY about migrating our DNS², which was the most important thing in our eyes, to have some DNS running somewhere, to be able to reach anything. We didn’t have any reachable services at that time, but it was a prerequisite for doing anything at all. If our domain was down, it would be pointless to do anything else.

We had to follow protocol; communication was very important. First of all, communication between colleagues. We had to inform our staff by using private email accounts, private phone numbers and hoped that the message would somehow spread. On that day, the HZB Scientific Retreat took place, so we were able to inform all the OU heads via a phone call. We also did an announcement via the fire department loudspeakers to inform people on our campus in Wannsee. In Adlershof, people put up some notes on doors and bulletin boards. And at the same time, we had to fulfil our duty to provide information to the many external stakeholders, all by telephone call or from private email addresses.

Only a few days later, regular on-site staff meetings were started to inform on the current situation. Getting information to people was surprisingly difficult: what happened, what happens next, what happened to my data, to my files, how can I do this, how can I pay this bill or do this calculation? At this moment, we did not have answers for many of these questions.

How did the attack impact the functioning of your research centre, both operationally and in terms of data security?

Our synchrotron radiation source, BESSY II, had to be shut down, that means no large-scale research was available for weeks. A great success so early after the attack was that the operation of BESSY II was already restarted on July 3rd and our partners at PTB and BAM had access to the light again. The beamlines operated by HZB were still out of operation at this moment. We didn’t have a running user coordination system anymore, so all measurements and the next calls for proposals had to be cancelled until December 2023. We announced this news on our social media channels and hoped it would spread to our users. We know this was a very difficult situation for all our users who were eager to perform their experiment at our light source but could not. The cyber-attack also had an impact for lots of PhD students who lost time to finish their thesis.

But the administrative processes were also affected and had to continue. The administration still had some paper forms in their drawers and could pick up old processes. And it was good to see that these old paper processes still worked, although very slowly, producing very large piles of paper, especially for paying bills, the yellow forms we had to sign off for payment, … It was interesting to get all those circulation binders, we had a lot more of those going around again. I actually like them for nostalgia’s sake. But all in all, it makes us appreciate even more how fast digitalized bureaucracy can be.

I’m also a move-up on the works council, and its processes started to work quite promptly again. For example, to hire new people, the works council has to sign off on that. Human resources and the works council were very quick to reestablish paper processes, which was good to see. The works council also went back to meeting offline very quickly.

When did HIFIS step in to assist HZB during this crisis?

Basically immediately, if you count the DNS. Without HIFIS, I don’t think we would have that tight connection with DESY, to be able to just send them an email from a private email account and negotiate how to migrate our DNS within a day. That was very good to see, how it worked.

In general, as soon as we requested something, HIFIS jumped in immediately, there was basically no delay. The following days we also started to use the Nextcloud-instance provided by DESY called DESY Sync & Share to upload information for our staff but also for the internal communication between departments. We had one read-only folder for the staff, where we uploaded FAQs and basic information, and we had internal folders for some of the departments, where they could coordinate themselves. We’re actually still using them to some extent today and we are just migrating back now, as our intranet is starting up again.

So that was less than a week, I don’t have the exact start date, but I do have it on the virtual machines which we requested from Forschungszentrum Jülich: that was 10 days after the attack. We had to do some internal negotiations first: how do we want to start up again? Do we want to rely on external cloud services, like commercial cloud services? Do we want to rely on the HIFIS virtualisation platform, or how quickly our own virtualisation platform would come back? But then it became clear rather quickly that our own platform would not be back online anytime soon, and before paying for resources, we obviously took the HIFIS route. I contacted Forschungszentrum Jülich and the colleagues there were very quick to give us OpenStack resources and we could deploy some virtual machines, which was very helpful.

What HIFIS services did you use in place of your own during the recovery period?

We mainly use DESY Sync & Share and OpenStack (HDF Cloud) from Jülich very extensively. We actually still use OpenStack to this day, to access our SAP for example. The first things we set up there were Linux based terminal servers and VPN³ solutions, but other tools followed, for example OpenProject⁴ to track our tasks and testbeds for other services which we rebuilt locally after figuring things out in the OpenStack cloud. We still use a lot of those cloud based tools today and are starting the progress of migrating back just now.

Surprisingly, it seems that the colleagues were happy to work with Linux, even to be able to work with anything at all. Now they’re happy to go back to their Windows but it was still nice to see that they are open for open-source solutions. Also, however not as extensively, we used Mattermost and Codebase from Dresden.

For DESY Sync & Share, we had public shares that were password protected. For the other services (OpenStack, Mattermost and Codebase), we used social accounts: Google accounts, GitHub accounts, maybe some ORCID, I don’t know. That’s also what made it a bit tricky. Because on OpenStack there were only two/three accounts for people who were managing the virtual machines. But on Mattermost and Codebase all sorts of HZB users tried to join their old Mattermost teams and GitLab projects and creating all these social accounts which now have to be rolled back very extensively to be migrated back to the actual HZB account. We didn’t want to create too much of a clutter and tried to keep it small. The HIFIS team also got some social accounts on DESY Sync & Share, but these have already been rolled back.

Were there challenges or limitations encountered while transitioning to HIFIS services and how were they addressed?

Especially with the social accounts being necessary, we dearly missed our identity provider and the connection to Helmholtz ID, it was a very, very big problem. If we had a backup IdP⁵ , it would have been so much easier to communicate with each other, to get access to resources, without always having in the back of your mind the rollback of the social accounts sometime in the future. That was a very, very big limitation. The HIFIS team supported us excellently, so we did not have any other problems at all.

From your perspective, how much did HIFIS contribute to the resilience and recovery of your research centre following the cyberattack?

From my perspective, it was absolutely critical to have the support of HIFIS. On the one hand, we had the provided HIFIS services, we were able to migrate our DNS to DESY very quickly, that was very helpful. But we also have access to the know-how from other centres, could talk to experts and learn what solutions they had in mind for things like multi-factor authentication (MFA), which we wanted to implement after the attack. Just talking to people who had different perspectives was extremely helpful.

We even received a lot of help offers from all kinds of HIFIS colleagues, which we couldn’t even accept at this time. They wanted to help, but we couldn’t provide them with accounts in our ecosystem, we were still offline at the time, but they were so eager to offer their help, that also felt nice.

Looking back, what lessons have been learned from this experience in terms of cybersecurity preparedness and response?

That was, yes, a lot of learning! What we have experienced within the last eight months you cannot learn at school or in a tutorial, you have to go through it to actually learn how things go and how they go sideways.

Obviously, there are several things we should improve- Specifically for the case when everything has to be shut down. We had plans for shutting down individual IT parts, but not for the whole thing.

One thing that became clear very quickly was that multi-factor authentication should have been implemented sooner. So, my strong recommendation is: If any centre hasn’t rolled-out MFA yet, they should start now, not tomorrow.

Another thing we have learnt: you need a good network segmentation. We didn’t have that strict network segmentation before. Now, we have implemented very strict network segmentation, and that makes rebuilding services difficult, because if you rebuild old services that were designed with an unsegmented network, and now you rebuild them with a segmented network, you have to untangle those old structures and bring that old service into the new network world, which takes a lot of time. In normal operations, we don’t always have the resources to do this, but now we not only have the opportunity, but also the obligation.

A big lesson to learn is that external IT emergency services must be prepared. We need external DNS fall-back, we need external identity management fall-back, storage fall-back, email fall-back… We had an opportunity to try out Grommunio as an email fall-back. T-Systems was very quick to set up a Grommunio server for us, where we could move to. Grommunio has its own issues, but it was still very important to have email communication back. Not receiving emails is really bad, as it turns out, if you are not able to receive emails, you’re kind of in the dark!

Having a fallback for emergency email services is very important, and basically anything that has to do with email: DNS, identity management and so on. And obviously the fall-back has to be secure and separate from all your internal IT.

Moreover, we should improve the availability of contacts in case of an IT emergency. A list with phone numbers should be stored somewhere safe, not only on our IT, but somewhere in the cloud, that would have made things much easier as well.

In what ways do you anticipate leveraging HIFIS services to strengthen HZB’s cybersecurity posture in the future?

To basically cover all the things I just stated, maybe except for the email part, which can be tricky. We don’t have any email provider in HIFIS currently and I don’t think we will have one anytime soon. However, providing external fallback storage or storage of emergency protocol resources in HIFIS would be very helpful. Having a fallback for DNS via HIFIS would be very good.

I believe that the HIFIS security work package, which is hopefully coming soon, will bring not only resources but also know-how into the Helmholtz world, into the HIFIS world.

We could rely a lot on HIFIS for knowledge and for resources, but this already tightly knit community needs to be even tighter. We just need to talk to each other more, we need to strengthen this community and the interaction between the Helmholtz centres, even if they’re not in HIFIS, we need to bring them in somehow to talk about security issues and to learn from each other. That’s basically it.

About the interviewee

Andreas Klotz works at Helmholtz-Zentrum Berlin where he manages the local HIFIS Cloud cluster. His experience focuses on Linux, Open-Source software, and Federated Identity Management. As many other HZB colleagues also did, he temporarily switched focus from his usual work to help HZB recover from the recent cyber-attack.

Want to know more about HIFIS?

After witnessing firsthand how HIFIS could help HZB, you might be curious to explore further how HIFIS can benefit you and your institution.

HIFIS provides and brokers digital services for everyone in Helmholtz and collaboration partners. These range from compute resources in the Helmholtz Cloud, to training courses on Continuous Integration in GitLab, to a directory for research software, all of which can be accessed with your home institution’s login thanks to the Helmholtz ID.

Find out how HIFIS can help you as a researcher, as a software engineer, as a cloud service provider or as an IT support expert.

You can also learn more about our commitment to resilience here.

¹ Remote Desktop Protocol (RDP): A network protocol developed by Microsoft which allow accessing remote computers with a graphical interface.

² Domain Name System (DNS): Colloquially sometimes called “the phonebook of the internet”, mapping technical network addresses (IP addresses) to human readable addresses (Domain Names) like www.helmholtz-berlin.de.

³ Virtual Private Network (VPN): A networking technology which allows to connect two separate networks in a secure way.

⁴ OpenProject: An open-source project management software.

⁵ Identity Provider (IdP): An authentication interface which is used to log in to remote services, which do not have access to internal identity management systems, by using local credentials.