“Hello, my name’s Andrés. I’m about to undergo a cyberattack”

This is the first-hand account of the 24 hours following the cyberattack on the University of Castilla-La Mancha, written by that university’s Director of ICT. The author has no hesitation in referring to this incident as a “terrorist attack” and reflects on the effects of such attacks and the need for greater cybersecurity.

My mobile phone rings

The time is about ten o’clock in the evening on Sunday 18 April. While I’m having supper at home, reflecting as on so many other Sundays on just how quickly the weekends fly by, I receive an alert signal on my mobile phone. This particular weekend was about to finish sooner than I had anticipated. At the University of Castilla-La Mancha we had installed quite some time previously a service of continuous surveillance of the university’s critical digital infrastructures and services. While alerts of this kind are thankfully rare, they nevertheless form a part of our everyday activities. This will not be the case today, however.

I examined the alert to determine which service was affected, and found that it was the university’s website. An alert of this kind triggers off a protocol of actions to be taken at various levels, so I decided to wait a moment to see whether, as on previous occasions, the service would be returned to normal by the team of operators working from the external surveillance centre. I remember thinking that it could be a problem with the servers in which the website is hosted or even with the content management system that we use to constantly update the website. I was wrong. Less than an hour later I received a phone call from my colleague Julian, our Systems Director. Getting a call like this on a Sunday evening could only be bad news: if Julian was calling me, this meant that the protocol had been raised to the next level and there was going to be a major impact on the service.

“Good evening, what’s happened? Has the website management system crashed?”, I asked, trying to keep a fairly relaxed tone in my voice despite my fears that the problem with the website must be serious and they were no doubt calling me to confirm that it was going to take some time to get back to normal service. What took my concern to an entirely different level was not the reply in itself but the tone in which it was delivered. The riposte was direct, but the tense voice on the other end of the line stammered a little. “No, it isn’t a problem with the management system. It’s the database: it’s all been encrypted. We’ve been hacked.”

The first assessment of the damage undergone was devastating: numerous servers had been encrypted and several databases were completely inaccessible

Following the protocols for situations of this kind, the first action had already been taken as soon as the encryption had been identified: external connections had been cut so as to contain the attack and interrupt all communication with the hackers. Logically we would need to take the next step in situ, going straight to the building where we host the university’s most sensitive technological infrastructure. My family couldn’t believe it when I told them I was going out in the car so close to the time limit for the curfew that was then in place. “They’ve done what???” With so little information available, it was equally difficult to try to explain to the Vice-Rector and to ask him to notify the Rector. We still didn’t know the extent of the threat, but we were already in a situation that was quite unprecedented for our university.

The scene of the attack

The city was already almost completely deserted. I passed very few other cars during the short drive to the university, making a conscious effort not to break the speed limit. When the security guard saw me, he told me that two people had already arrived at the building. Faces were tense and serious.

The first assessment of the damage was devastating. Numerous servers had been encrypted, various databases were completely inaccessible and, of course, there was a message with four huge letters to make it quite clear what was going on: RYUK. Seeing this didn’t have any particular effect on me any more, I’d already had enough shocks for one evening. The contact address was also quite clear: this was the access route for a negotiation to recover the encrypted data with a ransom the amount of which was not specified in the message. The decision taken next was unequivocal: the progressive shutdown of all the university’s technological infrastructure housed in the Data Centre. Now the prospect of the following morning really made my hair stand on end: the university would be faced with starting the week with no ICT support of any kind. By this time, we knew that all the main elements of the computer infrastructure had been hacked and it was not an isolated incident. All the servers that performed some sort of essential function had been encrypted, and some of these functions were farmed out to up to half a dozen servers distributed between the different campuses and even to the Cloud. And they were all encrypted. At this point we took a decision that would have a major influence on the future direction of events. I honestly believe that our technological architecture, resources and technical expertise are in line with the strict requirements of the university environment. Nevertheless, our analysis in the early hours of Monday 19 April, isolated as we were in our Data Centre, was dominated by our humble acknowledgement that the situation was more than we could handle. We could clearly see right from the start that we were not going to be able to get out of this without outside help. This was the first time we had ever had to deal with a situation like this, and we knew we needed somebody to assist us.

Our communication strategy was based on clarity and constant information

We decided to return home. It was two o’clock in the early hours of Monday and, just before leaving, I posted a message on Twitter to announce the incident. Even in the midst of the confusing situation we were experiencing, we knew that it was important to let everybody know that there was a problem. We normally provide information about serious incidents through this particular social network, and we decided to do the same for this extreme occasion too. I understood quite clearly that we had to do this, but had not anticipated the repercussions of my decision for the entire university community. My short message had the result of setting in motion all the ICT services of the university straight away on that Sunday night. The return home was another difficult moment, since we were leaving with the plain truth staring us in the face and the realisation that it was impossible to do anything else at this time of night. The security guard looked at us worriedly, seeing the intense concern sketched on all our faces. “You haven’t been able to solve the problem, have you?” “No, it’s going to be very complicated,” I answered exhaustedly. I will never forget that journey home and my feeling of nervousness with the police cars in the distance to enforce the curfew in place at the time. “If they stop me and I have to explain why I am out and about at this time of night,” I thought, “they will find the excuse so strange that they’ll believe it.”

A morning of uncertainty

Once the sun had risen that Monday, I started the process of seeking help. At a time like this, with your nerves at breaking point, you look for options where you think you will get not only a well-considered response based on sound experience, but also one that can be carried out as quickly as possible. Among the various options that I had identified the previous evening, I decided to call the Spanish telephone operator Telefónica. I was well aware of its contribution to reacting to other incidents and, in addition, I was convinced that they would respond rapidly as soon as I had explained the situation. This was indeed the case. At 9.15 a.m. we had already started the first meeting to analyse the situation held between the Telefónica team and the crisis response team that we formed with my colleagues. The first analysis was brutal: we would only know the full extent of the damage in the following 24 hours, but the type of attack resembled those suffered recently by other public administrative bodies and the recovery would be very slow. My report to the university’s governing body was equally brutal.

We were once again in an extremely nerve-racking and uncertain situation. On the one hand, we would need to wait for an assessment of the possibility of external assistance, and on the other hand we also notified the Spanish National Cryptological Centre’s Computer Emergency Response Team (CCN-CERT), where they asked us to create a ticket in their incident management tool. At the same time, being fully aware that external aid was essential, I found out about their contracting process through their emergency procedures. In under 90 minutes we had already obtained authorisation from our legal advisers and management team in order to launch this procedure. This may seem like a trivial detail, but contracting external services is no easy matter in public administrative bodies, and in this situation it was important to try to complete the full procedure.

It was going to be necessary to incorporate new cybersecurity features in the face of the highly probable danger of being subjected to a second wave of attacks

At the same time, while the teams started to put together the response strategy, I started to turn my attention to another essential aspect of moments like this: communication. I recommended that we should provide clear information about the problem we were undergoing as quickly as possible. The classes and lectures and the working day had already started and it was important to explain what was happening. We agreed to issue a first announcement through the institutional account managed by our communication agency, which not only totally agreed with our approach but also quickly drafter the first message and confirmed our communication strategy of clarity and constant information.

By the end of that Monday morning, we were in a position to hold our first crisis management meeting. These meetings would be repeated two or three times per day during the first week and on a daily basis until the incident was finally overcome. Nevertheless, it was the first meeting that laid the basis for the operation to follow. Identification of the malware used, the extent of the damage, the protection of critical features, the recovery of services and continuous surveillance were the basic elements for structuring the work to be undertaken. The in-house team would concentrate on the recovery of services, but while identifying right from the start that it was going to be necessary to incorporate new cybersecurity features in the face of the highly probable danger of being subjected to a second wave of attacks.

Organising the response

The afternoon of that Monday 19 April was a roller-coaster of emotions. Attacks of this type can combine the encryption of servers and databases with the encryption of dozens, or even hundreds, of other devices. Fortunately, this was not the case for us. Perhaps due to the rapid response on Sunday night or perhaps thanks to the good work carried out in the form of the renovation of user station equipment rolled out just a year before, during which the security measures applicable to the devices concerned were increased considerably. On the other hand, the seriousness of the attack was confirmed: the ransomware used was a known variant of RYUK and the number of servers affected was still not yet defined. Security copies thus became the most valued material available to us. The strategy of creating security copies establishes different storage spaces, but the risk of losing the activity concerned after the first day of the month or even of losing a larger amount of data still remained. In any case, there were elements on the basis of which we could start to recover the situation. The directory of users was available and the different levels of security copies gave us a certain degree of reassurance.

That evening, less than 24 hours later, the strategy for recovery, organisation and communication was now perfectly clear. These were the three axes around which we worked during the following weeks, and the establishment of which helped to identify the actions and responsibilities for action in each case.

The roll-out of applications in the Cloud enabled us to re-establish some twenty or so applications

The basic criterion for the recovery of infrastructures and services was clear right from the start: a high level of cybersecurity would be required for our technological architecture as a result of the risk we were facing. This involved modifying the hierarchy of privileges in terms of management and infrastructure; the establishment of policies of “zero trust” with regard to external entities and their application even with regard to some segments of the university’s own internal network; the continuous surveillance of the activities of user devices with a view to detecting malware patterns, for which the roll-out was required of EDR (Endpoint Detection & Response) facilities in our current antivirus system, made possible thanks to the support provided by Microsoft; the protection of the identities of users, which have been shown to be one of the most frequent vectors of attack, and against which double-factor user identification strategies have been developed; together with a strict requirement for operating systems with ongoing support in the user stations connected to the university network.

Another essential axis identified was the organisational aspect. Despite the jokes and other marks of good humour that continued on that first fateful day (we try to keep this spirit alive at all times), not to mention the piles of pizzas, cartons of Red Bull and endless pots of coffee, we could clearly see that this was a task to be continued over many days and that, just like in a long-distance race, we would need to conserve our resources accordingly. For the response team the following days were going to be long, every bit as long as that first Monday, but not indefinitely so, since it was vital to ensure everyone had the chance to rest. In addition, it was decided right from the beginning that we would work in a group, rather than establishing work shifts that could lead to big advances over a 24-hour period but then, due to a lack of communication between teams, end up causing delays and provoking errors. Finally, it was decided to keep the response team isolated from outside queries, pressure and anxieties. All questions to or from the team, even those involving colleagues in other units in the university’s ICT department, were to be channelled through a single contact point. I’ll leave it to the reader to guess who played the role of the “contact point” concerned.

The third axis on which we based our response to the crisis was communication. On the basis of assuming the situation in which we found ourselves and opting for transparency about the incident with regard to both our own university community and society in general, we agreed on a communication protocol that insisted on total transparency and a predictable pattern of provision of information. Right from Tuesday 20 April onwards, daily communication was ensured between the communication agency and the university’s ICT department. Just a few minutes after 9 a.m. during those two weeks I regularly received a phone call from my colleague in the communication agency during which we identified the latest news to be transmitted during the day-to-day progression of the incident. The press releases drafted by the agency and quickly approved by the Vice-Rectors responsible for ICT and communication respectively usually contained two dominant ideas: an element of progress and an element of raising awareness of the changes arising in the different departments, as a result of the increased levels of cybersecurity.

Much more than 24 hours

Although those first 24 hours turned out to be of key importance, it took us much more than 24 hours to resolve the incident. The recovery was organised in terms of a prioritisation of the relaunching of services that obviously involved during the initial phase the interdepartmental technological architecture functions, with the first team taking its place in the Wednesday of the first week, and in the other phases the services that combined a favourable balance in terms of their impact on activities and their recovery time, recovering the teaching and collaboration back-up systems right from the Thursday of the first week.

Some of the choices made with regard to the strategy of roll-out of digital services in the university first developed years ago also produced tangible benefits in this situation, in the same way that they also proved themselves during the lockdown in 2020. The roll-out of applications in the Cloud enabled us to re-establish some twenty or so applications in a single operation during the Friday of that first week. This was one of so many different lessons learned: the resilience of architectures of this kind in the face of ransomware attacks is greater than that offered by traditional architectures.

Nevertheless, complete recovery took several weeks. On the one hand, this was because all services were analysed for risks before returning once again to productive environments, even leading to the code refactoring of some applications. On the other hand, the network architecture, its segmentation and policies of confidence between zones changed to such an extent that the reincorporation of devices, and especially of those exposed to external contact, needed to be analysed in great detail. I also have to admit that this was because some departments (such as that of unified communications) did not have an adequate policy of continuity. Be that as it may, it is also clear that in this case we opted for a step forward rather than coming back to square one, which in reality meant that a project for the consolidation of unified communications foreseen for roll-out over a period of months was in fact rolled out over a few weeks, at least in terms of its basic services.

A final reflection

The expression used in the title of this account is of course not mine. It is an adaptation of the most famous sentence from the first film by Spanish director Alejandro Amenábar, “Tesis”. Nevertheless, if, instead of writing an article about this subject, I had been asked to summarise in a single sentence everything that happened, and especially the lessons learned from the experience, I would not hesitate to choose the same sentence: “Hello, my name’s Andrés and I’m about to undergo a cyberattack.”

Until you find yourself playing the role of the character in this film, you never fully realise the situation of vulnerability in which we find ourselves

To repeat that we do not devote sufficient effort to cybersecurity is clearly necessary. But it isn’t enough. Until you find yourself playing the role of the character in this film, you never fully realise the situation of vulnerability in which we find ourselves. This is true not just for public administrative bodies, but for all organisations in general. During the weeks that passed after that 18 April, I did not hesitate to describe the situation undergone by our university as a “terrorist attack”. While it’s true that it was an attack without victims, it caused damage to the infrastructure that were as vital for the organisation concerned as the brick and mortar of its buildings. Nevertheless, the response that I have observed is not commensurate with that foreseen against attacks on physical infrastructures. It is clear that such a response cannot often be provided due to a lack of sufficient resources, but it is also obvious that such resources will never be available to independent institutions and companies. We need a process of serious global reflection, at a national and even at a European level. The threat that we face has access to resources that are unimaginable for institutions like ours, and even for institutions that are much larger than ours.

We are wrong to develop a cybersecurity strategy that is based exclusively on prevention. Prevention is absolutely essential, of course, but I firmly believe that at this time the first criterion we need to apply is for each of us to admit the idea that we are going to become the target of a cyberattack.

About the author

Andrés Prado has a degree in Telecommunications Engineering from the University of Malaga and since 2008 has been the Director of Information and Communication Technology at the University of Castilla-La Mancha (UCLM), where he is also an Associate Professor for the Masters course in Computer Engineering. He forms part of the editorial team of the journal “RUIDERAe” and has published various articles related to the use of ICT in higher education. He has participated in a large number of events in the technological field and is a habitual speaker at conferences held by the ICT sector of the Conference of Rectors of Spanish Universities (CRUE), for which he has become the director of the “Master Plan and Strategic Actions” work group.

(*) This article was originally published in the journal BIT, produced by the Spanish Association of Telecommunications Engineers (COIT)

Also this year GÉANT joins the European Cyber Security Month, with the campaign 'A Community of Cyber Heroes'. Read articles from cyber security experts within our community and download resources from our awareness package on connect.geant.org/csm2022