On Friday 24th May, iNetwork hosted the “St Helen’s Cyber Attack: the impact & lessons learned” event. The session was hosted by iNetwork Director Shelley Heckman and was held in-person at the Kings House Conference Centre, Manchester.
Invited speakers from key areas of the council were asked to share and offer their perspectives of the attack. This included the Chief Executive, Kath O’Dwyer, Assistant Director of People Management, ICT & Digital, Ste Sharples, Head of ICT & Digital Delivery, Mark Byrom, and Director Policy & Transformation, Vicky Willet.
The attack occurred in August 2023 and this led to the ex-filtration of local authority data, delegates were given the opportunity on the day to hear a firsthand account of what happened, how the organisation was impacted, what they did to contain the attack, how they responded, and recovered from it.
The session began with an introduction from the St Helens Chief Executive with a recorded message from Kath O’Dwyer. This provided a brief overview of the attack and Kath shared how the procedures that they already had in place and the support of staff enabled them to effectively deal with the attack and bring services back online quickly. It took St Helens Council 10 weeks to recover completely.
Following on from the Introduction, the audience heard from Ste Sharples who discussed how to have strategic readiness in preparation for a cyber attack. Ste shared the framework that St Helens are implementing to follow for 2024 – 2027 which involves three interlinked strategies working together:
– ICT Strategy and Roadmap
– Data Strategy
– Digital St Helens
To ensure a successful implementation of these strategies, St Helens identified four key areas that would require focus, the first being cloud transformation (adoption and migration). Ste explained that when the cyber attack occurred, they were fortunate to have 66% of their data based in the cloud using SaaS and laaS as cloud platforms. During the attack none of their cloud services were impacted, they were just disconnected meaning the data stored there was retrievable. Going forward with the new framework, they have now increased their amount of data stored within the cloud to over 70% and they are upskilling their own staff in cloud technologies.
Ste then went on to discuss the second key area: data management. This was to align their new data strategy to data in the cloud model. Ste emphasised that it is important to understand what basic cloud backup is and that they have now implemented a proper hierarchical storage model. It was also explained that you should understand what different application systems offer, for example Microsoft 365 has staging beyond a 30 day retention. Whilst going through the transition, St Helens also reintroduced “air gap” backups with the idea being that this strategic approach would mean that their data could be recovered in a variety of ways.
The third key area that Ste spoke about is device and desktop software together with unified communications. The devices used by St Helens are substantively MS surface with Cisco AnyWhere Connect. Ste believes these are robust and reliable and they are also able to be used from any location. Similarly, St Helens telephony is now also available to them wherever they choose using platform 8×8. Staff also have Microsoft 365 that may be accessed through their own devices, if required, and having all of this in place enabled the St Helens team to still be able to access systems and data during the cyber attack.
Ste then went on to the fourth and final key area of network and buildings. The council has been strategically reducing their building estate and introducing in-office agile hubs where desks and rooms can be booked by staff as required. With the other procedures in place, the majority of staff are therefore able to work from home or a place of their choosing. Ste explained that after the attack there was no requirement for all staff to return to working in the office, most just needed to simply visit to bring their equipment for a password reset before having the freedom to work remotely again.
Following on from Ste, delegates heard from Mark Byrom who provided an in-depth account and timeline of the cyber attack. In the first instance Mark said that you had to be aware of attack detection and after a forensic investigation, it was confirmed that the attack began on 17th August 2023. The earliest indicator of a compromise alert was raised at St Helens on 18th August and involved a Microsoft Defender “risky logon alert” from domain account admin. After the weekend on Monday 21st August, St Helens were then notified by Internet Service Provider, JISC, that over 29 gigabytes of data had been uploaded to a cloud storage provider in New Zealand named Mega.io. At 9:20am on the same day, an internal assessment began and the council’s incident response plan was invoked with an emergency response team being set up. By 9:40 am on 21st August they also contacted the council’s third-party Cyber Security Incident Response Team (CSIRT).
After covering the stages of the initial detection, Mark then discussed the next stages that took place as they knew the incident was occurring. Via the police, St Helens immediately put a block and preservation order on the data at Mega.io to make this safe and by 11:20 am they had blocked the IP addresses identified from the data exfiltration CSIRT onsite. St Helens assembled an Executive Strategic Command Group (SCG) and Incident Response Team (IRT) who identified an initial set of domain accounts and servers with indicators of compromise before disabling the accounts and resetting passwords together with disconnecting identified servers from their network. They then moved on to have early communication with partner organisations and ensured that the National Cyber Security Centre (NCSC) were engaged. Mark informed that by 6:28 pm on 21st August there was no further evidence of attacks that occurred. All of these actions covered by Mark had taken place within the same day showing the swift response from the council.
Mark then went on to share the response. First the initial response was looked at which involved a coordinated information sharing and prioritised action allocation taking place between Communicate (the cyber security company) and St Helens. There were also initially twice daily meetings between the SCG and the Executive Board of St Helens followed by ongoing regular meetings with both SCG and IRT. A site visit was also made by the Merseyside Cyber Team to view the situation. Following all of this Mark covered the initial decisions that were made for responses which included disconnecting all 8 file servers. This caused significant disruption for the Council but the decision was made that this would be the best response to the incident. This also led to the coordination of password resets for all domain accounts. During this time the ICO were also informed of the incident and they decided no further action was required from them.
Once the initial response had been discussed, Mark went on to explain the response in the days and weeks that followed including round the clock forensic investigation and support. One key aspect of this was also how the incident was communicated, both to the media and partner organisations with updates being provided to these. Despite previous procedures being in place, Mark did advise that there was still some innovation on key systems required from their DevOps team to ensure that staff and partners were still paid during this time and to ensure that key areas still had support (for example social care systems). By the end of week 3 the root cause of the incident was identified.
The final part of the timeline Mark mentioned was the recovery stage. The infrastructure component recovery started with the council in week 1 with a priority for risk to life and financial system and within 10 weeks all systems had been recovered. Mark also explained the importance of an accurate incident status report being documented throughout the incident.
Mark then summarised his presentation with an overview of the findings gathered and the cyber improvements they are making following this. Although the incident was contained, the threat actor behind this was never identified. It should also be noted that this was not a ransomware incident as Mark confirmed that no requests for ransom were received by the council. The vulnerability for the attack was a Citrix NetScaler and it is understood that this incident was potentially a precursor to a much wider attack, meaning the effective response shown was of high importance. The cost of the attack was £120,000 on the cyber incident and response investigation plus £113,000 following this on enhanced cyber hardening. These figures also do not take into account further financial costs to the business that are difficult to quantify.
After a short break, Ste Sharples returned to present on what lessons have been learned from the attack. Up until now the presentations had discussed a mostly technical approach to the incident but here Ste used this opportunity to also discuss the physical and emotional impact on the staff involved. He expressed that fatigue is a big part to consider and although it is tempting to work nonstop to resolve it, it is of vital importance to keep a cool head. He also praised the Chief Executive Kath O’Dwyer for leading with the correct mentality, no blame was placed on anybody, she understood this was a thing that could happen to any organisation and she helped to ensure there was the correct environment for the effective response to take place. Ste also emphasised that whilst having numerous support offers from external parties was nice, it is important to still have consistency in your approach and this should be remembered before accepting any help. From a technical point, Ste also summarised that it is crucial to ensure that alerting is enabled for the security tooling that you have and again advised that they believe that cloud adoption is the key enabler to minimising the threat.
The day’s event was then rounded off with questions to the panel section facilitated by Shelley Heckman with Ste Sharples and Mark Byrom now joined by Vicky Willett. Vicky is the Director for Policy and Transformation and was able to offer a different perspective on the event. She actually joined St Helens on 4th September 2023, 2 weeks after the incident had taken place, and she worked on delivering the council’s corporate strategy delivery. She did not have technical background however soon got up to speed and her main task was to ensure everyone knew their roles. Vicky was also the lead officer for GDPR so needed to ensure all relevant parties kept up to date on this. From a communications perspective, she was about communicating the actual facts they had to staff and key members.
This event provided key insights for delegates into the impact and response to an organisational wide cyber incident and the audience were engaged throughout this section with numerous interesting questions being asked. Ste and Mark provided answers to questions of a technical nature, and Vicky responding to questions more on the organisation as a whole or on a staff level.
At iNetwork, we host a number of events tailored to the public sector, where members can learn and share knowledge and experiences within a safe space. We encourage members to visit our website, sign up to our newsletters and attend more of iNetwork’s future events.