By John Lorinc and Sakeina Syed
Although TPL officials maintained an official no-comment stance for much of the shutdown — TPL chief librarian Vickery Bowles told The Globe and Mail‘s Oliver Moore in late December that she’d been advised “not to talk about the threat actor” — the organization undertook a public relations effort beginning later in January to burnish its image.
In social media plugs from politicians, upbeat media coverage about its re-opening efforts and regular email blasts, TPL officials over the past month or so have sought to get the message out to Torontonians that their beloved and well-used library system was back in action. (The TPL contends that it has not used the services of a crisis communications consultant.)
What’s far less clear — and certainly not part of the TPL’s dialogue with its patrons — is the ultimate cost of this incident, a candid accounting of the reasons for such an unusually lengthy shutdown, and a clear explanation of how ready the organization is for future attacks.
By contrast, the British Library, which was attacked on the same day as the TPL, earlier this month published a lengthy and detailed paper entitled, Learning Lessons from the Cyber-Attack. This post-mortem (see sidebar below) provides a far more candid and specific account of what transpired, the causes and the recovery process than anything released to date by the TPL.
There are perfectly good reasons for the TPL and the City to be tight-lipped with technical information that could be exploited by cybercrime gangs. Still, these are publicly accountable institutions. The public surely has a right to know whether its municipal agencies have taken adequate measures to protect themselves, especially given some evidence that earlier warnings — e.g., from the Toronto Auditor-General as far back as 2021 — did not appear to produce a sense of urgency among TPL officials and its consultants. (TPL officials in a February, 2024, report to their board claimed otherwise: “TPL has proactively prepared for cybersecurity issues by prioritizing cybersecurity since January 2021.”)
During the 2024 budget deliberations, TPL revealed the City would be picking up the cost of dealing with the ransomware attack, i.e., so it wouldn’t come out of the TPL’s own funds. Spacing asked both the TPL and the City how much was spent — a number that precisely reveals the degree of risk associated with failing to adopt sound cybersecurity practices — but neither organization was prepared to disclose a figure. “This incident is ongoing and final costs are not currently known to the City,” a spokesperson said in an email. “Any costs to the City will be attributed to Finance and Treasury Services.”
Another question-mark is the duration of the downtime. Most organizations that are attacked by a ransomware gang recover within a week or a few weeks, although some do take longer, among them Indigo and the British Library, which is still, as of mid-March, working to return to full service. Every case is different because of the structure of the network, the state of existing software and the hardware, and so on.
TPL officials told the board in February that the library’s IT team and external consultants “used this attack as an opportunity to accelerate [the TPL’s previously scheduled cybersecurity improvement] plans.” Whether these efforts to dial up the implementation of newer and more robust systems resulted in a longer downtime isn’t entirely clear. “The delay was not a result of the acceleration,” says Critchley. “Rather, the work we had to do to secure and protect the network resulted in an acceleration and implementation of this previously planned work.”
As for the state of the now re-started network, and all the various sub-systems meant to protect the TPL from cybercrimes, the library says it has succeeded in moving beyond the “Developing” level of maturity — level 2 of 5 possible — that existed as of October 28, 2023.
As for how much progress, the TPL isn’t saying.
“Since the attack in October,” the February report to the TPL board states, “TPL has matured its information security program by implementing additional security controls, updated some hardware and software applications, and introduced new processes… As part of the rebuilding and restoration work, TPL has not simply replicated the former environment but rather used this attack as an opportunity to accelerate plans. Subsequently, TPL’s IT security program maturity level has increased when assessed within the National Institute of Standards and Technology (NIST) Cybersecurity Framework.”
Public institutions are increasingly targeted by cybercrime
Looking to the future, the TPL cyberattack is a harbinger of a “new normal,” says Matti Siemiatycki, a professor of geography and planning and Director of the Infrastructure Institute at the University of Toronto. “You have these big public institutions that are important to our cities — that are also increasingly vulnerable, the target of attacks,” he says. “You see the cascading impacts it has, and the challenges that can come from trying to get these systems back online.”
Siemiatycki notes the cyberattack on the City of Hamilton in recent months as a similar example of how municipal services and public institutions are increasingly targeted by cybercrime. In Hamilton, the breach in early February cut off city phone lines, obstructed the overtime pay system for city employees, and compromised many digital services. Closer to home, the Toronto Zoo was similarly compromised by a cyberattack that surrendered staff members’ personal information to the perpetrators.
Pointing to the agencies, boards, and commissions which govern city services, Siemiatycki says each will have “different systems and different degrees of readiness..I think it’s important — now that cities across North America and around the world are becoming targets — that the city plays a central role in making sure that all of the organizations under its management have the tools, techniques, and capacity both to protect themselves and then respond when incidents inevitably do happen.”
As for the question of whether TPL and the City’s public-facing response to the cyberattack would have been more effective with greater transparency, Siemiatycki says that there might be technical benefits behind the scenes to either approach. However, he says clarity from the City regarding the scale of the cyber threats that municipalities are up against can help “speak to the scale of investment” that will be needed, allowing for greater public awareness and engagement.
“If we’re going to be able to keep having these services that we’ve come to rely on,” he says, “I think we as a community need to understand the scale of the threat and how often this is happening, what’s being targeted and what’s at stake.”
Charles Finlay, who heads TMU’s cybersecurity research centre, agrees: “Everybody acknowledges the problem is extremely serious and growing, and resources need to be increased on an urgent basis in order to to try to mitigate these threats.”
He continues: “My perspective is, you either pay something now in terms of investing in law enforcement, people, processes and technology in order to prepare for these attacks and to mitigate these attacks when they happen, or you pay a lot more later. And you don’t just necessarily pay in money later. You pay in the trust that citizens have in their governments and in their communities.”
“I think this is a learning lesson for everybody,” says Scarborough councillor Paul Ainslie, who sits on the TPL board and chairs council’s general government committee. The library got hacked. The zoo got hacked. It was an eye opener for a number of organizations across the city. `Hey, we need to step up our cybersecurity, too.'”
Siemiatycki hopes there’s a silver lining. The months-long incapacitation of the public library served to drive home the far-reaching, singular value of the city service — an awareness that has potential to be transformative. “In many ways,” he says, “it’s a living room for the city.”
Learning from the British Library
In many ways, the Toronto Public Library and the British Library are very different institutions. One is local; the other, national. One is primarily a lending library with many branches; the other is a repository and research library situated on a single campus.
Yet the two institutions have comparable annual budgets — $234 million for the TPL in 2023 and $219 million for the BL in 2021/22 — and a similar number of employees (1,808 FTE and 1,702, respectively). They were also both hit by ransomware attacks on October 28, 2023.
In late February, the TPL released a 12-page final report on the attack, which included a summary of the library’s response, the impact on services, a section on “lessons learned” and a confidential attachment from legal counsel.
The British Library’s own “incident review,” an 18 page document entitled Learning Lessons from the Cyber-Attack, offers a substantially more detailed and candid self-assessment compared to the TPL’s report. This document — which should be compulsory reading for policy-makers who were involved in the TPL attack or other City of Toronto cybersecurity departments — explores “why our security measures were not sufficient” and concedes frankly that “the impact on the library’s systems and services has…been deep and extensive.”
Posted to the BL website on March 8, the incident review contains six sections — a deep dive into the incident itself, a look at the impact, a summary of the crisis response, a description of the technology infrastructure, an assessment of future risk, and 18 lessons learned.
The section outlining the incident includes highly detailed timelines, the name of the attacker, the size and description of the stolen files, the suspected point of entry, an admission that the lack of a system-wide multi-factor authentication likely contributed to the incident and an acknowledgment that the self-cleaning features of the ransomware virus, i.e., its ability to erase traces of itself, inflicted the greatest damage because it destroyed servers that could assist with recovery. None of this kind of detailed analysis was included in the TPL’s final report on the ransomware attack, or in any other documents it published after October 28.
Among the many learnings highlighted in the BL’s incident review, one directly addresses the vulnerabilities created by a “historically complex” IT network that consisted of sub-systems and hardware of varying ages and resilience. “There is a clear lesson in ensuring the attack vector is reduced as much as possible by keeping infrastructure and applications current, with increased levels of lifecycle investment in technology infrastructure and security.”
The TPL says it has no plans to publish a comparably detailed incident review.
– sidebar by John Lorinc
Part I: Toronto Public Library ransomware attack: Overview
Part II: Toronto Public Library ransomware attack: Unanswered Questions
Part III: Toronto Public Library ransomware attack: Was TPL adequately prepared to defend itself?
Part IV: Toronto Public Library ransomware attack: Where does the TPL go from here?
Part V: Q+A with Toronto’s chief librarian, Vickery Bowles
