The Internet, as anyone who works deep in its trenches will tell you, is not a smooth, well-oiled machine.
It’s a messy patchwork assembled over decades and held together by the digital equivalent of Scotch tape and chewing gum. Much of it is based on open source software that is grudgingly maintained by a small army of volunteer developers who fix bugs, patch holes, and ensure that the whole outbreak, which is responsible for trillions of dollars in global GDP, keeps running.
Last week, one of these developers may have saved the Internet from massive trouble.
His name is Andres Freund. He is a 38-year-old software engineer who lives in San Francisco and works for Microsoft. His job involves developing an open source database software known as PostgreSQL, the details of which would probably bore you to tears if I could explain them properly, which I can’t.
Recently, while doing some routine maintenance, Mr. Freund accidentally found a backdoor hidden in a piece of software that is part of the Linux operating system. The backdoor was a potential prelude to a major cyberattack that experts say could have caused massive damage if it had succeeded.
Now, in a twist befitting Hollywood, tech leaders and cybersecurity researchers are hailing Mr. Freund as a hero. Satya Nadella, CEO of Microsoft, excelled his “curiosity and skill”. Fan call him “the silver gorilla of nerds.” Engineers released an old, famous webcomic for developers about how all modern digital infrastructure is based on a project maintained by some random guy in Nebraska. (As they say, Mr. Freund is the random guy from Nebraska.)
In an interview this week, Mr. Freund — who is actually a soft-spoken, German-born coder who declined to be photographed for this story — said becoming an Internet folk hero was disorienting.
“I find it very strange,” he said. “I’m a pretty private person who sits in front of the computer and hacks code.”
The saga began earlier this year when Mr Freund was returning from a visit to his parents in Germany. While looking at an automated test log, he noticed some error messages that he didn’t recognize. He was jet-lagged and the messages didn’t seem urgent, so he committed them to his memory.
But a few weeks later, while running some more tests at home, he noticed that an application called SSH, which is used to connect remotely to computers, was using more processing power than normal. He traced the problem to a set of data compression tools called xz Utils and wondered if it was related to the previous bugs he’d seen.
(Don’t worry if these names are Greek to you. All you really need to know is that these are all small pieces of the Linux operating system, which is probably the most important piece of open source software in the world. The huge Most servers of (including those used by banks, hospitals, governments, and Fortune 500 companies — run on Linux, making its security a matter of global importance.)
Like other popular open source software, Linux is constantly being updated and most bugs are the result of innocent mistakes. But when Mr. Freund looked closely at the xz Utils source code, he saw signs that it had been deliberately hacked.
Specifically, it found that someone had placed malicious code in the latest versions of xz Utils. The code, known as a backdoor, would allow its creator to compromise a user’s SSH connection and secretly run their own code on the user’s computer.
In the world of cybersecurity, a database engineer who inadvertently finds a backdoor in a key Linux function is a bit like a bakery worker who smells a freshly baked loaf of bread, senses that something is off, and correctly concludes that someone has compromised the entire world’s supply of yeast. It’s the kind of intuition that requires years of experience and obsessive attention to detail, plus a healthy dose of luck.
At first, Mr. Freund was skeptical of his findings. Had he really discovered a backdoor in one of the most heavily vetted open source programs in the world?
“It felt surreal,” he said. “There were times when I was, I must have just had a bad night and had some fever dreams.”
But his dig kept turning up new evidence, and last week, Mr. Freund sent his findings to a group of open-source software developers. The news set the tech world on fire. Within hours, some researchers credit him with thwarting a potentially historic cyberattack.
“This could have been the most widespread and effective backdoor ever planted in any software product,” said Alex Stamos, the chief security officer of SentinelOne, a cyber security research firm.
If undetected, Mr. Stamos said, the backdoor would “give its creators a master key to any of the hundreds of millions of computers around the world running SSH.” That key could have allowed them to steal private information, plant malware, or cause major infrastructure disruptions — all without getting caught.
(The New York Times sued Microsoft and its partner OpenAI over copyright infringement claims involving artificial intelligence systems that generate text.)
No one knows who planted the backdoor. But the plot appears to have been so complex that some researchers believe only a nation with formidable hacking balls, such as Russia or China, could have attempted it.
According to some researchers who went back and reviewed the evidence, the attacker appears to have used an alias, “Jia Tan,” to propose changes to xz Utils as early as 2022. (Many open source software projects are governed through hierarchy; developers propose changes into a program’s code, and then more experienced programmers known as “maintainers” must review and approve the changes.)
The attacker, using the name Jia Tan, appears to have spent several years slowly gaining the trust of other xz Utils developers and gaining more control over the project, eventually becoming a maintainer and finally injecting the code with the backdoor earlier this year. (The new, hacked version of the code had been released, but was not yet in widespread use.)
Mr. Freund declined to speculate on who might have been behind the attack. But he said whoever it was was sophisticated enough to try to cover their tracks, including by adding code that made the backdoor harder to detect.
“It was very mysterious,” he said. “They clearly spent a lot of effort trying to hide what they were doing.”
Since his findings became known, Mr. Freund said, he has been helping teams trying to reverse the attack and identify the culprit. But he was too busy to rest on his laurels. The next version of PostgreSQL, the database software he works on, will be released later this year, and he’s trying to make some last-minute changes before the deadline.
“I don’t really have time to go and have a celebratory drink,” he said.
