In cybersecurity, there are several classifications of attacks. A supply chain attack is defined as an attack on a third-party vendor of a vital item in the chain. One of the more recent supply chain attacks was the SolarWinds attack, where hackers gained access to the SolarWinds network and introduced malicious code into the Orion IT management and monitoring software.
Social engineering is an attack vector that involves manipulating humans into gaining access to or information on vital systems. Most training on social engineering at the user level focuses on protection against phishing emails and phone scams.
The XZ backdoor, however, utilized both, and it could have been catastrophic in its scope.
What Do We Need To Do?!
To verify if your version of XZ Utils or liblzma have been backdoored, running ‘xz –version’ on the command line will show your version on most Linux distributions. On Alpine, running ‘apk list xz’ will show a more detailed version.
If you have a need to verify multiple servers, I have made publicly available an Ansible playbook for checking your version of XZ Utils. Using guidance from JFrog Security, I’ve also written a playbook to make their suggested fixes. You can find these on my GitHub page here: https://github.com/crfearnworks/ansible-CVE-2024-3094
Now, let’s get into the details.
What Is It?
The XZ backdoor targets the open-source library XZ Utils and the underlying library liblzma. This library contains functions for compression and decompression during operations on the Internet, including asymmetrical key usage within OpenSSH. The liblzma library is included in many Linux operating systems from the source distributions, such as Red Hat, Ubuntu, Fedora, Debian, Alpine, and Kali.
Andres Freund, a Microsoft developer, started investigating into memory errors from valgrind while on a Debian system, and found that SSH logins were taking far more CPU cycles than usual.
A malicious script, “build-to-host.m4” was originally added to the .gitignore file of the XZ repo, so it was not checked into version control. The script was added to the release tarballs of XZ Utils, making the source code and the releases different, which is common in projects written in C. The library did contain in source control two files that were buried in test folders: “bad-3-corrupt_lzma2.xz” and “good-large_compressed.lzma”. These two files are decoded in sequence by the “build-to-host.m4” script, which then extracts a shared object, “liblzma_la-crc64-fast.o”. That object is the backdoor that gets swapped in during the compilation of the library.
This backdoor stays dormant until it’s triggered with an SSH certificate that has a payload in the CA signing key N value. The attacker’s Ed448 key must be used to encrypt and sign this payload. Ed448 is a type of Elliptic Curve Cryptography algorithm that is implemented with the Edwards-curve Digital Signature Algorithm. It is similar to the Ed25519 algorithm, but where Ed25519 aims for a 128-bit security level, Ed448 aims for a 224-bit security level. So, we are talking about an algorithm that is used as an extra layer of security against analytical advances.
But wait, this SSH certificate is in the Certificate Authority signing key N value. What does that mean? When a certificate key exchange occurs—such as signing into a server over SSH with a private-public key exchange instead of a password, like industry best practices state—the traffic that can be seen over the connection looks like a normal exchange. This includes when the attacker’s Command and Control server signs into a server with the backdoor.
The xzbot GitHub repo explores how this backdoor could work by allowing users to utilize a known key pair in an isolated system to see how dangerous this is. The YouTube channel Low Level Learning gives a demo of this repo, and shows that this backdoor allows the attacker to gain root access to a server without an account.
A script not checked into source control, referencing two obfuscated binary files hidden in test files, which then extract a shared object, that injects a backdoor that’s triggered with a high-bit security key. The key exchange then allows the attacker root access. And this was released in upstream tarballs to major Linux distributions. That alone is scary enough, because if it had not been caught, it would have been disastrous.
But the technical aspect isn’t the only portion of this that is bothersome.
The Social Aspect
Lasse Collin is the creator of XZ Utils and has been the sole maintainer of the repo. This is an open-source project, and Collin has been doing this out of goodwill. In 2021, a user by the name of Jia Tan creates a GitHub account. One year later, February 6th, 2022, Tan submits their first commit to the XZ repo. In April, Tan started releasing patches to an “XZ_Utils_Unofficial” repo, targeting source code in the liblzma library. Another user by the name of Jigar Kumar offered suggestions to Tan.
Questions about maintenance on the project are sent to Collin, and on June 7th, 2022, Kumar called for a new maintainer of the XZ Utils project, blaming Collin for the lack of progress. Collin stated that mental health issues had limited progress, and stated Tan was working with him on “off-list” items. Collin also said that Tan would have “a bigger role in the future”. A week later, Kumar doubled down on the release of the project a week later, without showing any sympathy.
Over the course of the next two years since Tan submitted their first commit, they worked with Collin on releases in XZ Utils 5.2.x and 5.3.x. Tan announced on January 11th, 2023, that the primary XZ Utils repo was no longer on Collin’s git.tukaani.org site, but on GitHub’s site under “tukaani-project/xz”. This was also the release of XZ Utils 5.4.1. On May 4th, the OpenPGP signing key was changed to Tan’s public key.
Tan announced on January 26th, 2024, that XZ Utils 5.4.6 was released. This version is considered the stable version before the backdoor was fully implemented. This is also the last date that a reference to the tukaani-project GitHub page accompanied the announcement.
On February 15th, Tan added the build_to_host.m4 script to the .gitignore file of the repo.
On February 23rd, Tan added the two test files that contained the obfuscated binary backdoor.
On February 24th, Tan released version 5.6.0 of XZ Utils with the backdoor fully functional.
On February 26th, messages are exchanged between Collin and Sebastian Andrzej Siewior about threads and memory limits.
On March 9th, Tan released version 5.6.1 of XZ Utils, with improvements to the backdoor.
Freund did not discover the backdoor until March 29th.
Collin announced the project breach on https://tukaani.org/xz-backdoor/ on March 30th.
Since then, GitHub accounts have been suspended, repos have been taken down, and numerous posts and videos online have discussed the ramifications of this attack. While the events that contained the malicious code only spanned 23 days, the timeline of Tan’s involvement can be traced over three years.
What’s Next?
During the 20 days that the backdoor was released, several Linux distributions had pulled the backdoored version of XZ Utils. Some were immediately patched, while others were unaffected as they were used in pre-release versions (which were removed) or their versions of OpenSSH did not work with liblzma or system-notify, a backdoor prerequisite. Thanks to the security community, this was quickly rectified before any substantial damage could be done.
While this supply chain attack was thwarted, it was still highly sophisticated, and spanned years in the making. The social engineering of Tan gaining Collin’s trust to become part of the XZ Utils project is how collaboration can happen in the open-source community. Having that kind of support is how the open-source community is as robust as it is, and an attack on that feels blasphemous. The people who contribute to open source do it out of genuine interest, dedication, passion, and for the good of humanity. And with some of the software that we use daily being supported by that community, I feel that is the most important system to protect.
Sources:
https://www.akamai.com/blog/security-research/critical-linux-backdoor-xz-utils-discovered-what-to-know
https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/
https://jfrog.com/blog/xz-backdoor-attack-cve-2024-3094-all-you-need-to-know/
https://www.mail-archive.com/xz-devel@tukaani.org/index.html#00673
https://openwall.com/lists/oss-security/2024/03/29/4