You may have heard that there was a very critical Linux kernel vulnerability making the rounds. As with all important enough vulnerabilities, this one has a catchy name: Dirty Pipe (no logo, though). This blogpost attempts to explain how that vulnerability impacted Replit. The good news is that as far as we know, there weren't any successful exploitations of it!
That article linked above has the full explanation and is definitely worth the read because it narrates the journey from discovery to fix. In case you're in a hurry, the short description of that vulnerability is that it allowed any user to temporarily overwrite any file in the filesystem, without requiring any write permissions to do so. Temporarily because it didn't actually change the file, just the in-memory page cache, so if the kernel was under any sort of memory pressure, those changes would go away. There were a few more restrictions (mostly about the position, alignment, and length of the write), but other than that this allowed the attacker to make all sorts of very scary modifications to the system. Notably, the proof-of-concept code allowed any user to open a root shell by overwriting a setuid binary that had privileges to "become" root by the mere act of invoking it.
The moment our "security advocate" (in reality it's just one of our platform engineers in a funny disguise until we hire a full-time security engineer) realized that this was such a serious bug, we immediately tried the proof-of-concept code. And we were delighted that it didn't work! We very recently enabled the no new privs bit that negated the effects of the setuid bit, so the user was greeted with a normal shell instead of a root shell. This meant that the scariest part of this exploit (escalation of privileges) was not possible in our system. Furthermore, the container has a very limited set of capabilities, which meant that even if the root shell would have indeed been possible, the attacker would not have been able to make most changes to the system. Hooray for defense in depth!
Our initial happiness quickly dissipated, though. Even if the proof-of-concept didn't quite work all the way, it still had an effect: the files were still rewritten. So what's the worse that an attacker could do with that newly found power? Since we use Linux containers (through Docker), that means that the files in the root filesystem are shared in read-only fashion among all the containers in a system. So what if we tried to overwrite an important binary that everybody used (say, /bin/sh)? Turns out that the page cache is shared among containers too, so the modifications were visible to all repls in that one machine! This means that if a malicious user wanted, they could have been able to surreptitiously make changes to the shell, which means that they could make any modifications to any repl that happened to be running in that same machine. Exfiltration of secrets, modification of files, anything. So we needed to patch this ASAP.
Fortunately the kernel already had a patch available, so all we needed to do was to make a deployment and wait a bit. We got very lucky here, because this could be a very long battle to get mitigations in place, but the disclosure of this was well-coordinated. We were very happy that this moment was mostly anti-climactic.
By the way, if you tried to open any C# repl between 2022-03-09 and 2022-03-11, you might have seen a warning about a kernel bug preventing those repls from running. It turns out that it's a different, unrelated issue. Two different kernel bugs in the same week? What are the odds!? But that's a story for another day.
Loved to read about these shenanigans? Interested to learn more? Want to be our full-time security engineer and help protect the next billion software creators? Consider applying, because we are hiring for all engineering roles!
