Tue, Jul 12, 2022Leaky UIs
Building a robust and powerful UI without compromising on simplicity is complicated, and we're always exploring solutions that alleviate that problem. A few years ago, we rewrote our workspace architecture, and Amjad gave a talk about it. We're about to add a lot of firepower to our layout system (stay tuned!), but we noticed significant shortcomings in our abstractions. This blog post will outline the flaws and how the underlying data structure hurt the interactions. To keep this post concise, I will defer the reasoning behind the UI and UX to a future post. First, let's lay down some fundamentals. We want a tiled layout system where a group of panes (a.k.a tiles) are laid on the screen next to each other, vertically and/or horizontally, without overlapping. Each pane should have another pane adjacent to it or be at the edge of the screen. The screen is finite, and pane sizes vary. People should be able to resize their panes.
Fri, Jul 8, 2022Worldwide Repls, part 1: The Control Plane
During the ReplCon 2022 keynote, we announced that we were going to geo-distribute our infrastructure so that your Repls are much faster when accessed outside of the United States. The speed of electrons / light in a medium is a fundamental speed limit. Most of our users are several thousand kilometers away from the data centers where we host Repls (currently limited to the United States), so the round trip from them to the users' homes is going to necessarily take several hundred milliseconds just to traverse the series of tubes that is the Internet. Today, we have finished the first milestone to make this a reality, and that comes with some pretty neat side-effects! The control / data plane separation Up to this point, our infrastructure was designed in a way that allows for horizontal scalability: being able to have more capacity to run Repls by adding more VMs to our infrastructure. This allows us to quickly react to having more users at certain times of day without making any code changes, and is built upon having a Load Balancer that routes requests to connect to a container using the WebSockets protocol to individual VMs in our infrastructure. Some of you might remember that we had announced global routing last year, which we achieved by having a small set of servers in Mumbai, India. That worked well during testing, but after enabling it for everybody we discovered an interesting edge case: since the Load Balancer is the component that decides where any WebSockets connection goes and we have little control over that whole process, sometimes the India servers were a bit busier than servers elsewhere and the Load Balancer decided to route a connection to the United States. But what if this was a multiplayer Repl, or a reconnect to a Repl that was already running? In those cases, the VM where the request lands performs a transparent proxy of the WebSockets connection to the VM where the Repl is actually running. So now users would have to connect to a server in the United States (which was the old "normal"), but then that server had to talk to a server in India, which added an unnecessary ~250ms worth of roundtrip to the previous behavior. We had accidentally made things slower! Sadly, we had to revert this change 😢. So back to the drawing board we go. What became obvious is that we had to make changes such that the Load Balancer was not the source of truth of where Repls are placed. Enter the Control Plane. In Networking, there is a distinction between what are called Control and Data Planes: where the Control Plane is concerned with making changes to the network to make sure that packets go to their destination in the best way possible at that moment, and the Data Plane is the part of the system that actually moves the internet packets (a.k.a. the data). There is also the Management Plane, which is analogous to the Control Plane but manages other non-networking systems. So for us, we wanted to have a Control / Data Plane separation to clearly distinguish making modifications to the shape of the system: making a Repl start / stop in a particular VM vs. the WebSockets communication between the Linux container and the Workspace. A new abstraction to build upon
Mon, Jul 4, 2022Improving Domain Linking for Repls
For some time now, it's been possible to link your custom domain to your Repl. You buy that perfect domain name, connect it to your website hosted on Replit, and you're good to go! Custom domains help you create a unique home on the internet - for your blog, a product you're launching, or just a space to try new ideas. We are rolling out domain linking improvements - direct apex domain linking, magic domain linking, and multiple domain support. We also updated our domain verification logic! You can experience these right now if you're an explorer. Otherwise, these will make it to you soon! Domain Verification Logic Previously, we would verify your domain by attempting to access a special Replit proxy-enabled route on the domain. As the number of domains grew, our servers' time spent verifying domains increased. We wanted to make this process more efficient. We updated the logic to verify DNS (the internet's address book) records - speedy! - before further checks. Verification happens on a slightly different cadence, in batches now. In addition, every evening, we reverify domains. This daily re-verification process checks to see if we can still access your repl through your domain. If we encounter a problem, your domain linking status panel will update to show the domain is no longer "verified." Where possible, we'll show you the records that we see to make troubleshooting easier. Because we are now verifying DNS records, and services like Cloudflare's "orange-cloud" modify DNS, we added TXT record verification to save you from having to turn off Cloudflare (or other systems) protection to successfully link.
Sun, Apr 17, 2022QLTY SZN 1
A commitment to quality One of our competitive advantages is making bets on technologies before the rest of the market catches on. One such bet we've made is on CodeMirror. We switched our editor from a proven but stagnant technology, Monaco, and towards CodeMirror. We know we made the right decision because: We can now make changes, most importantly bug fixes, much more quickly. The new search panel we were able to build is much better than the version that shipped with Monaco. The new editor's improved performance is already showing up in better user onboarding success. CodeMirror's extensibility will enable us to ship one of the most most long-awaited features: Themes!
Wed, Mar 30, 2022We Built a Search Engine
For the past few months, we have been building a Replit-native search engine. It is remarkably powerful, and we are really excited for you all to try it out. We believe that you should be able to find anything on Replit in less than 30 seconds. This might sound simple, but when you have 100 million+ Repls, it becomes complicated. :) When you search for something on Replit today, you'll see a page with relevant results from the following categories: Repls Templates Code (yes, code)
Wed, Mar 16, 2022Escaping Dirty Pipe (a.k.a. CVE-2022-0847), mostly unscathed
You may have heard that there was a very critical Linux kernel vulnerability making the rounds. As with all important enough vulnerabilities, this one has a catchy name: Dirty Pipe (no logo, though). This blogpost attempts to explain how that vulnerability impacted Replit. The good news is that as far as we know, there weren't any successful exploitations of it! That article linked above has the full explanation and is definitely worth the read because it narrates the journey from discovery to fix. In case you're in a hurry, the short description of that vulnerability is that it allowed any user to temporarily overwrite any file in the filesystem, without requiring any write permissions to do so. Temporarily because it didn't actually change the file, just the in-memory page cache, so if the kernel was under any sort of memory pressure, those changes would go away. There were a few more restrictions (mostly about the position, alignment, and length of the write), but other than that this allowed the attacker to make all sorts of very scary modifications to the system. Notably, the proof-of-concept code allowed any user to open a root shell by overwriting a setuid binary that had privileges to "become" root by the mere act of invoking it. The moment our "security advocate" (in reality it's just one of our platform engineers in a funny disguise until we hire a full-time security engineer) realized that this was such a serious bug, we immediately tried the proof-of-concept code. And we were delighted that it didn't work! We very recently enabled the no new privs bit that negated the effects of the setuid bit, so the user was greeted with a normal shell instead of a root shell. This meant that the scariest part of this exploit (escalation of privileges) was not possible in our system. Furthermore, the container has a very limited set of capabilities, which meant that even if the root shell would have indeed been possible, the attacker would not have been able to make most changes to the system. Hooray for defense in depth! Our initial happiness quickly dissipated, though. Even if the proof-of-concept didn't quite work all the way, it still had an effect: the files were still rewritten. So what's the worse that an attacker could do with that newly found power? Since we use Linux containers (through Docker), that means that the files in the root filesystem are shared in read-only fashion among all the containers in a system. So what if we tried to overwrite an important binary that everybody used (say, /bin/sh)? Turns out that the page cache is shared among containers too, so the modifications were visible to all repls in that one machine! This means that if a malicious user wanted, they could have been able to surreptitiously make changes to the shell, which means that they could make any modifications to any repl that happened to be running in that same machine. Exfiltration of secrets, modification of files, anything. So we needed to patch this ASAP. Fortunately the kernel already had a patch available, so all we needed to do was to make a deployment and wait a bit. We got very lucky here, because this could be a very long battle to get mitigations in place, but the disclosure of this was well-coordinated. We were very happy that this moment was mostly anti-climactic. By the way, if you tried to open any C# repl between 2022-03-09 and 2022-03-11, you might have seen a warning about a kernel bug preventing those repls from running. It turns out that it's a different, unrelated issue. Two different kernel bugs in the same week? What are the odds!? But that's a story for another day.
Wed, Mar 9, 2022Betting on CodeMirror
At Replit, our mission is to bring the next billion software creators online. In order to achieve that ambitious goal, we need to make sure that the experience of writing, running, and sharing code is as seamless as possible. At the heart of that experience is the editor itself. As a result, we dedicate a huge amount of time and energy to improving the stability, performance, and accessibility of our editor. Monaco For years, we had been relying on Monaco to power that experience. Monaco is the open source editor behind VS Code, written and maintained by Microsoft. At the time, the decision to adopt Monaco was an easy one. It was packed full of useful features that helped users navigate and write code quicker, had built in support for a large number of popular languages, and looked familiar to those coming from VS Code and other popular IDEs. These reasons, along with its growing open source community, quickly made Monaco the de-facto way to write code on the web.
Wed, Feb 23, 2022All New Repls are Powered By Nix
For the past year we have been working hard to integrate Nix into our platform and rebuild our existing language experiences with package from Nix. We are excited to announce that all new repls are now powered by Nix. In this post we'll talk about what this means for the future of the platform and all the work that it took to get here. With Nix you can easily create templates for languages Replit didn't support before, pull in more development tools than ever, and access any cli application in the shell without installing anything: Every new repl being powered by Nix means that we've be able to make vast improvements to our existing languages: Long awaited Node.js v16 support 100x faster Python package management
Fri, Feb 11, 2022Announcing File Persistence in Hosted Apps… for Everyone!
Replit is your computer — for whatever you need to do. We announced last year that Hackers would be able to have their hosted apps be able to persist file changes since that made it possible to build a lot more apps. But we were not quite satisfied that only Hackers were going to get this new feature. Today we are announcing that we are opening this up for everyone! Why the change? Back in November, we knew that this new feature was going to have a small, but measurable, impact to our infrastructure. This meant that opening this up for everyone from the get-go was not ideal. We wanted to get this out to users as soon as possible, so we decided to be iterative. We crafted a plan to open this up for Hackers first to observe what a realistic load would look like, while pondering some potential optimizations that we could use to improve it. In addition, we added more monitoring just in case anything came up. After enabling this and blogging about it, we saw a very small difference between our predictions and reality. The plan had worked! And our hunch about something unexpected appearing was right on the money: the extra monitoring enabled us to find (and fix) an extremely low-probability bug that could only be observed when operating at Replit scale. With the new data, we could make data-driven decisions. We went through the planned optimizations and the additional load is now in a much better place, so we're comfortable opening this up for everyone.
Fri, Feb 4, 2022Making new Python repls 100x faster to start up
Python is currently the world's (and Replit's) most popular programming language. We've improved the Python experience during last year, with a Python package cache to make installs faster, and an integrated, multiplayer debugger to increase the understanding of what programs do. But there are still a few problems with Python. Packages are often very space-consuming, so they are installed into an ephemeral 2 GiB scratch disk to avoid filling up repl directories. Unfortunately this means that every time a Python repl starts, a lengthy package installation process must happen. This makes some Python repls take forever to start! Some other packages that have a large number of dependencies (like TensorFlow and Torch) were completely unusable because they don't even fit on the scratch directory. We decided to address these shortcomings and make Python a bit faster on top of that! Today, we're releasing a brand new Python template that has a lot of neat things. Newly created Python repls will now be based on nix so that additional programs and libraries can be installed, have a standard virtual environment stored inside the repl, and a brand new caching mechanism so that packages are installed even faster and they don't take up too much space. I had the wildest dream, that I was able to run the TensorFlow 2 quickstart on replit without running out of disk space or memory How does it work? We had a few goals in mind when we started designing this:
- Mon, Jan 31, 2022
Understanding Repl Resource Utilization
Every computer on earth needs these three essential resources in some form: Processor Memory Storage The computers we provide for Replit users, or Repls, have access to a virtual CPU, an allocation of RAM, and a virtualized filesystem. It’s important to understand resource utilization within the context of the software that you’re writing. To this end, we have started rolling out a new component in the file tree to make this more transparent and visible to all of our users.
Fri, Jan 21, 2022Going Where the Next Billion Creators Are
Mobile is the future of computing. But building good software creation tools for mobile devices is hard. Luckily, at Replit, we like to run toward the hard things. This week, we launched a completely rearchitected mobile IDE for web. If you’ve used it before, the new version won’t look much different, but it will likely feel different. Why We Care Mobile devices are the world’s most ubiquitous computers. At Replit, we have kids coding their next big idea on their phone on the way to school, instead of scrolling through social media. We have local communities in developing countries learning & teaching code together on their phones, in order to get hired for jobs. We have colleagues coding prototypes in multiplayer and demoing their work on their phones, while they are on the go. We even noticed users sharing Replit coding tutorials on TikTok: @coding4python3 Antwoorden aan @wewillcum #replit #code #coding #coding4python3 #python ♬ origineel geluid - coding4python3
Tue, Jan 11, 2022Migrating our Web App from Heroku to GCP
After many years of running on Heroku, Replit has fully migrated to Google Cloud Platform. Why leave Heroku in the first place? Heroku served us well, but ultimately we recognized that our mission to bring the next billion software creators online would require an entirely new approach to our infrastructure. Our infrastructure needs to be flexible, and we need to have control across the entire stack - both data and hosting - to make moves to support our mission. So, what did it take to migrate Replit? Step 1: Demo a Prototype We put together a quick demo of the website running in Google Cloud using a test database (with a fake dataset) to prove the concept would work. In this stage, we also gathered feedback and started to push the boundaries to understand where the problems might lurk later in the project. We took some time to investigate alternate hosting systems and even clouds. Ultimately, because the rest of Replit's infrastructure is in Google Cloud, it made the most sense to stay with what we know.
Wed, Dec 22, 2021Implementing RUI, Replit's Design System
At Replit, we have a small engineering and design team supporting millions of users. Our secret is investing in good tools that make us more productive. In this blog post we'll give you an insider look into how we implemented one such tool - the Replit design system (or RUI for short). The project started as a collection of growth pains: Designers were stretched thin on multiple projects and they couldn't be involved in small tactical decisions UI was inconsistent across the product Reusing UI code was hard, so engineers built new ones (for example, we had 7 different Avatar components) So, we set to build a design system that would help us scale. The high level goals:
Tue, Dec 14, 2021Repl Space and Templates
When we want to support a new technology, we're faced with the question: "Do we (the Replit team) build this capability into the Replit directly, or can our users add it themselves?". Usually, we want the answer to be the latter. Repls are like personal computers for your projects. And the more these personal computers can be customized, the more they'll be used in creative, unexpected ways. Internally, we refer to the extent to which a Repl can be configured as "Repl Space". So when we talk about building new features, we always ask "can it be done in Repl Space?", which is basically asking if we can build it on Replit. Over the past year, Repl Space has become significantly more powerful, and soon, Replit will be one of the most flexible computing platforms on the planet without compromising its simplicity. Now that we have the technical foundations in place, we're excited to bring more of that power into the product in the form of Templates and some new configuration options. Templates Where can I find templates?