Building the #1 LLM Comparison Tool with Bounties - nat.dev

The Replit Team

Bounties Case Study Cover Image
Bounties Case Study Cover Image
"Replit bounties are a great way to find talented developers for adventures!"



Nat Friedman / former CEO of GitHub

About the Bounty Posters

Nat Friedman is the former CEO of GitHub. He previously co-founded Xamarin, which was acquired by Microsoft in 2016. Before GitHub, Nat held several leadership positions at Microsoft, including Corporate Vice President of Developer Services. Nat is most known for his contributions to open-source projects, most recently nat.dev and the Vesuvius Challenge.

Matt Huang and Alexandr Wang also contributed to this Bounty reward.

Matt Huang is a co-founder and general partner at the venture capital firm, Paradigm, where he focuses on investing in blockchain and cryptocurrency projects.

Alexandr Wang is the CEO and co-founder of Scale AI, a data annotation and AI training company that provides training data and validation services for machine learning models.

About the Bounty

Since ChatGPT’s release in November, the level of consumer and developer interest in large language models (LLMs) has exploded. In January, Nat had the desire to create a playground, like OpenAI’s playground, that would allow developers to test and compare different models.

The UI would have all the features of the OpenAI playground, including text streaming, keyboard shortcuts, and model selection. But, it would also include access to models from Hugging Face, Anthropic, Forefront, and Cohere.

Nat knew he needed a developer who is a hacker at heart. The best place on the Internet to find hackers that can take an idea from 0 to 1 is on Replit – specifically, Replit Bounties.

nat_dev_Bounty_announcement
nat_dev_Bounty_announcement

After posting the Bounty, Alexandr Wang and Matt Huang jumped in as co-funders to fund the project and find a developer.

mat_huang_tweet
mat_huang_tweet

Selecting a Bounty Hunter

Applications from prolific developers began flowing in.

About a week after posting the Bounty and getting Matt and Alexandr’s contributions, one application came that was hard to ignore…

Zain Huda decided to skip the line of the application board and jumped on Twitter to not only apply, but also to demo a working prototype of Nat’s vision.

zain_application_tweet
zain_application_tweet

Check out Zain’s application tweet here.

After getting Nat’s attention and passing an initial interview screen, Zain was selected as the developer to execute the full vision for the Bounty.

Creating Nat.Dev

Zain and Nat dove in and began building the initial features that were at parity with the OpenAI playground including text streaming, highlighting, and temperature controls.

As the project developed, the scope increased to add features like real-time model comparison.

The project moved fast. In Zain’s words “Nat’s expectations of speed and quality were really big drivers for me… It was amazing to see someone who’s so successful in tech and how they view building things. It set the bar for me.”

Shortly after implementing the core components and initial model configurations for the playground, another developer, Alex Lourenco, joined the project to implement model comparison, hosting, and final polishes.

What's next

The website went on to become the most popular LLM-comparison tool in February 2023.

Nat screenshot of DAUs
Nat screenshot of DAUs

The project became so popular that the team quickly had to implement a credit system to mitigate costs. In April 2023, Nat open sourced the code to allow others to build their own playgrounds.

"Without [Replit Bounties] I would have never been able to get this experience, especially this early in my career. If you had told me I would be working on software with Nat Friedman a year ago, I would have thought you were crazy."



Zain Huda / Software Engineer at Meta and nat.dev

Try out the project!

Check out the playground right here at nat.dev.

Interested in viewing the open-source code? Check out the repository here.

About Bounties

Bounties is a marketplace where anyone can work with software creators from the Replit community. The Bounty Hunter community on Replit consists of the world’s top developer talent ranging from young prodigy hackers to senior software engineers.

Post on Replit Bounties and see what creativity you can unleash!

More blog posts