DEV Community

Cover image for Smol AI 🐣 vs Wasp AI 🐝- Which is the Better AI Junior Developer?
vincanger for Wasp

Posted on • Originally published at wasp-lang.dev

Smol AI 🐣 vs Wasp AI 🐝- Which is the Better AI Junior Developer?

TL;DR

AI-assisted coding tools are on the rise. In this article, we take a deep dive into two tools that use similar techniques, but are intended for different outcomes.

Smol AI’s “Smol-Developer” gained a lot of notoriety very quickly by being one of the first such tools on the scene. It is a simple set of python scripts that allow a user to build prototype apps using natural language in an iterative approach.

Wasp’s “GPT Web App Generator” is more of a newcomer and focuses on building more complex full-stack React + NodeJS web app prototypes through a simple prompt and fancy UI.

When comparing the two, Smol-Developer’s strength is its versatility. If you want to spend time tinkering and tweaking, you can do a lot to your own prompting, and even the code, in order to get decent results on a broad range of apps.

On the other hand, Wasp AI shines by being specific. Because it’s only built for generating full-stack React/NodeJS/Prisma/Tailwind codebases, it does the tweaking and advanced prompting for you, and thus it performs much better in generating higher quality content with less effort for a specific use case.

Image description

Will either of these tools completely replace Junior Developers in their current form? Of course not. But they do allow for rapid prototyping and testing of novel ideas.

Read on to learn more about how they work, which tool is right for the job at hand, and how you can use them in your current workflow.

Intro

The age of AI-assisted coding tools is fully upon us. GitHub’s Copilot might be the go-to professional solution, but since its release numerous open-source solutions have popped up.

Most of these newer solutions tend towards functioning as an AI Agent, going beyond just suggesting the next logical pieces of code within your current file, they aim to create simple prototypes of entire apps. Some are focused more on scaffolding entire app prototypes from an initial prompt, while others function as interactive assistants, helping you modify and improve existing codebases.

Either way, they’re often being described as “AI Junior Developers”, because they can take a product requirement (i.e. “prompt”) and build a pretty good — but far from perfect — first iteration, saving developers a lot of time.

This article is going to focus on two tools that aim to build somewhat complex working prototypes from a single prompt: Smol AI and Wasp AI. We’ll test them out by running the same prompts through each and seeing what we get.

By the end of it, you’ll have a pretty good understanding of how they work, their advantages and disadvantages, and what kind of tasks they’re best suited for.

Before We Begin

Wasp = } is the only open-source, completely serverful fullstack React/Node framework with a built-in compiler and AI-assisted features that lets you build your app super quickly.

We’re working hard to help you build performant web apps as easily as possible — including creating content like this, which is released weekly!

We would be super grateful if you could help us out by starring our repo on GitHub: https://www.github.com/wasp-lang/wasp 🙏

please please please

…even Ron would star Wasp on GitHub 🤩

⭐️ Thanks For Your Support 🙏


The Tools

Smol-Developer

Smol AI (described as a platform for “model distillation and AI developer agents”) actually has a few open-source tools on offer, but Smol-Developer is the one we’ll be taking a look at. It was initially released by Swyx on May 11th and already has over 10k GitHub stars!

It aims to be a generalist, prompt-based coding assistant run from the command line. The developer’s job becomes a process of iterative prompting, testing, and re-prompting in order to get the optimal output. It is not limited to any language or type of app it can create, although simple apps tend to work best.

Check out this tweet thread below to get a better understanding:

Running from the command line, Smol AI is essentially a chain of calls to the OpenAI chat completions (i.e. “ChatGpt”) endpoint via a python script that:

  1. takes an initial user-generated prompt
  2. creates a plan based on internal prompts* for executing the app with:
    1. the structure of the entire app
    2. each file and its exported variables to be generated
    3. function names
  3. generates file paths based on the plan
  4. loops through file paths and generates code for each file based on plan and prompt

The generated output can then be evaluated by the developer and the prompt can be iterated on to account for any errors or bugs found during runtime.

Smol-Developer quickly gained notoriety by being one of the first of such tools on the scene, in addition to Swyx’s prominence within it. So if you’re curious to see what’s being built with it, just check out some of the numerous YouTube videos on it.

One of my personal favorites is AI Jason’s exposé and commentary. He gives a concise explanation, shows you some great tips on how to use Smol-Developer effectively, and as a Product Designer/Manager he gives an interesting perspective on its benefits:

Wasp’s GPT Web App Generator

In contrast to Smol-Developer, Wasp’s AI tool, GPT Web App Generator, is currently an open-source web app (yes, it’s a web app that makes web apps). Since it’s release on the 12th of July, there have been over 6,500 apps generated with over 300 apps being generated each day!

Image description

Here’s a quick 1 minute video showcasing how GPT Web App Generator works:

So to give a bit of background, Wasp is actually a full-stack web app framework built around a compiler and config file. Using this approach, Wasp simplifies the web app creation process by handling boilerplate code for you, taking the core app logic written by the developer and connecting the entire stack together from frontend to backend, and database management.

It currently works with React, NodeJS, Tanstack-Query, and Prisma, taking care of features like Auth, Routing, Cron Jobs, Fullstack Typesafety, and Caching. This allows developers to focus more on the fun stuff, like the app’s features, instead of spending time on boring configurations.

Because Wasp uses a compiler and config file to generate the app from, this makes it surprisingly well suited for guiding LLMs like ChatGPT towards creating more complex apps with it, as it essentially a plan or set of instructions for how to build the app!

Take this simple example of how you’d tell Wasp that you want username and password authentication in your app:

// main.wasp file

app RecipeApp {
  title: "My Recipes",
  wasp: { version: "^0.11.0" },
  auth: {
    methods: { usernameAndPassword: {} },
    onAuthFailedRedirectTo: "/login",
    userEntity: User
  }
}

entity User {=psl  // Data models are defined using Prisma Schema Language.
  id          Int        @id @default(autoincrement())
  username    String     @unique
  password    String
  recipes     Recipe[]
psl=}
Enter fullscreen mode Exit fullscreen mode

Wasp’s config file is like an app outline that the compiler understands and can then use to connect and glue the app together, taking care of the boilerplate for you.

By leveraging the powers of Wasp, GPT Web App Generator works by:

  1. taking a simple user-generated prompt via the UI
  2. giving GPT a descriptive example of a Wasp app and config file via internal prompts*
  3. creating a plan that meets these requirements
  4. generating the code for each part of the app according to the plan
  5. checking each file for expected errors/hallucinations and fixing them

In the end, the user can download the codebase as a zipped file and run it locally. Simpler apps, such as TodoApp or MyPlants tend to work straight out of the box, while more complex apps need a bit of finessing to get working.

Comparison Test

Prompt 1: PONG Game

To get a sense for how each coding agent performed, I tried out two different prompts on both Smol-Developer and Wasp’s GPT Web App Generator with only slight modifications to the prompts to fit the requirements of each tool.

The first prompt was the default prompt that comes hardcoded into Smol-Developer’s [main.py](http://main.py) script:

a simple JavaScript/HTML/CSS/Canvas app that is a one player game of PONG. 
The left paddle is controlled by the player, following where the mouse goes. 
The right paddle is controlled by a simple AI algorithm, which slowly moves the paddle toward the ball at every frame, with some probability of error. 
Make the canvas a 400 x 400 black square and center it in the app. 
Make the paddles 100px long, yellow and the ball small and red. 
Make sure to render the paddles and name them so they can controlled in javascript. 
Implement the collision detection and scoring as well. 
Every time the ball bounces off a paddle, the ball should move faster.
Enter fullscreen mode Exit fullscreen mode

Note ℹ️
For Wasp’s GPT Web App Generator, I replaced the first line with “a simple one player game of PONG” since Wasp will automatically generate a full-stack React/NodeJS app.

Both were able to create a functional PONG game out-of-the box, but only on the second try. The first try created decent PONG starters, but both had buggy game logic (e.g. computer opponent failed to hit ball, or ball would spin off into oblivion). I didn’t change the prompts at all, but just simply ran them a second time each — and that did the trick!

Image description

Image description

For both of the generated apps, the game logic was very simple. Scores weren’t recorded, and once a game ended, you’d have to refresh the page to start a new one.

Although, while Smol-Developer only created the game logic, GPT Web App Generator created the game logic as well as the logic for authentication, creating games, and updating a game’s score, saving it all to the database (though the scoring functions weren’t being utilized initially).

To be fair, this isn’t really a surprise though as these features are baked into the design of Wasp and the Generator.

On the other hand, to get these same features for Smol-Developer, we’d have to elaborate on our prompt, giving it explicit instructions to implement them, and iterate on it a number of times before landing on an acceptable prototype.

This is what I attempted to test out with the second prompt.

Prompt 2: Blog App

Image description

This time, for the second app test, I used a default prompt featured on the GPT Web App Generator homepage for creating a Blog app:

A blogging platform with posts and post comments.
User owns posts and comments and they are saved in the database.
Everybody can see all posts, but only the owner can edit or delete them. Everybody can see all the comments.
App has four pages:

1. "Home" page lists all posts (their titles and authors) and is accessible by anybody. 
If you click on a post, you are taken to the "View post" page.
It also has a 'New post' button, that only logged in users can see, and that takes you to the "New post" page.
2. "New post" page is accessible only by the logged in users.
It has a form for creating a new post (title, content).
3. "Edit post" page is accessible only by the post owner. 
It has a form for editing the post with the id specified in the url.
4. "View post" page is accessible by anybody and it shows the details of the post with the id specified in the url: its title, author, content and comments.
It also has a form for creating a new comment, that is accessible only by the logged in users.
Enter fullscreen mode Exit fullscreen mode

Note💡
For the Smol-Developer prompt, I added the lines: “The app consists of a React client and a NodeJS server. Posts are saved in an sqlite database using Prisma ORM.”

As this was a suggested prompt on the GPT Web App Generator page, let’s start with the Wasp app result first.

After downloading the generated codebase and running the app, I ran into an error Failed to resolve import "./ext-src/ViewPost.jsx" from "src/router.jsx". Does the file exist?

Image description

One quick look at the main.wasp file revealed that the Generator gave the wrong path to the ViewPost page, although it did get all the other Page paths correct (highlighted in yellow above).

Once that path was corrected, a working app popped up at localhost:3000. Nice!

The video above was my first time trying out the app, and as you can see, most of the functionality is there and working correctly — Authentication and Authorization, and basic CRUD operations. Pretty amazing!

There were still a couple of errors that prevented the app from being fully functional out-of-the-box, but they were easy to fix:

  1. Blog posts on the homepage did not have a link in order to redirect to the their specific post page — fixable by just wrapping them in <Link to={/post/${post.id}}>
  2. The client was passing the postId as a String instead of an Int to the getPost endpoint — fixable by wrapping the argument in parseInt(postId) to convert strings to integers

Image description

And with those simple fixes we got a fully functioning, full-stack blog app with authentication, database, and simple tailwind css styling! The best part was that all this took about ~5 minutes from start to finish. Sweet :)

Note 🧑‍💻
The Generator saves all the apps it creates along with a sharable link, so if you want to check out the original generated Blog app code (before fixes) from above, click here: https://magic-app-generator.wasp-lang.dev/result/a3a76887-952b-4774-a773-42209c4bffa8

The Smol-Developer result was also very impressive, with a solid ExpressJS server and a lot of React client pages, but there were too many complicated errors that prevented me from getting the app started, including but not limited to:

  1. No build tools or configuration files
  2. The server was importing database models that didn’t exist
  3. The server was importing but not utilizing Prisma as the ORM to communicate with the DB
  4. Client had Auth logic, but was not utilizing it to protect pages/routes

Image description

Because there were too many fundamental issues with the app, I went ahead and added some more lines to the bottom of the prompt:

…

Scaffold the app to be able to use Vite as the client's build tool. Include a package.json file >with the dependencies and scripts for running the client and server.

This second attempt produced some of the changes I was looking for, like package.json files and Vite config files to bootstrap the React app, but it still failed to include:

  1. An index.html file
  2. Package.json files with the correct dependencies being imported from within the client and server
  3. A prisma.schema file
  4. A css file (although it did include classNames in the jsx code)

On the other hand, the server code, albeit much sparser this time, did at least import and use Prisma correctly.

So I went ahead for a third attempt and modified and added the following lines to the bottom of the prompt:

…

Scaffold the app to be able to use Vite as the client's build tool.

Make sure to include the following:

  1. package.json files for both the server and client. Make sure that these files include the >dependencies being imported in the respective apps.
  2. an index.html file in the client's public folder, so that Vite can build the app.
  3. a prisma.schema file with the models and their fields. Make sure these are the same models >being used app-wide.
  4. a css file with styles that match the classNames used in the app.

With these additions to the prompt, the third iteration of the app did in fact include them! Well, most of them, but unfortunately not all of them. Now I was getting the css and package.json files, but no vite config file was created this time, even though the instructions for using “Vite as the client’s build tool” produced one previously.

Besides that, no auth logic was implemented, imports were out place or missing, and an index.jsx file was also nowhere to be found, so I decided to stop there.

Image description

I’m sure I could have iterated on the prompt enough times until I got closer to a working app, but at ~$0.80-$1.20 a generation, I didn’t feel like racking up more of an OpenAI bill.

Note 💸
Price per generation is another big difference between the Smol AI and Wasp AI. Because more work is being done by Wasp’s compiler and less by GPT, each app costs about ~$0.10-$0.20 to generate (although Wasp covers the cost and allows you to use it for free), whereas to generate complex full-stack apps with Smol-Developer can cost upwards of ~$10.00!

Plus, there are plenty of YouTubers who’ve created videos about the process of using Smol-Developer and it seems they all come to similar conclusions: you need to create a very detailed and explicit prompt in order to get a working prototype (In fact, in AI Jason’s Smol-AI video above, he mentioned that he got the best results out of the box when prompting Smol-Developer to write everything to one file only — of course this limits you to generating simple apps only that are not so easy to continue from manually).

Thoughts & Further Considerations

At their core, SmolAI and WaspAI function quite similarly, by first prompting the LLM to create a plan for the app’s architecture, and then to execute on that plan, file by file.

But because Smol-Developer aims to be able to generate a wider range of apps, the expectation is on the Developer (or “Prompt Engineer”) to create a highly detailed, explicit prompt, which is more akin to a Product Requirement Doc that a Product Designer would write. This can take a few iterations to get right and pushes Smol-Developer in the direction of “Natural Language Programming” tool.

On the other hand, Wasp’s GPT Web App Generator has a lot of prompting and programming going on behind the scenes, abstracted away from the user and hidden within the Generator’s code and Wasp’s compiler. Wasp comes with a lot of knowledge baked in and already has a good idea of what it wants to build, which means the user has less to think about it. This means that we’re more likely to get a working complex prototype from a short, simple prompt, but we have less flexibility in the kinds of apps we’re able to create — we always get a full-stack web app.

In general, Wasp is like a junior developer specialized in web dev and has a lot of experience with a specific stack, while Smol AI is a junior developer that’s a generalist who is more versatile, but has less specific knowledge and experience with web dev 🙂

Smol AI Wasp AI
🧑‍💻 Types of Apps Varied Full-stack Web Apps
🗯 Programming Languages All Types JavaScript/TypeScript
📈 Complexity of Generated App Simple to Medium Medium to Complex
💰 Price per Generation — via OpenAI’s API $0.80 to $10.00 $0.10 to $0.20 
💳 Payment Method bring your own API key free — paid for by Wasp
🐛 Debugging Yes, if you’re willing to tinker Built-in, but limited
🗣 Type of Prompt Needed Complex and detailed, 1 or more pages (e.g. an entire Product Requirement Doc) Simple, 1-3 paragraphs
😎 Intended User Engineers, Product Designers wanting to generate a broad range of simple prototypes Web Devs, Product Designers that want a feature rich full-stack web app prototype

Other big differences lie within:

  1. Error Correction upon Code Creation
    1. Smol AI initially had a debugging script, but this has temporarily deprecated due to the fact that it expects the entire codebase when debugging, and current 32k and 100k token context windows are only available in private beta for GPT4 and Anthropic at the moment.
    2. Wasp AI has some error correction baked into its process, as the structure of a Wasp app is more defined and the range of errors are more predictable.
  2. Price per app generation via OpenAI’s chat completion endpoints
    1. Smol AI can cost anywhere from ~$0.80 to $10.00 depending on the complexity of the app.
    2. Wasp AI costs ~$0.10 to $0.20 per app, when using the default mix of GPT 4 and GPT 3.5 turbo, but Wasp covers the bill here. If you choose to run it just with GPT4, then the cost is 10x at $1.00 to $2.00 per generation and you have to provide your own API key.
  3. User Interface
    1. Smol Developer works through the command line and has minimal logging and process feedback
    2. Wasp AI currently uses a clean web app UI with more logging and feedback, as well as through the command line without a UI (you have to download the experimental Wasp release to do so at this time).

Overall, both solutions produce amazing results, allowing solo developers or teams iterate on ideas and generate prototypes faster than before. But they still have a lot of room for improvement.

For example, what these tools lack the most at the moment is in interactive debugging and incremental generation. It would be great if they could allow the user to generate additional code and fix problems in the codebase on the fly, rather than having to go back, rewrite the prompt, and regenerate an entire new codebase.

I’m not aware of the Smol AI roadmap, but seeing that it’s received a grant from Vercel’s AI accelerator program, I’m sure we will be seeing development on it continue and the tool improve (let me know in the comments if you do have some insight here).

On the other hand, as I’m a member of the Wasp team, I can confidently say that Wasp will soon be adding the initial generation process and interactive debugging into Wasp’s command line interface!

So I definitely think it’s early days and that these tools will continue to progress — and continue to produce more impressive results 🚀

Which Tool Should You Use?

Obviously, there can be no clear winner here as the answer to question of which tool you should use as your next “AI Junior Developer” depends largely on your goals.

Are you looking for a tool that can generate a broad range of simple apps? And are you interested in learning more about building AI-assisted coding tools and natural language programming and don’t mind tweaking and tinkering for a while? Well then, Smol-Developer is what you’re looking for!

Do you want to generate a working full-stack React/Node app prototype with all the bells and whistles as quickly and easily as possible? Head straight for Wasp’s GPT Web App Generator!



Help Me Help You 🌟
If you haven’t yet, please star us on GitHub, especially if you found this useful! If you do, it helps support us in creating more content like this. And if you don’t… well, we will deal with it, I guess.

https://media.giphy.com/media/3oEjHEmvj6yScz914s/giphy.gif

⭐️ Thanks For Your Support 🙏


In general, as Jason “AI Jason” Zhou said:

“I’m really excited about [AI-assisted coding tools] because if I want to user-test a certain product idea I can ask it to build a prototype very, very quickly, and test with real users”

Jason makes a great point here, that these tools don’t really have the capacity to replace Junior Developers entirely in their current capacity (although they will surely improve in the future), but they do improve the speed and ease with which we can try out novel ideas!

I personally believe that in the near future we will see more domain-specific AI-assisted tools like Wasp’s GPT Web App Generator because of the performance gains they bring to the end user. Code agents that are focused on a niche can produce better results out of the box due to the embedded knowledge. In the future, I think we can expect a lot of agents that are each tailored towards fulfilling a specific task.

But don’t just take my word for it. Go ahead try out Smol-Developer and the GPT Web App Generator for yourself and let me know what you think in the comments!

Top comments (15)

Collapse
 
craigmc08 profile image
Craig McIlwrath

It seems like purpose-built AI developers are always going to be more successful than general ones: shared across every web app, or server application, or any other kind of app, there will always be a lot of common knowledge that the agent needs. Putting that work onto the tool developer will make using it easier and allow for ever-increasing quality, as the tool developer iterates on the prompts.

GPT App Generator is a specific tool -> generating web apps from a description.

Smol AI is trying to be too general -> generating any app from a description. But these descriptions have to be so specific, at some point you have to wonder when you're just trying to program with English instead of code. And this makes sense: we've seen from failures in the past (like COBOL) how trying to make a general-purpose program description language to obviate the need for programmers just results in a bad programming language that still requires programmers.

Collapse
 
vincanger profile image
vincanger

this is very well put. it will be interesting to see how this holds up as LLMs improve over time.

Collapse
 
matijasos profile image
Matija Sosic

wow, well said :)

Collapse
 
nevodavid profile image
Nevo David

Of course Wasp AI, is that even a question?

Collapse
 
infomiho profile image
Mihovil Ilakovac

Smol step for a human, a big step for humanity :D

Collapse
 
vincanger profile image
vincanger

😂

Collapse
 
marinsmiljanic profile image
Marin Smiljanić

Great comparison, to the point!

Collapse
 
vincanger profile image
vincanger

Thanks, Marin!

Collapse
 
shayneski profile image
Shayne Czyzewski

Great comparison! I think Wasp really has a great future in the AI-assisted web dev space 🚀

Collapse
 
vincanger profile image
vincanger

Thanks!

Collapse
 
kevinlu1248 profile image
Kevin Lu

Also missed Sweep AI, an AI junior dev that's fully asynchronous, so you can offload your workflow to Sweep and work on something else. You can a GitHub issue for Sweep to address it by making a pull request. sweep.dev

Collapse
 
vincanger profile image
vincanger

What’s been your experience with it? It’s very expensive! Smol and Wasp AI are much cheaper

Collapse
 
kevinlu1248 profile image
Kevin Lu

Full disclosure: I'm one of the developers of Sweep. Our experience using it internally has been amazing, allowing us to ship way faster than before for small menial tasks.

We charge more because Sweep does more. Sweep's a fully autonomous AI junior dev with more scope rather than a convenient tool in your IDE or the like.

Collapse
 
matijasos profile image
Matija Sosic

wow, this is a super detailed comparison! I especially like the table where you can quickly see the differences between the two.

Collapse
 
vincanger profile image
vincanger

Thanks!