In 2021, I built a startup called Linvo, a LinkedIn automation tool. It generated good money but also had a high burn rate.
I can tell you that I lost most of my money as a young entrepreneur (entirely bootstrapped) 😆
The problem with my system is that it kept on failing; I am not the best programmer in the world, but let me elaborate.
Puppeteer sucks your memory
Linvo went into a few iterations during its lifetime. I used the tech that I know: React and NestJS.
Basically, any automated cloud scraping goes into making tasks and then solving them.
It looks something like this:
- Cron runs every 10 minutes, looks for new stuff to do, and sends it to the queue.
- Workers take tasks from the queue and process them.
- Save everything to the DB.
This is a very naive solution that was running on multiple EC2 (AWS) and Redis.
In theory, it should work - Yes, you would need to scale the workers horizontally, but that was not my problem.
I started to experience something else - memory leaks.
As you probably know, Puppeteer - a chromium automated browser, is heavy (same as Chrome on your computer), and I was running lots of them - they sucked both my memory - causing the servers to stop working and took the internet bandwidth.
Needed a solution
Refactoring sounds like - yeah, let's do that, but my Intercom support was filled with support tickets (I already had many users), and I knew I had to create some crazy refactoring.
I had an idea for a solution - let's use AWS lambda.
- Lambda start
- Do the task
- Die (most of the time)
It doesn't suck your internet.
It will die and not cause a memory leak.
It's a bit more expensive because Lambdas were not meant to run indefinitely—that would remove the whole concept, but I was desperate. (Well, the good thing is that you don't need to think about scale.)
Here is the new infrastructure:
- Event bridge cron running every 1 minute (that's the minimum)
- Lambda that creates six events for the queue (making it like the cron runs every 10 seconds)
- SQS queue (bye bye, Redis)
- Lambdas shooting out with auto-scaling (yay?)
Vendor Locking
As you probably understand from the context - now I am all in with AWS, but I didn't mind - only how I would test it on my computer.
Back in the day, I used Sam. It's almost the same as Serverless to create the entire environment quickly.
And yes, they do bring some local testing. They can take your lambdas and turn them into endpoints! Yay!
But I was using Event Bridge, SQS, and Lambdas. That was just not possible.
I have encountered localstack. It was supposed to help me build an entire AWS stack on my computer, but many things were not supported back then, such as Lambda layers. And I was overwhelmed in using it.
So I started to create my "own" infrastructure where I deployed one "test" SQS on AWS and ran local code with it (a function that calls a function), and it was - bad.
I did not work well in production; I felt like the word was squeezing up on me back in the day.
I just wanted to get rid of this startup; I even offered it to people for free, taking the customers and rid myself of that problem.
Long story short, I managed to sell the system for a low price, but lately, I discovered something interesting.
Winglang - Typescript like dialect for the cloud
I guess my pain was real, and even though I would be happier to find this back in the day - I am glad it's alive today.
Winglang helps you to build the same cloud infrastructure as I used, but the nice thing is:
- It doesn't require you to use any configuration files. You basically define the entire infrastructure with code (you can check their entire preflight and inflight documentation)
- You can run it locally - thank you, god, 🙏🏻🙏🏻🙏🏻🙏🏻 with all the infrastructure included.
- You can deploy it to any cloud provider, and you don't have to use AWS. They can literally generate any Terraform or cloud formation for easy deployment.
I talked to the founder of Winglang, Elad (builder of the famous CDK), and he said something that made me twitch.
You can literally work on your projects when you are on an airplane and write tests for it 🤯
I can't give them enough stars myself - but do me a favor and star their repository and of course, use it for your next project.
https://github.com/winglang/wing
Final words
I am angry at Winglang. I wish they had done it sooner. Now they show up. When it's all about AI?
I started a new YouTube channel mostly about Tech, open-source, and marketing :)
If that's something that interests you, feel free to subscribe to it here:
https://www.youtube.com/@nevo-david?sub_confirmation=1
Top comments (39)
I wished to see more examples on the main website → playground section.
Ideally, it would be great to consider the use-case driven approach and prepare or show all the examples. That's how you can explain things in a better manner. Also, it would be realistic too.
Not sure I understand your comment :)
Can you please elaborate?
@nevodavid Please navigate to Winglang Playground where you will see two samples listed. Those are really basic and doesn't provide a realistic use-case driven approach. Hence, the need to invest time in documenting and providing as many samples as part of the playground. I hope, it's practical and is required for every single developer.
Also request to provide more use-case or cookbook driven samples on winglang examples
Thanks for the feedback @ranjancse. We will try to add more examples to the playground, but since it is just a playground and not an IDE it cannot run more complex apps with more than 1 file, external JS or winglibs.
About winglang examples, we actualy have a dedicated repo for it that is more what you meant (I hope) - github.com/winglang/examples.
But we need to organize our materials better since it is confusing that there is an examples folder in the language repo (this is the one you found)
Oh wow!! Thanks for sharing the detailed examples repo.
Hi. @shaiber Can this github.com/winglang/examples be linked in the playground dropdown? Maybe something like "Find out more".
@ranjancse, I'm on the Wing team and we appreciate your feedback.
Thank you for suggesting this.
Great article @nevodavid! It almost makes me want to try and build Linvo with Wing :)
Haha, it's not fun, believe me!
Maybe with Wing :D
That's why I wrote "almost" :)
I am building a tech startup myself and I almost did it the bad way myself, just like you did. Fortunately, I listened for my instinct and did not do it the startup/hacky way.
I am a huge fan of tech startups and YCombinator. Among many great advices they give, one is that we should always build things that don’t scale. They mean it for both business and tech parts. They advocate building “ugly” interfaces and non-scalable systems, just so we can have something up and running very fast, so we can validate early. I agree to some extent. When you really have no clue on whether your startup is going to work or not, then, yes, by all means, do things in a hacky way. But when uou have a bit of validations, some insights and a few early adopters like I did when I started, then no, don’t hack your way through. Make it scalable from the get-go. I like it this way because 1. Even if your software is built in a hacky way but still still, it is rather difficult to scale it. Scaling from spaghetti to scalable systems and code is WAY HARDER than building something from 0. Having it built in a great scalable way from the start makes building things on top of it a breeze. And 2. No stress. If you get a bigger influx of users, you’re all set.
My approach is: instead of building a complex architecture and system which does 100 things but it’s hacky, better build a thing that does 2-3 things and they are absolutely killing it. My marketplace, for example. Account creation, authentication, service creating/adding, and payments, all work 100% with a system low coupled, scalable and robust. Granted, those are just the basic functionalitity but that’s the point. Depth vs breadth. Add a service, make it scalable and independent, teeeeeest it until it becomes obnoxious to do so and then you’re good for something else.
I understand what you are saying that’s why the first solution was naive.
By the time I made the second solution I already had a lot of users and revenue:)
I might be a bit biased (I led PMM for Eventbridge and SQS for a few years) but that's a solid stack for an application like Linvo. At the startup stage, vendor lock-in is just FUD and AWS is really good about supporting the startup ecosystem.
I hadn't heard of Wingless, but I'm going to check it out!
Thank you so much, Jesse!
And yes! it works great, my main problem was local testing
The gist of it is, we're doomed with Chrom(e/ium), and without it...
Wasn't aware of this story of yours @nevodavid
Never shared it 🙈
Great insights @nevodavid !
Thank you so much ❤️
Nice article! I love serverless and Winglang looks super useful for local dev
Thank you so much 🙏🏻
Thanks for sharing your experience! Wing sounds awesome to try.
Thank you so much Namee 🙏🏻
👏👏👏
🎉
I NEVER heard of Wing before, and that turns out to be a shame, because this looks like a very ambitious project - I absolutely applaud the effort these guys are putting into this, it looks really serious ... but it's pure coincidence that I came across this dev.to article, or I would never have heard about it!
So maybe you guys should really start promoting this more ... I mean, I knew about the Serverless framework, I heard about the (much lesser known) SST, and of course I knew about CDK - but I hadn't the faintest clue that Wing (Wing Lang?) existed - this might be your #1 problem to solve? :D
Also, I agree with the other commenter that you should probably put more effort into adding more examples - the repo (github.com/winglang/examples) is cool, but I still feel it's a little bit limited ...
Are there (for instance) examples on: How to set up a VPC on AWS? How to set up and "talk to" a relational database (RDS on AWS)? and so on ... I feel that more of these "real world" examples will convince people that your platform is a viable choice ...
But, big time kudos for this impressive platform - absolutely a laudable and serious effort!
Thanks for the feedback @leob!
I'm Shai, one of Wing's founders.
I guess the reason you haven't heard of Wing yet is that we're a pretty new project :)
This is also the reason why our examples and docs need more polish as you noticed.
We are following your advice and have started to add examples with more meat to the playground. We will also link to the examples repo from it.
You can also expect more real world examples in the examples repo soon.
Thanks, that explains a lot! So do you consider the product/platform (what shall I call it?) to be still in "beta", or is it "production ready"?
I think a lot of people are going to compare Wing to the likes of Serverless framework, SST, CDK and so on ... maybe you should add a section to your FAQ to compare your stuff with these "competitors" (or maybe you have that already, haha).
Keep up the good work!
So interesting to see public post mortem's on failed start up implementations. I feel I learn more from the failures than from the success stories. Much appreciated @nevodavid
Thank you so much, Jake!
You learn from every failure, that makes us who we are today 🙏🏻
Great article Nevo, BTW, I think you're an excellent developer :)
Thank you so much Nathan ❤️
To solve similar problem I built a small project preview
It gets product data from link using serverless scraping on AWS.
Looks good!
Gave it a star!
Just on Puppeteer, we use it and from my testing I have had no issues with memory. We create a "browser" instance that is always running and then when we need to we create a "browser tabs" to handle the HTML. After we are done with the tab, we just close it. Running htop you can see the system resources being utilized and then released after you close the tab.
Yes of course, but what is your concurrency, how many chrome tabs do you run at the same time, on how many servers?
I am assuming then that you did use tabs and you closed them after every event...And you observed memory not being released... How much data were you pushing into each tab? I don't want to come across as attacking just curious about your situation.
We have one server that does all our PDF generation used by multiple services.
Gotcha!
Yes, I was closing and opening tabs :)
But my tasks were a little bit more heavy as they automated Linkedin (go between the pages) and run for at least 5-6 minutes every time.
For how much you sold it, and who was the buyer?
It was to a Linkedin lead gen agency. I can't disclose the price 🙏🏻