This article is about how I tested a hypothesis I had by building a website in 3 days. It's not my first time building something to test business hypothesis but this time I wanted to do it faster and document the processes.
Hypothesis
The hypothesis is that people enjoy commenting 💩 and reading 😖 comments about news the same way they do with YouTube videos. There are some news sites offering comment option but I don't like all of them. For example Hacker News is great website but the "news" (sometimes just Wikipedia article) rarely interest me.
How to test the hypothesis?
The first step is to think how to test this hypothesis 👉AS EASILY AS POSSIBLE👈. The goal is to not build first version of the final product but test the hypothesis. If you can do it by interviewing people, then start with that.
I can't do a newsletter because commenting would be a problem. I could go some kind of forum and try to get people talk there. Or maybe Reddit. The thing is that it's always difficult to get first 100 people to even look the idea. On Reddit it would be a huge problem to get enough people to even look the subreddit. I couldn't say that people don't like the idea but it might be just that not enough people saw it.
There is a trick 🧙♂️. Site called Product Hunt. It's a website where users can post products (own or someone else). Other users can up vote 👍 them which increase their placement in the daily feed.
Product Hunt is a great way to get a lot of people to look the idea. You can't publish a form or subreddit on Product Hunt but it needs to be a product. That's why I need to build something I can publish on Product Hunt.
I decided to do a simple website 💻. There is no point of listing all possible news as the comments would spread. That is why I need to show only top x news of the day. I decided that 10 news a day is good.
One problem with 10 news a day is that I can't please everyone. I need to start with some small group of people. Because my goal is to launch this on Product Hunt, I should build it for the Product Hunt users. They are mostly startup and technical people (like me 🙂). I still want to list at least 3 news about some big topics and not just some niche tech news.
The step number two when building something is to decide what the MVP looks like. Most of the people should create exact list of things they want to be in the MVP because otherwise they start adding more and more features. This happens so easily and in many forms that it's hard to notice it until it's too late. I have done that mistake in the past a few times 🤦 and feel like I have learned to sense when to build something or not. I still do high level list of features for MVP.
MVP:
- Show 10 news that change every day (I'm deciding them)
- Users can comment news and reply other people's comments
- Users can like and dislike comments (to show the best comments and hide spam or trash comments)
Okay so now I have a good plan. The idea is to create Product Hunt launch and see how people respond. If everyone churns the same day, I just assume that the hypothesis wasn't right. 🤞
In Lean Analytics book the authors said how you should have exact definition of success because otherwise you might end up the worst situation possible, which is that you get stuck to idea that works but never becomes big. At this point I don't believe it make sense to do that kind of goal because MVPs are not for everyone (some expect good looking and ready products) and you learn probably something big. If it seems like people like the idea and you do the next version, it's time to define the goal.
The opposite is to look when to quit. Especially when doing something bigger, I have found it the best to do Quitting Contract (from The Dip book). There you define the situation where you quit. Too often people end up "living dead" zone where they try a bunch of stuff without understanding that the core hypothesis doesn't work. And some people in the other hand quit too early because they face the first problem. With smaller projects like this, it's easier to do more rational decisions as I haven't invested that much time.
Building
The final website is here: https://comment-news.web.app/
15th
I spent ~2 hours coding. I decided to use ReactJS because I'm familiar with it which allows me to build stuff faster and there aren't any limitations at this point.
I always do mockups using notepad and pen instead of tools like Figma. The reason is that I can do these in a few minutes when, at least for me, it would took much longer to do it on Figma.
I'm bad at estimating but I believe I can do this during the next week. Because there is no external deadline, I don't feel like I need to do any deadlines for myself either.
At this point I don't do any kind of TODO list of all the tasks. I think it's just faster to have high level view and then plan more accurately in daily or task basis. I make TODO every evening for the next day.
16th
1.30 pm
5.00 pm
🌅 end of the day
I got the website pretty far. I still need to add a lot of functionalities to the buttons and setup database. Right now I'm using hard coded data.
17th
11am - After making reply work, the code starts to get complex so I will spend some time cleaning 🧹 it. This is something I do once in a while. I don't really plan ahead but just build stuff. This approach always causes hard to read code. It's pretty easy to then just take an hour or so and clean it. I have found it much faster than trying to plan everything ahead and write clean code right away.
2.30 pm - The code is polished and now it's much easier to work on other stuff. I made replying work properly.
3.30 pm - Sorting is ready. I don't think having ability to sort by date is something people will use a lot so I could leave it out but I added it anyways because it kind of came automatically after doing vote sorting that's anyways needed.
"It was easy to add" is often the reason why MVPs have a lot of features and it's not a good thing. MVP should focus only the core feature and everything else should be left out even if it's easy to add and doesn't seem to make the code much complex.
Locally everything is ready. The next I will setup database. I decided to use Firebase because it's again something I'm familiar with and it's not causing any limitations at this point.
When I was coding the website I had some hard coded data. I used it only through one file. Now it's easy to just change this file to use database.
6.30 pm - I got the database working.
8.00 pm - Fixed some small things and everything seems to be working fine. I still need to improve the UI but the next thing I'm going to do is a tool I can use to submit the news everyday.
I don't really care what it is going to look like because I'm the only one using it. I need some UI and possibly also ability to use it on my phone so it's probably the best to just create another ReactJS website. I actually put it to the same website but under some secret 🤫 path and behind a password.
18th
9.00 am - I'm working on dev tool that I can use to add news to the feed. I'm not going to make it secure because it requires too much work. Basically anyone can add news to the feed but it's not so easy that people without technical skills could do it.
10.30 am - Dev tool is ready. The next job is to improve UI of the website. I try to keep it simple because I'm not good at designing websites.
I like to put bright colors 🌈 to different elements at this point to easily see all paddings and the size of them. Some people go 🥜 when they see this, thinking that I'm going to make it look like this.
Emojis 🗿 always make everything look better.
1.30 pm - Small break 🌮
2.00 pm - Back to code 🧑💻 (or more like this 👉 💻🖥🖥)
I'm not sure if the website looks like this on my iPhone because the pixels aren't exactly right even though Chrome says it should look like this.
With "right" numbers it should look like this which is much better.
I try to deploy it at this point so I can easily test it on my real phone. I'm going to again use Firebase for this because they have great hosting solution that's super easy to setup for ReactJS projects.
Okay it looks like the first image which is the worse 😟
Got it! 🎉
It's 2.30 pm and the final UI related thing is the feed. This is what it looks like right now.
I don't want to touch it much. I like this kind of colorful design. I want to change the color of text, I need to decide how multiple items will look, and I will add a button to the right side in case people don't understand that the text is link.
This might make someone with better color sense to 🤮 but it looks cool, imo, so it's fine.
I don't think you should take everything in the product too seriously and try to make it look like some big corporation did it. Do things that make you happy and the users will notice it and like it more than trying to pretend a big company.
3.15 pm - Icon done. I just took news paper emoji 📰 and put it top of the blue color I used on the website. The name is at least for now "Comment News". I bet most of the people don't pay attention to these things and that is why I don't either.
The next I add the most important thing, analytics.
4.00 pm - I decided to use Google Analytics because again I have used it in the past and it's not limiting anything at this point. I track page loads to see how many people visits but also button clicks to see how active they are. Basically I just want to see how many people came to the website, how many of them return later, and do they vote, comment, or do any other action.
5.00 pm - I'm interested to not just see if people want to comment news and read other people's comments but also see if they like the format of getting 10 news a day. It saves a ton of time and often 10 is enough to stay updated of everything. That is why I added an input box where users can put their email to get the feed sent to their inbox everyday. I'm not going to implement that feature right away but I collect emails to database to see the demand but also have them later if needed.
I'm going to take a break to eat something and then I will figure out what to do next.
5.45 pm - Okay, so I thought that it's really important to have users because the whole point of the website is the comments. Because I need to start from zero, I need to pretend like there are users. I thought about doing some kind of automation where I scrape tweets from Twitter 🐦🔫 and submit them as comments (assuming that I find a thread where people talk about that news) but then I realize that it's probably too much.
The whole point of this experiment is to launch it on Product Hunt and then just see how they respond to it. I need to add a lot of comments at least the first day so that when the Product Hunt users come to the website they think that it's used a lot. Then hopefully those users start to comment the news, the next day. But to make sure there is always comments, I could also make fake comments for the second day. After that I might do some fake comments but the goal is that users start to comment or the website doesn't work and I shut it down. So there are only 1-3 days when I seriously need to write comments and it's faster to do them manually than trying to create some scraping tool.
I thought about buying a domain but I don't want to spend $1 for .xyz because I'm not sure if it's any better than the subdomain Firebase is offering and $9 for .com is too much for my wallet 💰 (I'm cheap).
The only thing left is to prepare Product Hunt launch. The same time I will initialize database for tomorrow. I will do the final review of every feature to make sure there aren't any major 🐛.
8.00 pm - I haven't though much about picking top 10 news but when I started doing it, I realized how hard it actually is. I first opened a bunch of different news sites and tried to find if there are some overlapping topics. There kind of were but then in some news I wasn't sure if they were about the same topic. I right away realize that it requires too much time. Then I went to Google News. They list pretty well the most interesting topics from different categories and the best is that they collect "all" the sources talking about that topic.
I started manually picking topics and trying to copy paste all the articles to notepad but after 30 minutes and 3 topics I realize how much time it would take 🤨. I don't need to do it that many day but it's still too much work for me. I needed to create a scraper.
When building automation for MVP, it's important to think how much it takes to build it and how much time it saves. It's often hard to estimate but I would say that only automate stuff that saves you double the time it takes to build. Just because often software building estimations seem to double.
The scraper takes URL of the topic and then it scrapes 5-10 sources (I don't want too many) and information about them. In the future, I could easily automate it in a way that it also picks the topics removing all work from me but I rather do it manually first to make sure the topics are interesting
The scraper is almost ready. I will continue tomorrow. I believe that I'm available to launch the website on Product Hunt the day after tomorrow.
I want to highlight that this is what always happens. I was ready to take the screenshots and launch the website tomorrow but then I realize something I didn't notice before which causes one extra day. This is not the same as building more and more features which might look very similar. Be careful if one day turns into two and two days turn into three. Then you are probably adding features, not fixing the mistakes you didn't think in the first place.
19th
10 am - Scraper is ready. I can give title, summary, and URL. Then it extracts all the other data automatically and saves it to the database.
I first built the developer tool using ReactJS. Then I started to add news and I realize that it wasn't actually the best solution. The original work was waste of time but I don't regret it. I don't think I could have predicted that it's too hard to use without first really using it. It's just important to have some time for this kind of mistakes because it's inevitable to have them. "Move fast and break things"
11 am - I ate and did some small fixes to the website. Those started to get too small 🦠, the kind of things you don't want to waste time doing when building MVP.
1 pm - I did some more fixes mostly to scraper to reduce probability 🎲 of mistakes. I cleaned the database and added some news to prepare screenshots for Product Hunt.
I was taking screenshots for Product Hunt but then I notice this 😱
The source icons look bad. Really bad. It's pretty small part of the whole website but I want to fix it as I have the whole day (these are the words people use when they start building more and more stuff to MVP 🙂).
4 pm - I'm back after a few hour break 🎾. I just notice that mobile feed is also looking bad so I need to improve it.
First I start with the source problem. I tried to look if there is some easy to way to get more high resolution photos. I couldn't find any solution to that. The other problem with icons is that some are rectangular which doesn't fit into the square space I have. Because of these problems I decided to not use the icons but show text that says the source name.
Maybe good icons would have looked better but it's fine like this.
So this is the mobile problem
Looks better
The website is ready! 👏
7 pm - I wrote description, comment, and tag line for Product Hunt. I took some screenshots too.
I checked analytics to make sure it's working as it might be the most important part. I actually realize that I had setup it incorrectly that made it to track nothing. I'm glad, I looked it once more because I was available to fix it. 😌
It's good practice to have some path of actions you follow every time and in the end, repeating it as many times as it takes to do it once without any problems.
8 pm - The database is now cleaned and I started to fill it with some news for tomorrow. The idea is that the feed changes everyday 12 am PST which is here 🌍 10 am. This gives me time at morning to prepare the feed before users see it. The idea is to schedule Product Hunt launch tomorrow morning to happen a few minutes after 12 am PST when the new day changes also on Product Hunt ⬜🐻🟥.
I'm a little bit freak out 😰 at this point. I notice some small UI things but I'm not going to touch anything that small at this point because it might cause more work and then I might not be available to publish it tomorrow. It's really common that people start to do stuff in the last minute and make mistakes or just make the whole product more complex causing more problems which postpones the launch.
20th
9.00 am - I scheduled Product Hunt launch for 10.30 am (12 am PST).
"YouTube like comments for 10 trending news of the day" is the tag line I'm going to use. Originally I thought something without the YouTube thing but then I started to think that do people get the idea if I just say that they can comment news. At least I right away like the idea when I compare it to YouTube comments that I enjoy reading. It might be confusing but I rather put it there and take the risk of being confusing than leave it boring.
10.30 am - Adding news, fake comments, and fake votes took longer than I expected but I'm still on time. Choosing 10 news was a little bit difficult. I wish I have promise to do only 5 but it's too late to change anything.
I also realize that right now analytics is not tracking what news people read but are they reading news at all 😐. I of course want to be available to track the news reading to see what kind of news they like the most but it's too late to do it for the launch at this point. I try to do it in a few hours.
Right now I'm just going to comment, reply, and vote to make it look like there is traffic on the website. I'm also going to promote the Product Hunt launch in a few places. I'm not sure how important it is but I launched something recently and it got only 3 upvotes so I don't want that it happens again. I believe that the feed is designed in a way that good products can get noticed even without extra promotion.
11 am - I ate and came back. Doesn't look good. Just one up vote on Product Hunt and it's from me. I will add a few comments to all of the news and try to promote the posting after that.
Tweeted, send messages to friends, and announced it on a few Slack channels. I also posted it to Hacker News.
1 pm - 3 upvotes. Two are from me and one is from friend. Doesn't look great but then in the other hand it's mid night in PST. I do some other stuff as there is nothing I feel important in this project.
4 pm - 4 upvotes. One from stranger. 4 users have visited based on Google Analytics.
21st
9.30 am - There are still 4 upvotes and at this point it doesn't any more matter as there is a new feed. I got 7 visits to the website. I notice that the email sign up wasn't working which is a little bit annoying.
I added 10 news for today's feed. I start to think that I need to automate this process as I'm not interested of doing it more than maybe one more day. This way I also remove all the manual work so I can just let the website live its own life and move to something else. I don't really think the website will explode 💥 on its own but I keep it running 🏃 for this article and just in case I want to show it to someone.
22nd
One user visited during yesterday and it was me 😶. I automated news scraping script in a way that I don't need to do anything 🤖.
23rd
Yesterday I had 3 visitors. One was me, one from Slovenia 🇸🇮, and one from United States🇺🇸. American left the page right away while Slovenian spent 8 seconds 🙏.
26th
There haven't been any visitors since 22nd.
I didn't focus this much since the last update. I even try to avoid thinking about this 🙈 because the way it ended annoys me. I didn't get as many users to look the idea on Product Hunt as I originally expected and I don't have any solution to get more people to look at it. I could spend a week or more trying to get people try it but I don't really want because the idea is not something I really believe but did just for testing purpose.
Conclusion
I overestimated the amount of people who would look the idea on Product Hunt. There is no feature that would allow me to see how many people saw the listing. Based on the amount of people who visited the website, I couldn't do much conclusions because the amount is not statistically significant.
I assumed that Product Hunt is working in a way that good products get noticed without extra promotion. Now I'm guessing that the sorting algorithm is too much bias towards the upvotes instead of giving all the products proper amount of user views before deciding if it's bad or good.
It might be also that my listing wasn't good enough or maybe in theory the idea doesn't sound very good.
Did this really proved that people don't like reading YouTube like comments about news? The data that I collected shows me that people didn't like this idea. Is it that they don't like reading comments about news? Maybe, I can't say it for sure. The variable I didn't remove with this MVP was that people couldn't use their current source of news but I required them to use 10 news I chose.
Now I start to realize that maybe the comments on YouTube aren't as important as I thought. If the video is not good, no one will read the comments. The same might have happen here. I expected that the comments could be the main feature but it is actually extra feature that makes the product better.
Based on all this I would say that there is possibility that this hypothesis was incorrect. I couldn't see any proof that people would like this idea. That is why I conclude this by saying that this hypothesis was incorrect.
Someone might say that it's really hard to make any predictions from the data I got, and I agree. But I'm not trying to come up a vaccine that needs to be tested perfectly. Startup hypothesis should be tested with the right balance of efficiency and accuracy. I believe that I learned something about the things I wanted from this quick, 6 day, experience. I might have missed something important that might have give the opposite conclusion but I don't see it the worth of effort to continue.
One important thing that I learned from Lean Startup is that you should never do mistakes and then just say "at least we learned something". It might be the most common excuse to fail. There should always be a question you try to answer not that you come up questions after you have failed.
There are two things that might have helped to do more accurate predictions: user interviews and getting more people. User interviews are kind of interviews where you talk with people who possibly might use the product. You don't ask directly what they think about it but things like what news sources they use, why they use them, are there comments, do they talk about the news with their friends, etc.
I hope you learned something new by reading this lengthy article. If you realized something I probably missed, please let me know so I can learn. The goal of writing this wasn't just to teach people but learn and then also allow me to later review it as I have learned more. I also notice that it helps me to analyze my decisions better and avoid common mistakes like building MVP too far. I'm going to continue documenting in the future projects as it seems to be worth of the time.
Follow me on Twitter 🥺
@RealLankinen
Originally published: https://www.notion.so/lankinen/Testing-Startup-Idea-in-6-Days-bbdddcc18e6e46478ab55786d162f83b
Top comments (0)