Today is my first day working with the incredible team that runs this amazing website! I am happy to announce that I have joined the DEV team as a ...
For further actions, you may consider blocking this person and/or reporting abuse
The most painful part of the DEV.to open-source codebase is its dependence on Alogia.
So if the mention of Elastisearch is to replace that service that would ease development and would be quite welcomed.
I wonder if having an SRE will drive the platform away from Heroku to just directly utilize cloud computing services such as AWS. Since I would think an SRE would want to have fine-tuned control over configuration since Heroku can certainly be stifling.
The Upgrade to Postgres 11 made me think there is some clunkiness with upgrading where if the infrastructure was directly managed on AWS my users would never need to be warned of such an upgrade and no interruption would be felt.
Congratz Molly.
These are very good observations and definitely enters into where and why we need help.
That is indeed a thing we want to do. Beyond the pain of having a dependency like Algolia, search has the capacity to become more central to what we do and moving to a leading open source provider seems ideal. So yeah, Molly is here to get stuff done along those lines.
Hopefully, DEV.to makes use of Amazon Elasticsearch. I have a cluster running on ExamPro so I can use full-text search on my large bank of practice exam questions.
Elasticsearch is notorious for going bonkers and using 99.99% of CPU. One of the few times it better to use a managed service is Elasticsearch since it already fined-tuned on AWS to avoid having to put in the crazy amount of people-power debugging cluster configuration issues.
i googled ExamPro out of curiosity about what sort of stack youre running and all I can find is stuff related to OB-GYN... is that accurate?
Exampro.co. You may be mistaking us for the.co.ukor the.comElasticsearch is one of those tools that is very easy to get started with but if you want to use it at any sort of scale you really have to understand how it works otherwise it is VERY easy to crash it. When we started using it at Kenna we had 500 million docs and were processing a few million a day and we were maxing out our 20 node cluster. By changing our data architecture and how we ran queries we were able to take that exact same cluster and store 5 billion docs in it and process 300+ million of them a day.
@molly_struve would love to see a detailed writeup on ElasticSearch. Do you have any resources you can point us to, for an absolute beginner on the topic?
Elasticsearch has great getting started docs. I have been meaning to write a beginner post on it but for now all I have are a couple scaling posts:
Scaling Elasticsearch Part 1: How to Speed Up Indexing
Molly Struve γ» Apr 3 γ» 6 min read
Thanks @molly_struve ! I'm following you and will look for a beginner post, hopefully sooner rather than later! I have content from a bunch of MySQL database tables I'd like to index (very much like dev.to, actually) and am doing my best to stay away from Algolia and not have to re-invent the wheel.
I found this and this but would love something dead-simple to understand and get started with.
I have plans to add search feature to one of my products...
I'm wondering what's painful about having a dependency on Algolia?
cc: @ben
Mostly because it's a proprietary product that AFAIK is not based on Elasticsearch, which means that you're tied to a service with less accessible expertise than self hosted or SaaS Elasticsearch based offer.
Regarding performance, benchmarks, features and everything else I don't know much about those differences yet, we'll obviously have to evaluate those if we consider switching, but that's applicable to any type of stack change.
Thanks ryhmes for the explanation.
I would like to mention those facts:
The current UX when searching in DEV is elegant, fast... just great (I barely switch to google trick
searchTerm site:dev.to).I remember once Ben indicated the software price factor is negligible.
Elasticsearch requires extra work (setup, provision, code... etc).
All that said, is Elasticsearch adding any specific value? Or am I missing something?
I think in that thread they were discussing about Heroku vs other PaaS/SaaS cloud solutions, Algolia is external to Heroku
I think the discussion is pretty similar...
Heroku vs. DigitalOcean = PasS vs. IaaS = ReadyToUse vs DIY
Algolia vs. SelfHosted Elasticsearch = SaaS vs. IaaS = ReadyToUse vs DIY
The real different discussion would be this one:
Algolia vs. SaaS Elasticsearch (like AWS) = SaaS vs. another SaaS = ReadyToUse vs ReadyToUse
Since you mentioned that specific point, is the price the main concern?
AWS has its own SaaS for search called Amazon CloudSearch
So really it would be
Algolia vs Amazon CloudSearch if you want to say = SaaS vs. another SaaS
Amazon Elasticsearch is just AWS managing the clusters for you so you have high availability, durability and security out of the box and don't have to worry runaway CPU.
If you need apples to apples, I would consider Elastic Site Search(previously known as Swifype). It is based on Elasticsearch, owned by the same company and has nice GUI and rank tweaking capabilities.
Managing Elasticsearch yourself is painful, I mostly struggled with version upgrades.
@andrewbrown
Yep that's what I meant by AWS (whether CloudSearch or AWS Elasticsearch)
Those must be all provided by Algolia too, is there anything else that would urge me to switch?
@buinauskas , the same question applies to Elastic Site Search (ofc in DEV context) π€
Obviously I lack a lot of details here. But if it was me, I would carefully measure switch from PaaS to anything that I'd have to manage myself (that's both infrastructure and index mappings). I'd rather focus building and tweaking features instead of maintaining infrastructure and index transitions.
I'd probably only switch if some of these start bothering me too much:
@yaser
For your own projects, there is certainly nothing wrong with Alogia or Elastic Site Search.
For an open-source project where the development environment needs to be replicated is a different story. By using Amazon Elasticsearch a CloudFormation template can be written which will provision all required cloud resources required to run the app in production.
By using Elasticsearch which is an open-source project the development environment can be dockerized.
The pain with Alogia is that you have to signup for an account and its set to a trial. So to continuously develop on DEV.to you have to keep opening trial accounts or you have to pay for Alogia.
AWS also has credits for open-source projects and in addition to has a free tier which is
750 hours per month of a t2.small.elasticsearch.
Now, why would you choose Elastic Search Site/Alogia over Amazon Elasticsearch?
Elastic Search Site and Alogia or going to definitely come with many conveniences and likely will lead to greater agility when developing.
AWS gets its superpowers when you buy into the whole AWS ecosystem since my many applications can be chained together. Elasticserach is commonly used in a pattern called ELK which is one button to deploy on AWS via a CloudFormation template.
Isn't this same for any kind of development environment?
It's also Algolia, not Alogia ; )
Also ELK is not a pattern, just a stack of applications.
To me, it feels either money goes to Algolia (or any other tool) vs. that same money on something that needs more maintenance.
@buinauskas
Yep, that's what I'm trying to verify since the very start of the discussion (let alone that DEV search feature might get broken during the switch).
@andrewbrown
Thank you for sharing your view!
Which is not the case with DEV, so I wouldn't really recommend the switch.
I agree with you... partially π
I checked their open-source offer, and it seems a fair deal: algolia.com/for-open-source/
They offer you their software for free if you are a non-profit, and DEV is a business.
For the open-source contributors, yes they should be using Algolia services for free. After all, a contributor is mostly not making money out of his contributions.
I think instead of the switch into another service, algolia team (@jesswest @r4ph_t ) might consider the idea of adding a monthly free-tier for developers (and / or providing an official mock docker image like this one).
Edit: I have mentioned the topic in their feedback section, topic link here.
This is of course a concern and your whole argument makes sense (though it might be a little biased towards embracing AWS as a cloud provider) but that's not the only factor in a decision, after all DEV is also a company.
If "tomorrow" the company goes bust because its employees spend 7 days a week managing infrastructure instead of developing the actual product nobody is going to be happy, even though the code is opensource and one could pick up from there :D
I think as all technological choices all aspects need to be considered, including your argument towards using opensource :)
BTW tying the product to AWS SaaS services could possibly be another type of vendor lockin (especially if you "buy in the whole ecosystem" as you mentioned), not more or less than having Algolia as a search engine. Same goes for CloudFormation instead of Terraform I reckon.
There's obviously a will about embracing open technologies and a better interoperability but finding the right balance between SaaS services and DIY is a struggle for every company at all stages...
We still haven't done any thorough evaluation on this, I'm afraid we don't have a satisfactory answer yet :)
Hey Molly, I'm really excited to see that you've joined DEV and I'm looking forward to all the things you gonna do and write here :) I'm a software developer with a tendency to DevOps and I'm really interested in learning more about SRE!
Congrats Molly! Looking forward to you sharing your SRE wisdom and experiences from inside DEV π€
Congratulations, Molly! This is so well deserved and an excellent match of skills and talent. Looking forward to seeing your contributions to the Dev Community in this new role! πͺπ»
I've never met you but enjoyed your last post Congratulations on your new role. Let your passion shine!!!
Congrats Molly! I should have suspected a move to Dev when you said you were leaving Kenna...
Holy cow! Congrats Molly!!!! I'm so excited for you.
This is so awesome! Congrats Molly! Iβm sure you will make a huge impact on the DEV platform and community in this role. Iβm looking forward to seeing everything you are able to build. Will you get to build out a team eventually?
That's the goal!!!
Super cool!
This is just great! I wish you success here at Dev and your contributions make this platform even better!
BTW, I don't know why but when you published you were leaving Kenna Security I suspected you'd go working at Dev. Never really bother to ask but my suspicious was true π and it's super awesome!
With the #hacktoberfest event, I started my first contribution to Dev. This is a great platform and I have already seen tons of interesting things in its codebase.
You working here now will make it even more interesting to read, learn, and contribute.
ππ½ππ½
PS: your dogs are super cute :)
Wohoo... Congrats Molly!
Great, now I want a corgi too ππΆ
Not going to lie they are the BEST dogs and super smart!
Congrats Molly!
We are so excited to have you on the team, Molly! ππ½
Glad you joined, big win for the community β€οΈ
Congratulations, Molly! I'm sure you'll be a great addition to the DEV team.
Are you kidding me? How it comes dev.to hire all that cool people.
Congrats!!
Wooooo so excited to have you on!!! πππππ
Congrats Molly! See you in the next RubyConf! :)
I already said on the Twitterz, but congrats again Molly! Your posts and community involvement have been inspiring and can't wait to see your work for Dev.to.
Congratulations and good luck!
We bootstrapped our SRE team this year, and it's been a whirlwind. Hope your path is another.
Congrats! Really good article. Do you have any books, video resources you recommend for getting into site reliability engineering?
Thank you!!!
The SRE Google book is great and honestly the only book I have read. Otherwise, I enjoy reading blog posts and articles as I come across them. Nothing stands out, I usually read posts based on my needs and projects at any given time.
Congratulations Molly! π I look forward to reading more of your posts and experiences with the DEV team.
Congrats Molly and I enjoyed reading your intro :) Your dogs are adorable!
You rock ; )
OMG this is so exciting!!!