In this post, I'd like to introduce crawly which is an elixir library for crawling websites.
My idea for this post is to be a quick introduction to how we can use crawly. For this little example we're going to extract the latest posts titles on our website and write it to a file, so let's do it.
First of all, we need elixir installed, in case you don't have it installed you can check this guide.
Once we have elixir installed let's create our elixir application with built-in supervisor
mix new crawler --sup
In order to add the crawly dependencies to our project, we are going to change the deps function in the file mix.exs and it should look like this
defp deps do
[
{:crawly, "~> 0.10.0"},
{:floki, "~> 0.26.0"} # used to parse html
]
end
We need to install the dependencies that we just added running the command below
mix deps.get
Let's create a spider file lib/crawler/blog_spider.ex that is going to make a request to our blog, query the HTML response to get the post titles, and then returns a ParsedItem which contains items and requests. We are not going to leave requests as an empty list to keep it simple.
defmodule BlogSpider do
use Crawly.Spider
@impl Crawly.Spider
def base_url(), do: "https://www.codegram.com"
@impl Crawly.Spider
def init(), do: [start_urls: ["https://www.codegram.com/blog"]] # urls that are going to be parsed
@impl Crawly.Spider
def parse_item(response) do
{:ok, document} = Floki.parse_document(response.body)
items =
document
|> Floki.find("h5.card-content__title") # query h5 elements with class card-content__title
|> Enum.map(&Floki.text/1)
|> Enum.map(fn title -> %{title: title} end)
%Crawly.ParsedItem{items: items, requests: []}
end
end
Now that we have our spider created it would be nice to save what we're extracting into some file. To do this we can use a pipeline provided by crawly called Crawly.Pipelines.WriteToFile. For that, we need a config folder and a config.exs file:
mkdir config # creates the config directory
touch config/config.exs # creates an empty file called config.exs inside the config folder
mkdir -p priv/output # creates output folder inside priv where we are going to store our files
Now let's create the configuration to save the response from our spider into a file.
use Mix.Config
config :crawly,
pipelines: [
Crawly.Pipelines.JSONEncoder, # encode each item into json
{Crawly.Pipelines.WriteToFile, folder: "priv/output/", extension: "jl"} # stores the items into a file inside the folder specified
]
Now that we are good to go, we can open the elixir repl
iex -S mix
And then we can execute our spider
Crawly.Engine.start_spider(BlogSpider)
The spider is going to be executed by a supervisor and then we should see a new file inside priv/output folder. In my case the lasts posts showing in the first page are
{"title":"\"High tech, high touch\": A communication toolkit for virtual team"}
{"title":"My learning experience in a fully remote company as a Junior Developer"}
{"title":"Finding similar documents with transformers"}
{"title":"UX… What?"}
{"title":"Slice Machine from Prismic"}
{"title":"Stop (ab)using z-index"}
{"title":"Angular for Junior Backend Devs"}
{"title":"Jumping into the world of UX 🦄"}
{"title":"Gettin' jiggy wit' Git - Part 1"}
This is just a simple example of what is possible to do using crawly. I hope you enjoyed this introduction and remember to be responsible when extracting data from websites.
Top comments (1)
This blog provides a fantastic introduction to using crawly, an elixir library for website crawling. It's great to see how straightforward it is to extract data and perform tasks like saving it to a file using crawly's features. Platforms like Crawlbase can complement crawly's functionality seamlessly.