Browsing The Web With Crystal

If you've been in the web development industry for any decent amount of time it's likely you've heard mention of tools like Selenium, WebDriver, and Puppeteer. These are mainly touted as "testing automation frameworks" because they enable developers to automate the testing of web apps by saying "go here", "click this", "take a screenshot and compare it against a previous screenshot and make sure nothing's changed". These tools are also helpful with web scraping, as they allow you to see web pages exactly as the browser sees them. This means JavaScript rendered content is scrapable!

To date Selenium is probably the most popular of these tools. It's used everywhere, from testing environments and CI's to web scrapers and pentesting tools. It is extremely powerful, but has one major drawback. Java.

Anyone who's used written or used a Java application knows that it has one major drawback. Memory usage. Java applications are heavy, which limits their usefulness on systems with less resources and containerized applications. This doesn't stop people from trying, but why use extra resources if you don't have to?

So, with speed and memory efficiency in mind, I set about recently to devise my own solution for the Crystal ecosystem; a solution I decided to call Marionette after the Firefox Marionette protocol.

Exploring Marionette

Of course before wrapping any kind of API you have to do some research. Unfortunately, Marionette is not super well documented. Marionette and WebDriver share a lot in terms of methods and functionality, with Marionette building on top of the functionality that the WebDriver protocol provides.

Naturally my first Google search was for "firefox marionette protocol", which led me to this page which describes the protocol, but not in near as much detail as I'd hope. That being said, it does at least describe the format that a message must take when being sent to Firefox. Unfortunately after browsing the whole Marionette section of that site I didn't find any information on what commands the Marionette protocol accepted. So I continued my search.

Something I had seen in the Marionette documentation was mention of a Python client. Now I make it very well known to people I talk to my extreme distaste for all things Python, but if it could help me figure out what methods the Marionette protocol had available it would be worth it. So I followed a link to their reference client and dove into the Python code (which I was happy to find was actually very well written).

After a little bit of digging I found what I was looking for! This file includes most of the methods that the Python client uses for interacting with Marionette,including the names of the methods and the format in which messages are sent. And so began the process of building my own client.

Building the Client

The main client functionality was fairly straightforward, seeing as I had a fully functioning Python client to work off of. I also managed to find a client written in Go that was very similar, but not quite as powerful as the Python client. I still have never managed to find a list of RPC commands that Marionette accepts, but between those two clients I was able to build my own fairly easily.

Now this project wouldn't have been possible, at least not in the amount of time I've been able to get it working, without the help of NeuraLegion, my employer who is sponsoring its development.

NeuraLegion is an application security company with a SASS AIAST solution powered by AI. Or, for people who don't speak penetration tester, a company that tests the security of websites and other applications using software that's powered by artificial intelligence. That software has to be capable of browsing the web in the same way as you or I, but it also needs to be able to do things you normally don't do in every day web browsing, such as modify HTTP headers, send attack payloads in a POST request to an open API route, etc. Because of this, just a standard Selenium type client wasn't good enough.

We needed a proxy.

Building the Proxy

There are two main types of proxy: a forward proxy which is used to forward outgoing requests from a private network or intranet to the Internet, usually through a firewall, and a reverse proxy which retrieves resources on behalf of a client from one or more servers; these resources are then returned to the client, appearing as if they originated from the proxy server itself.

Firefox, and thereby marionette, has built in support for forward proxies. All you have to do is provide it with a address for the HTTP proxy, HTTPS proxy, and potentially FTP and SOCKS proxies and it will tunnel all requests through those before hitting the browser. Unfortunately this requires a separate proxy for each, and tunneling HTTPS requests turns out to be much more difficult. So it was decided that a reverse proxy would make more sense.

I won't go into full detail about how the reverse proxy works, if you want to check out the code you can find it here. Basically what it all amounts to is this:

When Marionette is launched with the extended option set to true a Proxy object is instantiated and the server is launched. The Browser#goto method then forwards all navigation requests to the proxy and tells it what page to fetch. The proxy fetches the resource and does a little gsub magic on the page's content, replacing all internal links with a link to the proxy server instead. All internal resources are then tunneled though the proxy as they're fetched, allowing us to potentially be able to modify javascript, images, etc. before they reach the page. Everything is then packaged back up and returned to the browser as if nothing had happened.

Basically what this means is that you can do whatever you want with pages before the browser even knows they've been touched, which can be very powerful for AppSec.

Using It

Ok so I've done enough boring you with how Marionette works under the hood, let's look at how you actually use it.

First, in case it wasn't obvious or if you just jumped down to this section, Marionette requires Crystal to run. So if you don't have Crystal installed you may want to go download it if you intend to follow along. You will also need Firefox installed.

The first thing you'll need to do is fire up your terminal and create a new Crystal project.

crystal init app browser
cd browser

Then add Marionette as a dependency in your shard.yml file.

dependencies:
  marionette:
    github: watzon/marionette
    version: 0.1.0

Now run shards install in your terminal and Marionette should be installed to the lib directory.

Now let's open up src/browser.cr and add the following content:

require "marionette"

Marionette.launch do
  goto("https://dev.to")
  puts title
  puts url
end

Make sure Firefox is not currently running, if it is this won't work since Firefox only allows one instance at a time. Now in your terminal run

crystal run ./src/browser.cr

After a few seconds you should see a bunch of debug statements in blue and in white

DEV Community 👩‍💻👨‍💻
https://dev.to/

Congrats! It worked! But that's not even close to all Marionette has to offer. Why don't we do something fun, like take a screenshot.

require "marionette"

Marionette.launch do
  goto("https://dev.to")
  save_screenshot("dev.to.jpg", full: false)
end

As of the time of writing, this is what "dev.to.jpg" looks like:

The full: false option allowed us to just capture what's in the viewport. If you didn't have that option set to true you would end up with a very large image.

For our last little test, let's visit a bunch of sites and then export a HAR file. A HAR file, for those that aren't aware, is a JSON-formatted archive file format for logging of a web browser's interaction with a site. Basically it is able to store all the details about a page load.

require "marionette"

Marionette.launch do
  goto("https://www.google.com")
  goto("https://neuralegion.com")
  goto("https://watzon.tech")
  export_har("multisite.har")
end

After this finishes running you should have a file called multisite.har in your current working directory. You can test that it worked correctly by using Google's HAR Analyzer.

I have tried to document Marionette extremely well in the readme and in the official documentation, so if you're interested in the project please go take a look. Contributions are always accepted, as are suggestions.

Thanks for reading this. Please don’t forget to hit one of the the Ego Booster buttons (personally I like the unicorn), and if you feel so inclined share this to social media. If you share to twitter be sure to tag me at @_watzon.

Some helpful links:
https://crystal-lang.org/
https://github.com/kostya/benchmarks
https://github.com/kostya/crystal-benchmarks-game
https://github.com/crystal-lang/crystal

Find me online:
https://medium.com/@watzon
https://twitter.com/_watzon
https://github.com/watzon
https://watzon.tech