Exploring AWS from the terminal with VisiData

VisiData for Exploratory DevOps

The folks behind VisiData recently released version 2.0.

VisiData is a tool I wish I found sooner - a tool for quickly exploring many types of files from your terminal. To celebrate its 2.0 release I want to help show some ways it's helped me. Many of VisiData's uses are well-covered by tutorials, video case studies and documentation. I'm not a data scientist though, so I'll start somewhere totally different. Let's explore S3!

Exploring S3 from the Terminal

I often want to quickly peek at data files in S3. With VisiData I can browse a bucket in a way that recalls terminal file explorers like Midnight Commander. I can open files (CSV, deeply nested JSON, Parquet...) directly as new sheets in VisiData's tabular interface.

Here's a demo that uses VisiData to explore weather data in a versioned S3 bucket:

Extensible with Python

Most of the functionality in the demo above comes out of the box with VisiData. The only work I needed to do was create a plugin that tells VisiData how to speak S3. And to be fair, that's mostly a wrapper around the excellent s3fs library! VisiData 2.0 includes a Plugin Author's Guide that can help you extend VisiData as you see fit.

VisiData as a Piece of the Puzzle

Sometimes if I'm on a screenshare, people notice VisiData and ask about it. The strange part is that as much as I love VisiData, it doesn't actually replace other tools I use. It's replaced specific uses for sure...

• jq: I still use and love it, but VisiData feels smoother for interactive exploration.
• glom: Rather than inspecting/exploring data with glom first, I explore with VisiData and potentially use glom to define repeatable transforms.

VisiData also works very well as a side "quick view" tool to consume output from other tools I use regularly:

• Terraform: Add a few lines of Python to your ~/.visidatarc file to automatically open .tfstate files as JSON.
• Cloud Custodian: Open local policy output files or use S3 glob matching to check output for a range of dates.
• AWS CLI: Pipe its JSON output to vd -f json.

Some of those use cases deserve their own post! The gist is that if you feel at home in the terminal and need a tool for ad-hoc data exploration and analysis... VisiData is probably worth a peek. It'll do as much or as little as you need, and let you get on with your life.