Jenni B for Honeycomb.io

Posted on Nov 2, 2020 • Originally published at honeycomb.io

Spread the Love: Appreciating Our Pollinators Community

#observability #opensource #community #honeycomb

Have you heard the buzz about observability with Honeycomb 🐝? It’s the best tool on the market for observing your systems in real time to reduce toil and delight users. But don’t listen to us, listen to our buzzing community of “Pollinators”--this blog post is dedicated to them 💖

An invite is sent to our shared Slack workspace “Pollinators” to everyone who signs up for Honeycomb. As a customer advocate at Honeycomb, my goal is to help you find the right path at Honeycomb, and Pollinators is the welcome wagon that helps me help you 🌟

Whether you’re kicking the tires, you’re new to observability, or your team is actively evaluating Honeycomb, the first step is signing up for Honeycomb and jumping into what your new world with observability can look like. Sign up for free at honeycomb.io/signup or reach out to our sales team for an Enterprise trial, and check the email you signed up with for the Pollinators Slack invitation. (Or follow this direct link to request an invite 😉)

Who’s in Pollinators? Well, hopefully you soon 😊! The Hive’s rallying cry is Honeycomb Users and Honeycombers helping Honeycomb Users. All Honeycombers are in the Slack workspace. Our superstar engineers, developer advocates, and customer success team are in there, ready to answer your questions! Who else? Our customers, fellow trial-ers, and folks in our free tier. So if you have a question about how to get started, what to try with your stack, or questions about topics like SLOs (Service Level Objectives), tracing, events, and more, our community is here to help 🐝💖✨

Thank you to everyone who actively participates in Pollinators. Every day, we see unbounded kindness, curiosity, and helpfulness. And the Bees from Honeycomb are so thankful to have folks in our community discuss not only their technical questions, but their philosophical approaches to observability.

With love,

Jenni & the Bees

Some Recent Highlights

Getting Started with a Little Help from Our Friends

$Nicholas Moorcroft (Zappi.io) 9:01 AM Hey again! I'm still on manual tracing and I'd now like to get my traces to propagate between my SPA and back-end Rails apps. I may have missed something but I don't see anything in the docs about cross-app propagation. Scenario: I'm wondering if there is a specific honeycomb header I can manually attach to my back-end requests? Example code: trace = { span_id: 123, trace_id: 444 } xhr.setRequestHeader('X-Honeycomb-Trace', trace) Where my Rails beeline will automatically check for the header, extract the trace and then use the span_id 123 as the parent for the next "unit of work" done by my Rails app.$

Nicholas Moorcroft, Software/Observability Engineer at Zappi, is based in South Africa--thankfully, our community is timezone-agnostic with Pollinators all over the globe! Honeycomb’s product and community are deeply ingrained with the philosophy that learning from each other makes the journey that much better. With a little help from our friends, Nicholas got his tracing with the Ruby-Beeline, which is automatic instrumentation for Ruby applications, up and running at the start of his Honeycomb evaluation with our free-tier.

In Nicholas’ own words:

“What was really helpful at that point was when Pedro from carwow pointed out a part of the ruby-beeline code which seemed relevant. It provided some direction for me and I was able to find the actual part of the code I was interested in. The thing I appreciate most about a Slack approach to community and engagement is that it can, and has, brought together a group of active users. Both technical and non-technical and allowed us to converse with one another. This is awesome in an age where companies can have varied tech stacks and face similar, but different, challenges when instrumenting new technologies. Additionally we get the benefits of having a direct line of communication with the business itself, be that for feedback or just general chatter.”

Zappi is an ad and innovation testing platform built by brands for brands, and is now one of Honeycomb’s newest Pro-tier customers 👋Welcome to the hive, Zappi!

Teamwork Makes the Dream Work - Discover Errors, Fix the Problem, Share the Knowledge, & Share the Love 💖

Ryan Ordway, DevOps Engineer at Oregon State University Libraries & Press (OSULP), gives Honeycomb some sugar in the #love channel where Ryan was able to show his team that their chain certificate for their API was expired.

In Ryan’s own words:

“It ended up not being a MAJOR issue because the API call that was failing can fail without preventing the rest of the app from loading, but it was causing a little bit of slowness. But having the API call instrumented (with the beeline, since it instruments external HTTP calls) allowed us to see the errors and alert the team on campus that manages that API so they could fix it.”

Ryan’s team is using Honeycomb Triggers for these particular issues--Triggers let you receive notification when your data in Honeycomb crosses the thresholds you configure, you can think of them to be like alerts.

Ryan has also been a guest contributor to the Honeycomb Blog, see his piece here on how adopting observability at OSULP & Honeycomb: “The most important thing for us is saving money. I’ve finished migrating all of our tracing from our other vendor to Honeycomb, using the same little Honeycomb sandbox from when I started this adventure. Even better, I did it without having to pay more money. That’s huge!”

In the future, Ryan hopes his small but mighty team will be measuring their Service Level Indicators (SLIs) and SLOs with Honeycomb 🐝💪 Thank you Ryan for all of your contributions to our community, content, and product, we wouldn’t be the same without you!

Test in Production, Watch Deploys in Real-Time, Share Context Across Teams

Honeycomb’s rallying cry has always been that we test in production, and much of our community does the same. Matt Button, Senior Infrastructure Engineer at Geckoboard, discovered something peculiar going on in the backend and the team was able to deploy the fix & see the results in real-time 👀

In Matt’s own words:

“We were using some trace waterfalls to show a frontend engineer how the work the backend did differs when you change certain arguments in the request. While looking at one of the traces we noticed that one of the DB queries was actually taking up quite a large chunk of the request time, even though it should’ve been one of the fastest in that request. When we looked into it we realised that query wasn’t using the indexes we’d defined for it, which meant the time to execute that query could vary a fair bit, as seen in the “before” section of that pic. It turns out the id field was never going to have mixed-case characters, so the query didn’t need to filter on UPPER(id). That was the fix we rolled out in the screenshot (above).”

Having full-stack visibility is crucial for being able to ask any question of your system.

Geckoboard built their own system health dashboards and had been using standard log search tools since the time when log search was new and shiny. But as their business has scaled up, so did their need to understand who is affected by a given issue, and Honeycomb helped them start connecting the dots.

Read more about Geckoboard’s experiences with Honeycomb:

Geckoboard Case Study: Geckoboard Gets the Context They Need to Keep Customers Happy Without Breaking the Bank
Honeycomb for incident response, featuring Matt Button: Take huge leaps with Honeycomb for Incident Response

Thank you Matt for your endless contributions to Honeycomb & the Pollinators community!

Sampling: One of Life’s Unavoidable Truths - Refine Your Data Without Losing Fidelity

Sampling is like eating your vegetables - you have to do it or else you’ll be in pain later 🥦💪 but also like vegetables, sampling is unique to your system, different depending on what you want, oddly shaped, and (may) give you gas 😂 depending on who you ask. Benjamin Franklin missed the third unavoidable truth in life, so I’ll revise it here: “in this world, nothing is certain but sampling, death, and taxes”.

For teams running workloads at scale, the question becomes: are all of my events worth keeping? How can I reduce overall event volume while maintaining fidelity? Sampling is a complex science and intuitive art, just ask your local statistician, or any engineer trying to find a needle in a haystack.

The best resource when trying to learn new techniques is to learn from each other:

Molly Struve, Lead Site Reliability Engineer at dev.to, shares her Redis & SQL Sampler with Pollinators as a template for people to learn from! Honeycomb has partnered with the Forem team to use their open-source production code to help pay it forward for the community. You can read about their Honeycomb journey in our HoneyBytes series, including the latest sampling post where we walk through the very implementation Molly shared above!

In Molly’s own words:

“The project I work on, Forem is a completely open source platform which makes it super easy to share our Honeycomb integration with others. Always happy to help where I can. Honeycomb is still pretty young so I think it’s important to have example projects that people can use as templates for their own setups”

Thank you for your dedication to educating others Molly!

Additionally, Marco Rogers, Senior Software Engineer at Mode Analytics, shares his approach to dynamic sampling in the Open Source Ruby Beeline SDK Channel and what ensues after is an awesome 🤩 exchange between users and Honeycomb collaborating with tips, tricks, and sampling kickflips.

In Marco’s own words:

“We needed to control the overall volume of events, but we wanted to keep every single thing that might indicate a problem. So dynamic sampling for us was about how to identify a successful (boring) event vs one that might have useful debugging data on it. Turns out that’s not as easy as throwing away all of the 200 status codes.”

Thank you for all of your contributions to Pollinators, Marco! The new-bees have lots to learn from you!

At Honeycomb, we recognize how important & tricky it is to sample correctly. Below are some resources (like the ones mentioned in Marco’s message) to help you understand and apply the sampling method that works for you.

HoneyByte: Get a Taste for Sampling blog post (the one about dev.to!)
Getting At The Good Stuff: How To Sample Traces in Honeycomb blog post
Honeycomb Docs: Best Practices for Dynamic Sampling
Refining Systems Data without Losing Fidelity, talk by Principal Developer Advocate Liz-Fong Jones

System Optimization - Find Your Bottlenecks

In distributed systems, small inefficiencies can create performance bottlenecks at scale. Pedro Chambino at carwow was able to observe some experimental optimizations with Honeycomb--again, testing in production!

Plus, seeing examples of how other people use Honeycomb can help you get more out of the tool. In Pedro’s own words:

“I think it has been more about being connected with the community and being able to learn about new features/changes to Honeycomb and how other people are using them. This was specially true with sampling, heatmap, and SLOs. Also, learning how we can use Honeycomb in more contexts, e.g. buildevents and airflow.”

Taking It to the Tweets

We love when the community takes to Twitter to share their Honeycomb experiences with the greater tech community. Here’s a fun one from Julian Simioni, Co-Founder of Geocode Earth:

Follow @honeycombio on Twitter, there’s plenty more where that came from. And tag us in your own stories!

Shout Out to Our Community of Contributors

While we’re showing our appreciation, check out our community of contributors. This is only going back to December 2019! When you upstream your improvements into our OSS codebases, it makes those changes available to the rest of the user community. Help us help you 💞

Go Beeline

Ruby Beeline