Don't get me wrong. I appreciate visual materials, I think that automatisation of repetitive tasks is a must and I strongly recommend you to cover your production servers with all kinds of monitors.
But making dashboards for top management was my most hateful job as a product analyst. And I can tell you exactly what's wrong with it, and how you can navigate through it, if you absolutely have to.
Problem 1: It's not what it looks like
Let's say that we want to see how many users we get every day. What can be easier, right? It's just all unique users' ids from daily log. Or is it?
Well, not quite. What about those people who don't allow cookies' usage and they don't have an id? What about those people who have some security add-ons installed in their browser, who have multiple different ids throughout a day? What about bots? Search engine robots? People with browser, that downloads all the tabs in background at every restart?
So, you make workarounds for that issues, you filter out UserAgents for search engines, you filter out too frequent requests for bots, etc. The resulting chart doesn't show the number of users, it shows the next best thing — estimation for that number.
The thing about data is that it never stays the same (unlike your shipped code). Maybe there is a release of new GDPR guidelines, maybe an intern dropped a line of code which was assigning an id to a user. One way or another, the day would come when the data changes and your chart immediately gets messed up.
How to deal with it
- Stick to the most unambiguous values that you can find. It's fine to make chart of the "number of successful billing transactions" or "number of clicks on an ad banner".
- Keep all calculations for the charts as simple as possible. Remember that every additional level of logic that you add to your code, would double the trouble in a long run
- You don't have to stick to the data available at the moment. Talk to the dev team, come up with extra events to log, which will help you to avoid unnecessary complexity in your algorithm.
- When there's no way to avoid having the gap between log events and high-level data to show on a chart, make it very clear, that it's not some magical "true value". That it's an estimation, and it will inevitably get messed up. Not because you didn't see it coming, but because the management asked for too abstract of a value, that doesn't have an unambiguous representation in logs.
What's problem #2? We'll talk about it next time.
Top comments (9)
I have a true love/hate relationship with dashboards. I am a huge data nerd, but really struggle with making things look pretty. Unfortunately, non-programmers don't care much for how cool the code behind the dashboard is, just how its displayed.
It's also nice to see people working through the same issues I am when it comes to dashboards, so thanks for your perspective.
I feel you. I like making things look pretty, but making a dashboard look pretty is quite a challenge. Especially when you are limited by tools approved by security office in corporations, or by budget and quality of open source tools in startups.
Either way, you either waste even more time to implement it yourself, or you have an ugly dashboard. At least in most of the companies I worked in, the reality looks like that.
I loved the part you said data never stays the same unlike the shipped code :)) it's people being afraid of releasing new code so they ship less often
Oh yeah, thanks!
I can give you another perspective for this: data analysts don't have code review, solid tests or QA engineers, we have to do it all by ourselves, every man for himself. So it makes sense to ship less.
But it backfires either way :)
Sure, that's right. But I was thinking a bit more general.
In my opinion, the brilliance of the sentence is how it humorously describes one of the biggest problems with software so simply in a few words! Software is fragile and it's important to understand.
Data doesn't stay the same (like users, technologies, vulnerabilities, regulations, etc.) so the code goes obsolete and broken.
Oh, didn’t think of that, thank you.
That an inspiring way to read that sentence.
Interesting perspective. I have not thought about this before.
These days, I am pushing an idea of "dashboard-less" decision making. How can the problem be solved upfront.
For example, "number of successful billing transactions" or "number of clicks on an ad banner" - we ask the why. Why do you want to know successful billing transactions? Is it to track errors? Like Billing Transactions failed/success? OR to track it against a KPI?
If it's against a KPI then only highlight this metric if it's falls "near fail" condition, and not before. Not to clutter their day... sometimes we don't even need dashboards but just alerts/emails on key events.
This is only valid for Strategic/Tactical type systems, and not analytical which for sure needs some UI to assess the data. Also, not easy to sell this idea as everyone wants a "dashboard" because they want a "dashboard" :D
Absolutely agree.
I think we don’t need most of those charts and it’s just a cargo cult. I’m so happy to hear, that someone sees it the same way.
And yeah, selling this idea might be hard especially when you are a junior analyst talking to top management.