Bug detective — When returning JSON goes wrong

As a developer, I spend a lot of time on fixing bugs. If you are a software developer, you can relate to this, right? I love to create new things and develop new features. However, there are a lot of days that most of my time is spent around debugging. Some days are really hard and you have to act like a detective to fix a bug. Those are your detective days. I think most developers have such days from time to time.

In this post, I will introduce a bug case and I will explain the process of fixing it. The case is based on a real problem that a coworker and I had to debug in the past, while working on a PHP codebase. I changed some variables and functions names to avoid sharing private code. This programming language has a lot of bad reputation because a lot of people wrote really bad code in it. But you can do that in any programming language, right? The focus of this post is not in PHP. You can keep reading it even if you never coded in that programing language before. The main focus is on the case itself and in the process of finding the cause. In the end, I will give you some tips to help you on those detective days.

Let’s just get started with the case!

The case

You work on a web application and one of the pages has a table, which is really important for your users to allow them to make use of your app. One day, you receive a lot of bug reports about this table not showing any data at all. Not a single row was there. Like any bug that is blocking your users, you have to fix it right away.

The first thing you try to do is to reproduce the behavior that your users described. You just do that and of course, they were right. There was no data at all. You check the browser’s developer tools and there is not a single error being thrown. You check your server’s logs and there are no errors there either.

You start investigating…

The investigation

Let’s ask the first question… How the table works and where its data comes from?

An HTTP request is triggered by some code in Javascript to an endpoint written in PHP, which is supposed to return a JSON array. That array contains several objects that will be used to “feed” the table’s content on the frontend.

Pretty simple, right?

What is the endpoint returning? Let’s check the response, maybe it is returning an empty array… But it’s not. The response is completely empty. Not an empty array. There is nothing in the response body but its status code is a 200 OK. Now it starts to become really weird. Not a single error and not an empty array. How is it possible that no error is thrown and you get an empty response body? Something really weird is happening on the server.

How the server endpoint’s code looks like?

<?php

$data = getDataFromDatabase();

echo json_encode($data);

?>

For those of you who never saw any PHP code before, the way you initialize a variable is like this:

$variableName = value;

Let’s break it down:

• getDataFromDatabase is a function in the project’s code base that sends a query to the database and returns an array of associative arrays (key-value structures, something like Javascript objects, in PHP).

• json_encode is a built-in function, provided by the programming language, that converts an associative array or an array of associative arrays to JSON.

• echo is a very well known PHP construct that just outputs one or more strings. In this case, it is outputting the $data array in a JSON format, because of the json_encode call. If everything goes well, you get a JSON array out of the $data variable.

At this point, I challenge you to take a small break and think of what can be causing this endpoint to return nothing.

Did you think about it?

You need to ask yourself… Is there any issue with the getDataFromDatabase function? Is there any error connecting to the database that is being caught and not being thrown to the caller?

Let’s try a top down approach. Which means, let’s start in the top level, which is the endpoint’s code in this case, if something is wrong here, you need to take a look inside the getDataFromDatabase function. Because this is the only function you can look at. json_encode is a PHP built-in function. It should be working properly. Let’s take a look at the $data variable’s content. To do that, you just need to call var_dump, which is a function that dumps the content of a given variable.

<?php

$data = getDataFromDatabase();

// Let's output the $data variable's content
var_dump($data);

echo json_encode($data);

?>

After this change, now your response will look weird because you are dumping the variable’s content, which is expected. When the var_dump executes, you can see an actual array with a lot of associative arrays inside. This getDataFromDatabase function seems to be working just fine, right?

Right now, the only explanation is that, something is wrong with calling the json_encode function. But it is a built-in function. It should behave as expected.

At this point you are thinking that I lied and this is all fake, which I really understand. But, as I told you in the beginning of this post, this case is based on a real problem that I had to debug. Something like this actually happened to me. I ended up asking myself…

“Is it possible that a built-in function like json_encode is not working properly?”

And also… WTF?

This is one of those problems that you spend many hours on it, trying to fix it and the more you dig into it, the more you say “WTF?”. You start to become really desperate and you start googling about your problem. Before doing that, let’s just look at the PHP website and search “json_encode”. There is something really interesting in the “Return values” section:

“Returns a JSON encoded string on success or FALSE on failure.”

Since the endpoint is returning nothing, the json_encode function must be failing and returning false. Let’s change the code to call var_dump to see what json_encode is returning:

<?php

$data = getDataFromDatabase();

// Let's output the $data variable's content
var_dump($data);

// Let's output the result of calling json_encode
$encoded = json_encode($data);

var_dump($encoded);

echo $encoded;

?>

You will see in the endpoint’s response the output of dumping the $data variable and another line with FALSE. That’s right, your HTTP endpoint code is returning nothing because json_encode function is failing and when you pass FALSE to echo, it doesn’t output anything . There are a lot of reasons why this could be failing. When this happened to me, some of my associative arrays had some NaN (Not a Number) values. Looks like it cannot be encoded to JSON.

How do we get the error when json_encode function fails?

• Pass JSON_THROW_ON_ERROR option to the function

• Call json_last_error function to get the actual error from the last json_encode call

• Call json_last_error_msg function to get the error message from the error occurred in the last json_encode call

What was the cause of this bug? You will have to look at the code that actually creates the data that you get from the getDataFromDatabase function. All these hours spent just trying to understand why the endpoint’s response was empty could be avoided if an error was being thrown. I think most times, developers don’t think too much much about errors or they try to write code that fails silently. This is a good example why, in most cases, the best solution is to just throw an error. Your users will get it but you can fix it much faster and start looking at the right place.

What are the takeaways from this?

Some days, I spend long hours debugging. I think most developers can relate to this. The story I told you in this post is based on a problem, that I actually had to fix, that was blocking users. You can get some takeaways that can help you in most of your detective days:

• Use a top-down approach. This means, you start looking at the most high level code that seems to be the problem. In this case, you can avoid looking inside the getDataFromDatabase function because you can start looking at the endpoint code itself. If you look inside at that function first, you would spend much more time in this entire investigation

• Look at the documentation. If you are using some built-in functions you can’t just assume they always work the right way and there will never be errors. Try to understand how they work and how to do the proper error handling

• Avoid to write code that fails silently. If something is not supposed to fail don’t’ just think that it will never happen and you don’t need to do anything. Write code that gets the error and if you don’t know what do with it, just throw it. If something goes wrong your users will be affected but you will be able to fix the problem much faster

• Most times, the bug is not where you think it is. This goes in line with previous bullet. If you throw errors instead of thinking they will never happen, you might start looking at the right place in the beginning of your investigation

• Take some breaks and drink water or eat something. You will need extra energy. This can be hard while you have users and your boss yelling at you because something is not working. But really, try to do it. Otherwise your brain will just “explode” and you will take much more time that is necessary to fix the problem.

I hope you enjoyed this story and took away something that will help you in your life as a software developer. Specially in those days when you have to fix really weird bugs.

Happy coding! :)

Let me know what you think and follow me for more cool content about dev stuff :)

codingsam @ Twitter