Specification-based Testing: Explore the program

The third step of the specification-based test is to explore the program behavior. You can skip this step if you wrote the code that you are testing.

This step is a good way to understand code that you need to test but was not implemented by you. In this scenario, you don’t have the complete specification. So you’re trying to understand the specification that the code was built on by doing a reverse engineering process.

The idea here is to read the code. Execute it through tests with simple inputs, debug it, and understand it. Then explore to see all the code that interacts with this specific part of the code that you are testing. Are there some prerequisites that you can spot on, are the related parts making decisions based on this result? Is the return used right away, or require some processing before or after?

All of this will help you understand not just the code that you are testing, but gain domain knowledge of the code parts that interact with this code.

Example:

Let’s see in practice. I’ll continue using the code example that is being used in this blog series (The 8XY4 ADD with carry flag from the Chip-8).

Assume that the class has these two attributes:

• An int (that will be storing values at max 255) array called registers (size 16)
• An int variable called pc (program counter)

// 8XY4
// ADD
int registerxIndex = ((instruction & 0x0F00) >> 8);
int registeryIndex = ((instruction & 0x00F0) >> 4);

int firstToSum = registers[registerxIndex];
int secondRegisterToSum = registers[registeryIndex];

int registersSum = (firstToSum + secondRegisterToSum);
registers[registerxIndex] = registersSum & 0xFF;
if (registersSum > 255) registers[registers.length - 1] = 1;
else registers[registers.length - 1] = 0;
pc += 2;

The first step is to start reading the code, and taking notes in the process. Let’s take a look line by line:

int registerxIndex = ((instruction & 0x0F00) >> 8);

By the variable name, it seems that is picking the register X index. (In this code a bitwise operation is happening, I won’t dive into too many details here. It is taking a number, isolating a specific 4-bit section from it, and then moving it 8 places to the right to make it easier to work with. It will result in a number between 0 and 15).

int registeryIndex = ((instruction & 0x00F0) >> 4);

This code seems to do the same as before but for register Y. (is taking a number, isolating a specific 4-bit section from it, and then moving it 4 places to the right to make it easier to work with. It will result in a number between 0 and 15).

So far we conclude that we have a single instruction that we can extract positions X and Y from it.

int firstToSum = registers[registerxIndex];

Then we used the position to get the value to be processed.

int secondRegisterToSum = registers[registeryIndex];

And then the second one.

int registersSum = (firstToSum + secondRegisterToSum);

It did a simple sum

registers[registerxIndex] = registersSum & 0xFF;

Then store the 8-bit result of the sum in the register X position. Since the register array is an array of ints, where this int can have a max value of 255. A number higher than 255 can overflow the value, so we store only the 8-bit part of the sum.

if (registersSum > 255) registers[registers.length - 1] = 1;

If the result value of the sum is bigger than 255, the last position of the array of registers gets updated to 1. It seems that it’s a flag to signal that the result is a number bigger than an 8-bit could represent and this should be considered in the next operations.

else registers[registers.length - 1] = 0;

Otherwise, 0 is stored in the last position of the register array. The sum did not overflows.

pc += 2;

The pc (program) variable is incremented by two.

So to wrap up our understanding and the specifications that we have extracted so far:

• It extracts two index values X and Y from an int instruction.
• It gets the value in the register array for those positions.
• It adds the values and stores the 8-bit result in the X array.
• If the sum exceeds 255 which is the limit of an 8-bit number it has to store this information in the last register of the array.
  • Otherwise is set to zero
• The pc variable is always incremented

Now we have a much better understanding just by reading the code and taking notes.

Time to Explore!

Now that you already reverse-engineered the specification by reading the code. It’s time to explore the program through tests.

Start planning your tests by asking some questions about the knowledge you just acquired.

In our example, you could:

• Check what happens if you set a value in X and Y, it changes the values of X? And what about Y?
• If you set values in registers that don’t match the indexes from the instruction (like setting registers 3 and 4 when the instruction reads 5 and 6), what happens? Will any value change?
• What happens if your result sum overflows the 8-bit so you get a better understanding of the code, especially the code that wasn’t implemented by you?
• Does the variable PC is incremented in every scenario?
• What if you did not initialize the register values?
• What if both values are 0, or only one?

Just by asking these questions, you can start creating a handful set of test scenarios that will help you understand better the code, by interacting and questioning the knowledge that you acquired.

If some doubts come in the way, ask your colleagues to clarify what you did not understand.

Take the exploration time also as an opportunity to explore the code that is related to the code you are testing, ask more questions to yourself:

• Do I need to process something before interacting with this code?
• Where does the data that will be processed by this code come from?
• How this carry flag is being used?
• Where these registers will be used after executing this operation?

This helps you grow your domain knowledge about the project you are working on.

Have a code that you need to test and understand? Use this technique and get a good understanding of how everything works.

Want to learn more about this topic?

In the following days, I’ll dive into each of those steps in more detail, follow me, and do not miss my next blog posts of this series when it comes out.

In my next blog post, we are going to discuss how to identify partitions, explore inputs and outputs, how they act together, and how each one can interfere with the result. So you can identify when you need to check for errors, and invalid inputs, and understand better the prerequisites to run the code.

Stay tuned to learn more! Don't miss out!

Willian Moya (@WillianFMoya) / X (twitter.com)

Willian Ferreira Moya | LinkedIn