Start testing with PHPT tests in PHPUnit

Over the years, automated testing has become an established practice in software development. It thus has also become an essential skill to learn for any developer. In the past few months, I have talked to several developers who had recently started with testing in PHP. Some of them expressed they found their testing framework of choice (often PHPUnit) quite difficult to get started with.

In a sense, this is not surprising. Getting started with testing is already challenging on its own. You have to choose units to test and learn to write testable code. You must learn to distinguish essential behavior from implementation details. At the same time you have to get to know your testing framework. Most popular and mature testing tools need quite some boilerplate and background knowledge. This adds to the already steep learning curve. As people learn best by tackling one step at a time, this is not ideal. It would be much better if we had an easy testing framework that juniors can use to learn the basics of testing without the extra overhead.

PHPT tests

Enter PHPT tests. Through work on some small bug fixes in the PHP core (and speaking about that) I learned about the PHPT test format. This is the test format used for testing the PHP interpreter itself. It is straightforward and simple to get into. The main idea is that a test contains a PHP script that prints output, and the output expected from that script. An elementary PHPT test would look something like this:

--TEST--
Basic arithmetic - addition
--FILE--
<?php var_dump(42 + 1); ?>
--EXPECT--
int(43)

Here, the --TEST-- section provides a short description about what the test aims to test. The --FILE-- section contains a script that prints some output. In this case we output the result of adding two integers. After --EXPECT-- comes the output we expect from the --FILE-- section.

Note that we use var_dump for outputting the result value. If we would use echo or print the output would be just 43. In that case we would not know whether that was the integer 43 or the string “43”. That makes us unable to verify that the result is of the correct type. Therefore we prefer var_dump: it shows the type of the value.

The PHPT test format is not feature-rich. Still it supports all the constructs necessary to write sensible tests. You can use them by adding these sections:

• --EXPECTF--, --EXPECTREGEX--: Instead of specifying the exact desired output with --EXPECT--, one can specify a pattern for the expected output. This pattern can be provided either as a printf-like string or a regular expression.
• --SKIP--: You can add this section to describe when a test should be skipped. This can be used to check for a specific platform or the presence of a required PHP extension.
• --CLEAN--: When the test creates some temporary artifacts (like files on disk) the code in this section can clean them up.
• --INI--, --ENV--: Sometimes a test needs to run with specific settings like php.ini directives or environment variables. You can specify those settings in these two blocks.

More information about the PHPT test structure can be found on the website of the PHP QA team.

Learning to test with PHPT

When learning to test, we can see the frugal feature set of PHPT as an advantage. It allows one to focus on learning the concepts of testing and testability without getting lost in features of a specific testing framework. These features are useful for developers that are already accustomed to testing. Still it is better to build understanding of how they work under the hood first. In fact, juniors will probably find a way to emulate them in a ‘naive’ way using concepts they already know:

• Want to test that a function throws an exception? Use a try-catch. No need for expectException() yet.
• Have multiple tests that look similar, but with different data? Put them in a for- or foreach-loop. This paves the way for a data provider later on.
• Sharing some initialization logic between tests? Move it to a function. Later on such a function can become a setUp-method.
• Need a test double? Create a dummy implementation within the test yourself. This helps to learn what a mocking library does under the hood. It also encourages to keep interfaces small and method chains short. Patterns for the different types of test doubles will emerge more naturally. We don’t have them all coming from createMock() anymore. This helps juniors to learn how e.g. a stub differs from a mock.

As a bonus, PHPT tests make it easy to write characterisation tests. Take any code fragment that exercises the system under test in some way and put it in the test. Then run it to get the current output and register that as the expected output. This strategy provides a great opportunity to learn how one can use characterisation tests as an aid to refactor legacy code. It also teaches how they are brittle and should eventually be replaced by tests that properly specify desired behavior.

Using PHPT tests with PHPUnit

One little-known feature of PHPUnit is that it actually has built-in support for PHPT tests out of the box. All we have to do to enable this is to add a directory with the .phpt suffix to the phpunit.xml configuration file:

<?xml version="1.0" encoding="UTF-8"?>
<phpunit bootstrap="vendor/autoload.php">
    <testsuites>
        <testsuite name="My Test Suite">
            <directory>test</directory>
            <!-- (line below added) -->
            <directory suffix=".phpt">test</directory>
        </testsuite>
    </testsuites>
    <!-- ... -->
</phpunit>

If we have done so, PHPUnit will also look for .phpt files in the test directory, and execute them as PHPT tests. With this configuration we can start mixing ‘normal’ PHPUnit tests and PHPT tests. Still we can run them as if there were no difference. We just support both test formats side-by-side without any extra infrastructure or tooling.

This means developers experienced with testing can still write their usual PHPUnit tests. Developers that have recently started out with testing now have a simpler way of writing tests though. Instead of the following test from the PHPUnit manual

<?php
use PHPUnit\Framework\TestCase;

class StackTest extends TestCase {
    public function testPushAndPop() {
        $stack = [];
        $this->assertSame(0, count($stack));

        array_push($stack, 'foo');
        $this->assertSame('foo', $stack[count($stack)-1]);
        $this->assertSame(1, count($stack));

        $this->assertSame('foo', array_pop($stack));
        $this->assertSame(0, count($stack));
    }
}

you can write this equivalent PHPT test:

--TEST--
Array as a stack
--FILE--
<?php
$stack = [];
var_dump(count($stack));

array_push($stack, 'foo');
var_dump($stack[count($stack)-1]);
var_dump(count($stack));

var_dump(array_pop($stack));
var_dump(count($stack)); 
?>
--EXPECT--
int(0)
string(3) "foo"
int(1)
string(3) "foo"
int(0)

Notice the differences in boilerplate/prerequisites to understand the two tests. For the PHPUnit test one needs to know about the PHPUnit TestCase class and have a basic understanding of OOP. Also you have to know that methods starting with test are invoked by the PHPUnit test runner. Lastly, you need to be able to pick the appropriate assertion method.

With the PHPT test juniors can take any existing snippet of PHP code that produces some output and turn it into an automated test. They might already use such a script for manual testing. Now they can convert it to a test that runs with the rest of the testsuite.

Limitations

Of course, most of us do not write tests using PHPT, but use a framework like PHPUnit, phpspec or atoum. There are good reasons for that. Using the PHPT format and running it with PHPUnit has some serious limitations:

• Features. Most test frameworks have several helpful features built-in. These help to reduce the amount of code to write for certain tests or clarify why certain tests failed. Think of data providers and integration with mocking libraries. But there are also test dependencies, ways of expecting exceptions and specialized assertion methods. We have discussed that most of these can be ‘emulated’ in tests themselves. Still, having these in our test framework reduces the amount of boilerplate we need to write. It makes experienced testers much more productive.
• IDE support. There is no real support for PHPT tests in most IDE’s. With proper file associations the code between <?php ?> tags is usually still highlighted. The PHPT sections will not be, though. Additionally, I don’t know of any IDE that can quickly run a single .phpt file as a PHPT test from the editor.
• Unsupported sections. The PHPUnit test runner has only limited support for PHPT tests. Several PHPT sections are not supported by this runner, most of them related to simulating HTTP input.

Conclusion

I see no use in switching to PHPT tests if you are already experienced in writing tests. Still, using PHPT tests might be a nice way for less experienced developers to get started with testing. With little boilerplate and a small feature set, it helps them to focus on ways of testing and writing testable code. That is difficult enough in itself, and deserves their full attention. More advanced features and constructs will arise naturally and help to discover how a testing framework works under the hood.

Most important: in PHPUnit, PHPT tests can be enabled alongside ‘normal’ tests with just a single line of configuration. Such a strategy provides a smooth upgrade path from basic PHPT tests to a more mature testing framework. It does not even require additional tooling. Help your developers to start testing, enable PHPT tests!

Enjoyed reading this post?

My colleagues and I regularly write webdev-related blog posts like this to share the things we learn or discover. You can find all our posts on the Moxio blog. Get notified of new posts by following me or Moxio on Twitter or by subscribing to our RSS feed.