DEV Community

Alexey Melezhik
Alexey Melezhik

Posted on • Edited on

Simple blackbox testing with Raku and Sparrow6

#raku #rakulang #sparrow6 #testing

Sparrow6 is a Raku automation framework with embedded tool that allows to create blackbox testing scenarios.

The feature is called Task Checks DSL.

Blackbox testing is based on the idea of agnosticism of internal structure of an object being tested. Sparrow6 takes a very simple approach allows one to create regexp based rules to analyze external scripts output.

Following are some examples.

Plain string search

Just create a task check file that includes a searched string

task.bash

!bash
echo "Hello World"
Enter fullscreen mode Exit fullscreen mode

task.check

Hello
Enter fullscreen mode Exit fullscreen mode

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:

20:01:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/plain-string] Hello World
[task check] stdout match <Hello> True
Enter fullscreen mode Exit fullscreen mode

Regexp search

Sparrow6 allows to use Raku regular expressions to search in script output:

task.check

regexp: ^^ Hello \s+ World
Enter fullscreen mode Exit fullscreen mode

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:

20:03:20 12/23/2019 [/home/melezhik/projects/blackbox-testing/regexp] Hello World
[task check] stdout match <^^ Hello \s+ World> True
Enter fullscreen mode Exit fullscreen mode

Sequential search

Sometime one need to test if a sequence of lines appears in a script output. begin:, end: notation is used to check sequences:

task.bash

!bash
echo "ON"
echo 1
echo 2
echo 3
echo "OFF"
Enter fullscreen mode Exit fullscreen mode

task.check

begin:
  ON
  1
  2
  3
  OFF
end
Enter fullscreen mode Exit fullscreen mode

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:

20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] ON
20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 1
20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 2
20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 3
20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] OFF
[task check] stdout match (s) <ON> True
[task check] stdout match (s) <1> True
[task check] stdout match (s) <2> True
[task check] stdout match (s) <3> True
[task check] stdout match (s) <OFF> True
Enter fullscreen mode Exit fullscreen mode

If switch first two digits, the test will fail:

task.bash

!bash
echo "ON"
echo 2
echo 1
echo 3
echo "OFF"
Enter fullscreen mode Exit fullscreen mode

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:

20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] ON
20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 2
20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 1
20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 3
20:37:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] OFF
[task check] stdout match (s) <ON> True
[task check] stdout match (s) <1> False
[task check] stdout match (s) <2> False
[task check] stdout match (s) <3> False
[task check] stdout match (s) <OFF> False
=================
TASK CHECK FAIL
Enter fullscreen mode Exit fullscreen mode

Range search

If one doesn't care about lines order and need to check
that lines are included between certain lines, it's possible to reshape the previous example using range notation: between {'RexExp1'} {'RegExp2'}, where RexExp1 sets a regexp pattern for the very first string in the range, and the second one does the same for the very last string in the range, all other strings should appear within the range in no particular order.

task.check

between: {'ON'} {'OFF'}
  1
  2
  3
end:
Enter fullscreen mode Exit fullscreen mode

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:

20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] ON
20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] 2
20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] 1
20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] 3
20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] OFF
[task check] stdout match (r) <1> True
[task check] stdout match (r) <2> True
[task check] stdout match (r) <3> True
Enter fullscreen mode Exit fullscreen mode

Dynamic search

Sparrow6 equips one with an ability to generate search criteria in run time, using a variety of programming languages ( Bash,Perl,Python,Ruby,Raku,Powershell)

Let's reshape the last example using generator: notation:

task.check

generator: <<CODE
!perl6

  say "between: \{'ON'\} \{'OFF'\}\n",(1 ... 3).join("\n"), "\nend:";

CODE
Enter fullscreen mode Exit fullscreen mode

Changing shebang we choose different language to implement dynamic search.

Let's do implementation on Bash.

task.check

generator: <<CODE
!bash

  echo "between: {'ON'} {'OFF'}"
  for i in {1..3}; do echo $i; done
  echo 'end:'

CODE
Enter fullscreen mode Exit fullscreen mode

Captures

Captures allow to extract certain chunks from searched strings and process them.

Let's count a total sum for all numbers included between "ON" and "OFF" strings:

task.check

between: {'ON'} {'OFF'}
 regexp: (\d+)
end:

code: <<CODE
!perl6

my $total;
for captures()<> -> $c {
 $total+=$c[0]
}

update_state(%( cnt => $total ))

CODE
Enter fullscreen mode Exit fullscreen mode

Test run:

perl6 -MSparrow6::DSL -e 'my %state = task-run; say %state.perl'

Output:

23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] ON
23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] 2
23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] 1
23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] 3
23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures]
OFF
[task check] stdout match (r) <(\d+)> True
{:cnt(6)}
Enter fullscreen mode Exit fullscreen mode

Asserts

Asserts allow to set predicates - statements that return true of false and create related checks.

In previous example let's check if total sum of numbers is equal 6:

task.check

between: {'ON'} {'OFF'}
 regexp: (\d+)
end:

generator: <<CODE
!perl6

my $total;
for captures()<> -> $c {
 $total+=$c[0]
}

say "assert: { $total == 6  }", " total sum is 6";

CODE
Enter fullscreen mode Exit fullscreen mode

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:

19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] ON
19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] 1
19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] 2
19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] 3
19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] OFF
[task check] stdout match (r) <(\d+)> True
[task check] <total sum is 6> True
Enter fullscreen mode Exit fullscreen mode

Passing parameters

One can pass parameters handled in dynamic search constructions.

task.check

between: {'ON'} {'OFF'}
 regexp: (\d+)
end:

generator: <<CODE
!perl6

my $amount = config()<amount>;

my $total;
for captures()<> -> $c {
 $total+=$c[0]
}

say "assert: { $total == $amount  }", " total sum is amount";
CODE
Enter fullscreen mode Exit fullscreen mode

Output:

perl6 -MSparrow6::DSL -e 'task-run("{$*CWD}",%( amount => 10 ))'
19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] ON
19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] 1
19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] 2
19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] 3
19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] OFF
[task check] stdout match (r) <(\d+)> True
[task check] <total sum is 10> False
=================
TASK CHECK FAIL
Enter fullscreen mode Exit fullscreen mode

Streams

The last interesting feature to be mentioned here is called streams.
Steams allow one to iterate over blocks of data found during sequential or range search.

In the last example consider the case when we have more then one block of numbers between 'ON' and 'OFF' delimiters.

task.bash


echo "ON"
echo 1
echo 2
echo 3
echo "OFF"

echo "ON"
echo 3
echo 3
echo 3
echo "OFF"

echo "ON"
echo 2
echo 2
echo 2
echo "OFF"
Enter fullscreen mode Exit fullscreen mode

Let's write an iterator for all the blocks found:

between: {'ON'} {'OFF'}
 regexp: (\d+)
end:

code: <<CODE
!perl6

  my $stream-num = 1;
  for streams_array()<> -> $s {
    for $s<> -> $i {
      say "stream#{$stream-num}: {$i[0]}";
    }
    $stream-num++;
  }

CODE
Enter fullscreen mode Exit fullscreen mode

Test run:

perl6 -MSparrow6::DSL -e 'task-run

Output:

19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] ON
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 1
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] OFF
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] ON
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] OFF
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] ON
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] OFF
[task check] stdout match (r) <(\d+)> True
[task check] stream#1: 1
[task check] stream#1: 2
[task check] stream#1: 3
[task check] stream#2: 3
[task check] stream#2: 3
[task check] stream#2: 3
[task check] stream#3: 2
[task check] stream#3: 2
[task check] stream#3: 2
Enter fullscreen mode Exit fullscreen mode

Conclusion

Task Checks DSL is a simple, yet powerful tool allow one to write blackbox tests. The feature is a part of Sparrow6 core and provided out of the box.

Thank you for reading and Merry Christmas!

Top comments (0)

Read next

gadekar_sachin profile image

# Ultimate Guide: Debugging Techniques for QA Automation Engineers

Sachin Gadekar -

byte_sized profile image

Hello World!

Jacqueline Daly -

siddhantkcode profile image

When k6 eats up your RAM: Slashing memory usage in load tests

Siddhant Khare -

alexandre_vandamme_241212 profile image

How I Made $6,200 in a Single Bug Bounty

Alexandre Vandamme -