Sparrow6 is a Raku automation framework with embedded tool that allows to create blackbox testing scenarios.
The feature is called Task Checks DSL.
Blackbox testing is based on the idea of agnosticism of internal structure of an object being tested. Sparrow6 takes a very simple approach allows one to create regexp based rules to analyze external scripts output.
Following are some examples.
Plain string search
Just create a task check file that includes a searched string
task.bash
!bash
echo "Hello World"
task.check
Hello
Test run:
perl6 -MSparrow6::DSL -e task-run
Output:
20:01:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/plain-string] Hello World
[task check] stdout match <Hello> True
Regexp search
Sparrow6 allows to use Raku regular expressions to search in script output:
task.check
regexp: ^^ Hello \s+ World
Test run:
perl6 -MSparrow6::DSL -e task-run
Output:
20:03:20 12/23/2019 [/home/melezhik/projects/blackbox-testing/regexp] Hello World
[task check] stdout match <^^ Hello \s+ World> True
Sequential search
Sometime one need to test if a sequence of lines appears in a script output. begin:
, end:
notation is used to check sequences:
task.bash
!bash
echo "ON"
echo 1
echo 2
echo 3
echo "OFF"
task.check
begin:
ON
1
2
3
OFF
end
Test run:
perl6 -MSparrow6::DSL -e task-run
Output:
20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] ON
20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 1
20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 2
20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 3
20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] OFF
[task check] stdout match (s) <ON> True
[task check] stdout match (s) <1> True
[task check] stdout match (s) <2> True
[task check] stdout match (s) <3> True
[task check] stdout match (s) <OFF> True
If switch first two digits, the test will fail:
task.bash
!bash
echo "ON"
echo 2
echo 1
echo 3
echo "OFF"
Test run:
perl6 -MSparrow6::DSL -e task-run
Output:
20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] ON
20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 2
20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 1
20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 3
20:37:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] OFF
[task check] stdout match (s) <ON> True
[task check] stdout match (s) <1> False
[task check] stdout match (s) <2> False
[task check] stdout match (s) <3> False
[task check] stdout match (s) <OFF> False
=================
TASK CHECK FAIL
Range search
If one doesn't care about lines order and need to check
that lines are included between certain lines, it's possible to reshape the previous example using range notation: between {'RexExp1'} {'RegExp2'}
, where RexExp1 sets a regexp pattern for the very first string in the range, and the second one does the same for the very last string in the range, all other strings should appear within the range in no particular order.
task.check
between: {'ON'} {'OFF'}
1
2
3
end:
Test run:
perl6 -MSparrow6::DSL -e task-run
Output:
20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] ON
20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] 2
20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] 1
20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] 3
20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] OFF
[task check] stdout match (r) <1> True
[task check] stdout match (r) <2> True
[task check] stdout match (r) <3> True
Dynamic search
Sparrow6 equips one with an ability to generate search criteria in run time, using a variety of programming languages ( Bash,Perl,Python,Ruby,Raku,Powershell)
Let's reshape the last example using generator:
notation:
task.check
generator: <<CODE
!perl6
say "between: \{'ON'\} \{'OFF'\}\n",(1 ... 3).join("\n"), "\nend:";
CODE
Changing shebang we choose different language to implement dynamic search.
Let's do implementation on Bash.
task.check
generator: <<CODE
!bash
echo "between: {'ON'} {'OFF'}"
for i in {1..3}; do echo $i; done
echo 'end:'
CODE
Captures
Captures allow to extract certain chunks from searched strings and process them.
Let's count a total sum for all numbers included between "ON" and "OFF" strings:
task.check
between: {'ON'} {'OFF'}
regexp: (\d+)
end:
code: <<CODE
!perl6
my $total;
for captures()<> -> $c {
$total+=$c[0]
}
update_state(%( cnt => $total ))
CODE
Test run:
perl6 -MSparrow6::DSL -e 'my %state = task-run; say %state.perl'
Output:
23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] ON
23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] 2
23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] 1
23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] 3
23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures]
OFF
[task check] stdout match (r) <(\d+)> True
{:cnt(6)}
Asserts
Asserts allow to set predicates - statements that return true
of false
and create related checks.
In previous example let's check if total sum of numbers is equal 6
:
task.check
between: {'ON'} {'OFF'}
regexp: (\d+)
end:
generator: <<CODE
!perl6
my $total;
for captures()<> -> $c {
$total+=$c[0]
}
say "assert: { $total == 6 }", " total sum is 6";
CODE
Test run:
perl6 -MSparrow6::DSL -e task-run
Output:
19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] ON
19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] 1
19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] 2
19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] 3
19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] OFF
[task check] stdout match (r) <(\d+)> True
[task check] <total sum is 6> True
Passing parameters
One can pass parameters handled in dynamic search constructions.
task.check
between: {'ON'} {'OFF'}
regexp: (\d+)
end:
generator: <<CODE
!perl6
my $amount = config()<amount>;
my $total;
for captures()<> -> $c {
$total+=$c[0]
}
say "assert: { $total == $amount }", " total sum is amount";
CODE
Output:
perl6 -MSparrow6::DSL -e 'task-run("{$*CWD}",%( amount => 10 ))'
19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] ON
19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] 1
19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] 2
19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] 3
19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] OFF
[task check] stdout match (r) <(\d+)> True
[task check] <total sum is 10> False
=================
TASK CHECK FAIL
Streams
The last interesting feature to be mentioned here is called streams.
Steams allow one to iterate over blocks of data found during sequential or range search.
In the last example consider the case when we have more then one block of numbers between 'ON' and 'OFF' delimiters.
task.bash
echo "ON"
echo 1
echo 2
echo 3
echo "OFF"
echo "ON"
echo 3
echo 3
echo 3
echo "OFF"
echo "ON"
echo 2
echo 2
echo 2
echo "OFF"
Let's write an iterator for all the blocks found:
between: {'ON'} {'OFF'}
regexp: (\d+)
end:
code: <<CODE
!perl6
my $stream-num = 1;
for streams_array()<> -> $s {
for $s<> -> $i {
say "stream#{$stream-num}: {$i[0]}";
}
$stream-num++;
}
CODE
Test run:
perl6 -MSparrow6::DSL -e 'task-run
Output:
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] ON
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 1
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] OFF
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] ON
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] OFF
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] ON
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] OFF
[task check] stdout match (r) <(\d+)> True
[task check] stream#1: 1
[task check] stream#1: 2
[task check] stream#1: 3
[task check] stream#2: 3
[task check] stream#2: 3
[task check] stream#2: 3
[task check] stream#3: 2
[task check] stream#3: 2
[task check] stream#3: 2
Conclusion
Task Checks DSL is a simple, yet powerful tool allow one to write blackbox tests. The feature is a part of Sparrow6 core and provided out of the box.
Thank you for reading and Merry Christmas!
Top comments (0)