DEV Community

Cover image for Raku: features, community and main interpreter/VM
Tib
Tib

Posted on • Originally published at linuxfr.org

Raku: features, community and main interpreter/VM

#raku #discuss

Some Raku goodies in bulk

(This is a translated/improved version of the french article Raku en 2020 from the same author)

Thank you to Elizabeth Mattijsen who helped to proofread and with advices.

Table Of Contents

In this first blog post in a serie of two, you will find a general introduction with some community insights and details about interpreter and VM (Rakudo and MoarVM).

In a second part you get a long description of the Raku features to discover why Raku is worth talking about and learning.

(Next blog post is about past implementations and history details)

This blog post is made with love ❤️

About Raku

Perl 6 Raku is a programming language powerful and mature that was difficult to implement (which will be described in the second part of this post).

Raku née "Perl 6" is no longer the next version of Perl 5 (will be Perl 5.36 or Perl 7). Raku has been renamed in 2019 to clarify this state. But Raku shares a lot with Perl 5:

Lately, Raku has become too different from Perl 5 that we can simply say "Perl" to designate Perl 5 (or Perl 7) and we would prefer "Raku" in any circumstances to talk about Perl 6. There was a time where Perl 6 community was slightly against this renaming, but the overall opinion changed and the renaming was done smoothly between 2018 and 2019 (see next blog post of the série for more details).

You can follow Raku news by subscribing to the newsletter Rakudo Weekly News.

Raku has a small community but some "killer apps":

  • its IDE "Comma"
  • Cro, a toolbox for the web (applications, services)
  • Sparrow an automation framework, and its modules
  • a (small) CPAN

Big names from Raku world

Before going into details of renaming and Raku versions, let's hightlight some past and present big contributors to Raku:

  • Jonathan Worthington: Current lead developer of Rakudo (interview)
  • Elizabeth Mattijsen: very involved in Raku
  • Patrick Michaud: former lead deveoper of Rakudo, NQP, etc....
  • Audrey Tang: currently Taïwan minister (!), formerly main developer of Pugs (Perl 6 implementation in Haskell, see this old interview or this interview)
  • Damian Conway: teacher, speaker, spokesperson and prolific developer
  • Larry Wall: creator of Perl (BDFL) and creator of Raku. No need to present him! In 2021, no longer involved.
  • Moritz Lenz: writer and big contributor
  • Flavio S. Glock: lead developer of Perlito

Not all are still active and the list is not exhaustive!

The Raku community decided recently to setup a steering council.

Raku(do)

The most advanced implementation of Raku is Rakudo (interpreter) with NQP (builtins) on MoarVM (virtual machine).

When Rakudo is released, the bundle Rakudo + MoarVM is called Rakudo Star (or Rakudo *) ⭐

Since 2015, the implementation of Raku is "features-complete" and the effort on the compiler and VM are more about fixing issues and improving performances.

Raku had multiple official releases (the most important being in 2015) :

  • first public version of Rakudo * in july 2010 (major version but not feature-complete, still based on Parrot → Rakudo + Parrot)
  • 6.c "Christmas" in 2015 (announcement) (Rakudo + MoarVM)
  • 6.d "Diwali" in november 2019 (details of the several changes) (Rakudo + MoarVM)
  • 6.e will be the next major version (Rakudo + MoarVM).

From where comes this versioning scheme?

Simply from a recurring joke about the release date of Raku. It was common to answer "will be ready for christmas" 😀

In addition to major version, the team working on Raku produces monthly minor versions.

Some books

Before going into language details, here are some books to help you learn more about Raku:

Some links

The technical details starts here, I hope you're ready 😀

The power of Raku

Raku features are impressive, I wanted to start by this :)

Raku takes roots from the linguistic skills from Larry Wall, therefore Raku is very influenced by natural language. Raku is multi-paradigm (procedural, functional, concurrency programming and object oriented).

The values of Raku can be found in the Raku manifesto (1, 2, 3)

Like its cousin Perl:

  • it's a rich language with a notion of context
  • its regular expressions are very powerful and Raku even integrates grammar handling directly in the language (!)
  • it has a CPAN.

Contrary to Perl 5:

  • interfacing with native code is deadly simple
  • oriented object programming is integrated in the language (framework + object types)
  • the language can deal with concurrency and provides a full toolbox for it
  • Unicode is handled "by default"

For more technical details, here is the Raku glossary and also a features comparison between the compilers (it gives a good overview of the language). In the coming paragraphs, the Raku features are explained more in detail.

Courses like courses.raku.org or raku.guide are probably good "showcases" of Raku features.

Typing

  • Gradual typing (dynamic and static typing): 
# Dynamic
my Cool $var = 42;

# Functions using $var will coerce to another type 
# upon their needs

# Static 
my Str $var = "Raku";
Enter fullscreen mode Exit fullscreen mode
  • Annotation in signatures: 
sub ab(Int $a, Int $b) { return $a+$b; }; 
say ab(1,1);
Enter fullscreen mode Exit fullscreen mode
  • Type inference
  • Allomorphs (and polymorphic evaluation): IntStr $foo
  • Junctions: 
my $junction = 1|2|3;
$junction += 4; # junction now equals 5|6|7
$junction += (1&2); # junction now equals (6|7|8)&(7|8|9)
Enter fullscreen mode Exit fullscreen mode
  • Subsets: allow to extend isomorphic classes, can be used as constraints, while being resolved at runtime: 
subset Even where * %% 2;  
say 1 ~~ Even; # False
Enter fullscreen mode Exit fullscreen mode
  • Coercion: 
sub i-cast(Int() $n) { say $n.^name }; 
i-cast "42";
Enter fullscreen mode Exit fullscreen mode

Constraints + coercion ("I want a string object and I cast into integer object"):

sub f(Int(Str) $want-str-cast-to-int, Str() $cast-to-str) { } ;
f '10', 10;
Enter fullscreen mode Exit fullscreen mode

Or conditional constraints ("my argument should be an integer object or a string object"):

sub xyz($n where $_ ~~ Int | Str ) { say +$n }
Enter fullscreen mode Exit fullscreen mode
  • Native types: 
my int8 $verysmall = 42; 
say $verysmall;
Enter fullscreen mode Exit fullscreen mode
  • Auto‑boxing : automatic conversion from native type to object
  • Auto‑vivification in the hash/arrays: 
my %a; 
%a{'autovivification'}++; 
say %a{'autovivification'};
Enter fullscreen mode Exit fullscreen mode 
my @list = (1, 1, * + * ... *).grep({ $_.is-prime });
say @list[4]; # This generates elements 0..4. 
# Elements 5..∞ haven't been generated yet
Enter fullscreen mode Exit fullscreen mode

Routines

  • Argument: 
sub plus2($foo) { return $foo+2; }
Enter fullscreen mode Exit fullscreen mode
  • Typed: 
sub mul3(Int $foo) { return $foo*3; }
Enter fullscreen mode Exit fullscreen mode
  • With attribute rw or copy: 
sub make42($foo is rw) { $foo = 42; }
Enter fullscreen mode Exit fullscreen mode
  • Multi‑dispatch: 
multi sub sayit( Int $n ) { say "Number: $n"; } 
multi sub sayit( Str $s ) { say "String: $s"; }
Enter fullscreen mode Exit fullscreen mode

The dispatch is based on the number of arguments, their types, the constraints (we can even play with it with functions nextsame, nextwith, callsame, callwith).
See also "Proper narrowness analysis" for the best candidate resolution.

  • Slurp of remaining arguments and flattening of arguments (like in Perl 5).

Operators

Async, parallelism and concurrency

A full toolbox to handle asynchronous, parallelism and concurrency :

Object oriented programming and Meta object programming

Ability to define our own operators (seen above) and features. The meta object programming is partially handled in the virtual machine:

  • single or multiple inheritance
  • introspection
  • "edit" a class after instantiation (example)
  • composition/encapsulation of methods
  • trust relationship which allows access to private methods of another (parent) class without making them inherit
  • delegations has $.base is rw handles <meth1 meth2>
  • type whatever "*" (which gave its name to the cast "Rakudo star")

Grammars

Support for grammars (grammar,token, rule,regex) in the language!

grammar Calculator {
    token TOP { <calc-op> }

    proto rule calc-op          {*}
          rule calc-op:sym<add> { <num> '+' <num> }
          rule calc-op:sym<sub> { <num> '-' <num> }

    token num { \d+ }
}

class Calculations {
    method TOP              ($/) { make $<calc-op>.made; }
    method calc-op:sym<add> ($/) { make [+] $<num>; }
    method calc-op:sym<sub> ($/) { make [-] $<num>; }
}

say Calculator.parse('2 + 3', actions => Calculations).made;
Enter fullscreen mode Exit fullscreen mode

Interfacing

Native language support is really very easy!

  • Easy interfacing with native languages (Native Call): 
use NativeCall;

# Interface a function from unistd.h
sub getpagesize() returns int32 is native {}; 

# Interface a function from a shared library (e.g. libcalculator.so)
sub add(int32, int32) returns int32 is native("calculator") {};
Enter fullscreen mode Exit fullscreen mode 
use Test::More:from<Perl5>;
plan tests => 1;

use Inline::Perl5;
my $p5 = Inline::Perl5.new;
$p5.call('print', 'Hello World');
Enter fullscreen mode Exit fullscreen mode

Unicode processed at the grapheme level

Very high level of Unicode management, in particular thanks to primitives in the MoarVM virtual machine.

The language operators themselves can be UTF-8 (like the ones you create).

Other bulk features 

my $cube = -> $x {$x ** 3};
say $cube(3); # 27
Enter fullscreen mode Exit fullscreen mode

By Alphabetic order:

my $pow = {$^x ** $^y};
say $pow(3, 4); # 81
Enter fullscreen mode Exit fullscreen mode

Or even:

for 0..9 {
    say "$^n2, $^n1";
}
Enter fullscreen mode Exit fullscreen mode

Or with naming :

my $pow = {$:base ** $:exp};
say $pow(:base(25), :exp(2)); # 625
Enter fullscreen mode Exit fullscreen mode

Rakudo Star (from 2010)

The first major version of Rakudo Star (or "Rakudo *") was based on Parrot (PCT + ParrotVM), but we will ignore this version to focus on the current implementation (Rakudo + MoarVM).

Compilation (Rakudo)

Rakudo is a compiler for Raku that implements the entire language and can target several different VMs (the main one being MoarVM). Most of Rakudo is written in "simplified Raku" (NQP). Rakudo also implements precompilation to optimize its performance. In addition to that, Rakudo improves the syntax tree using several optimization methods, here are some examples:

  • the converted range iterations are in native loops
  • de-virtualization of private methods with compile-time resolution
  • removal of (statically) dead code
  • anonymize a variable
  • reduce the scope of a variable
  • unfolding of the junctions

NQP

NQP is a compiler like MiniPerl for Perl 5. It is a bootstrapping tool which helps to compile the Raku parts of Rakudo and compile the libraries before running have compiled the libraries. Unlike MiniPerl for Perl 5 (which is an interpreter that can interpret all Perl syntax but miss some batteries: means without modules mixing Perl code and native code), NQP can only compile a "simplified Raku". NQP refers to both the compiler and the source code contained in files with the ".nqp" extension.

Some NQP code looks like this:

method symtable() {
    %!symbol := nqp::hash() if nqp::isnull(%!symbol);
    %!symbol
}

method evaluate_unquotes(@unquotes) {
    my $result := self.shallow_clone();
    my $i := 0;
    my $elems := nqp::elems(@(self));
    while $i < $elems {
        $result[$i] := self[$i].evaluate_unquotes(@unquotes);
        $i := $i + 1;
    }
    $result
}
Enter fullscreen mode Exit fullscreen mode

NQP is part of Rakudo and seems inspired from MiniPerl6/KindaPerl6.

Perlito (see below) also implements the same kind of "simplified Raku".

Execution (MoarVM)

MoarVM stands for Metamodel On A Runtime VM (official site). It's a registrers-based virtual machine like Parrot. Jonathan Worthington is the founder and architect of MoarVM.

Here are some features of MoarVM:

  • internal representation closer to Raku (compared to Parrot) ("On the other hand, there are features that are hard to implement efficiently if the VM doesn't support it.", according to an interview with Flavio Glock );
  • superb Unicode support, with strings represented at the grapheme level
  • management of asynchronous and concurrent programming with control structures for concurrent programming, synchronous sockets, timers, processes, semaphores, blocking queues, etc.
  • precise garbage collection, generational (young → semispace and old → big buckets) and managing the competition
  • able to manage continuations
  • bounded serialization, serialize the code of the loaded modules for later and save start-up time by avoiding reconstructing the intermediate representations of this code
  • runtime loading of code
  • optimizations: tail call eliminations (do not stack in the call stack if the function return is in the call), dispatch, elimination of dead code, on stack replacement (replacement on the routine stack not optimized by a routine optimized), escape analysis (transform heap allocations into allocation on the stack or even in a register) and scalar replacement (eliminate memory allocations)
  • JIT compilation
  • profiling of the call stack and allocations
  • heap snapshotting: mainly for further scans (investigation and improvement of garbage collection, for example)

I also recommend reading this introduction to MoarVM.

MAST

MAST stands for MoarVM AST (Abstract Syntax Tree).

MBC

MBC stands for MoarVM Byte Code. Here what looks like MBC:

; Grab value out of Scalar
decont r4, r1 

; Grab type ; de-Scalar if needed
wval r5, 1, 34
decont r3, r5

; Ensure arg is an Int
istype r6, r4, r3
assertparamcheck r6
Enter fullscreen mode Exit fullscreen mode

Implementation based on Rakudo + MoarVM (since 2012)

MoarVM does not have an assembler.

Rakudo Moar

Below is another architecture diagram taken from a presentation on Rakudo/MoarVM optimizations (2014):
Rakudo Architecture

Conclusion

This first blog post treated more about current information about Raku.

I hope it gave you the envy to try it 😘

Next blog post will dig into historical bits of Raku (and implementation archeology) with several technical sketches 💃

It took me a lot of time to gather these traces of a dead world, but believe me it's very interesting so just let yourself be guided!

Top comments (0)

Read next

thj profile image

Should I use Tailwind? 5 things to consider — real experience [2024]

T.H. Jacobs -

arghodev profile image

🌀What’s New in TypeScript: Essential Updates, Advantages, and Tips for 2024

Argho Dev -

mitchiemt11 profile image

Has Broadcom Damaged VMware's Reputation?

Mitchell Mutandah -

magsimuhammadessa profile image

The story behind John Doe

Muhammad Essa -