How to create a RESTful API to the PicoLisp Database

In the previous posts we have created a database and generated content that can be displayed in the browser. The format we chose was human-readable but not so suitable for automatic processing.

Therefore we will now create a REST-API that enables other services to access the data in JSON format.

What is a REST-API?

REST stands for Representational State Transfer, which means that it fulfills the following properties:

• client-server model,
• stateless protocol (no session information needed),
• cacheability,
• layered system,
• uniform interface.

In simple words, what we want is to create an interface to our PicoLisp database that can be accessed from outside. The client doesn't have to speak PicoLisp, and the output format should be a standard machine-readable format such as JSON.

Let's create an API that any client can access via a simple GET-request. Similarly to the previous example, the record is specified via the URL.

Creating JSON output in PicoLisp

JSON (Java*Script **Object **N*otation) is a very popular, language-independent data format that can be used to exchange data between different services.

It supports a number of data types:

• Numbers,
• Strings, delimited with double-quotes and backslash as escape symbol,
• Boolean (true or false)
• Arrays using square bracket notation with comma-separated elements,
• Objects as a collection of name-value pairs, where the names are also strings,
• null as empty value.

PicoLisp comes with a library to create json-output. It can be found in the library files under json.l. There are four functions:

• checkJson (X Item),
• parseJson (Str),
• readJson (),
• printJson (Item).

For our purpose, we will only need the printJson (Item) function. Its functionality is very simple: it takes a list of cons-pairs and prints them in the JSON format.

Let's open the REPL ($ pil +) and try it at an example:

: (setq A 
   '( ( name . "Smith" ) 
   ( age . 25)
   ( address 
      ( street . "21 2nd Street") 
      ( city . "New York") 
      ( state .  "NY" ) 
      ( zip  .  10021 ) ) ) )

We have a nested list of cons-pairs, i. e. a list of cells where the "key" is found in the CAR and the "value" in the CDR. The keys are internal symbols (that's why it's name and not "name").

We can execute the printJson function on this:

: (printJson A)
{"name": "Smith", "age": 25, "address": {"street": "21 2nd Street", "city": "New York", "state": "NY", "zip": 10021}}

The function prints out a valid JSON string.

Creating a cons-pair list

In order to be able to use printJson, we need to convert the database output into a structured cons-pair list. This will require some manual formatting. Up to now, we only queried specific attributes for a record, like ( : nm ). However, we can also get a list of items using the getl function.

Let's test it in the REPL. We can start the database without the server using any of the previous scripts, for example this one:

$ pil family.l -family~main +
family: 

It opens us a prompt directly in the family namespace. Let's get the item {A1} in list format:

family: (getl '{A1})
-> ((({A2} {A4}) . kids) ("Margaret Rose" . nm) ({A3} . mate) ({A11} . ma) ({A33} . pa) (705091 . dat) ("Countess of Snowdon" . job))

As you can see, it's a cons-pair list, but with some differences to our desired format:

1. Key and value are reversed. instead of ( "Margaret Rose" . nm ) we need ( nm . "Margaret Rose" ).
2. The symbol names at mate, ma, pa and so on should be replaced by the person's names.
3. The date 705091 should be formatted.

Let's go through it step by step.

Changing CAR and CDR in the cons pair

How can we convert ( "Margaret Rose" . nm ) to ( nm . "Margaret Rose" )? Well, this is quite easy: We can build a new cons-pair using the cons function, which takes two arguments: CAR and CDR.

: (cons 1 2)
-> (1 . 2)

We can apply this to all items in the list using mapcar and an anonymous function:

(mapcar
   '((X)
      (cons (cdr X) (car X)) )
   (getl This ) 

Let's test it in the REPL:

family: (mapcar '((X) (cons (cdr X) (car X))) (getl '{A1}))
-> ((kids {A2} {A4}) (nm . "Margaret Rose") (mate . {A3}) (ma . {A11}) (pa . {A33}) (dat . 705091) (job . "Countess of Snowdon"))

Formatting the list

To get a better overview, let's write down each cons-pair of the getl output:

• (({A2} {A4}) . kids)
• ("Margaret Rose" . nm)
• ({A3} . mate)
• ({A11} . ma)
• ({A33} . pa)
• (705091 . dat)
• ("Countess of Snowdon" . job)

Looking at the car, we have four cases: It is either a list of +Person objects, a number representing a date, a Person object, or a string. Let's treat either of these cases separately. We can switch between different "cases" using the cond function:

(cond ('any1 . prg1) ('any2 . prg2) ..) -> any

Multi-way conditional: If any of the anyN conditions evaluates to non-NIL, prgN is executed and the result returned. Otherwise (all conditions evaluate to NIL), NIL is returned.

Case 1: The CAR is a number.

If the value (let's call it V) is a number, return the formatted value. We can test whether it's a number with the num? function.

(cond
   ((num? V) (datStr V))

Case 2: The CAR is an +Person object.

To check this, we can use the isa function, which takes a class and an object. If yes, we want to get the name property of this object. We can get it with the ; function:

(cond
   ((num? V) (datStr V))
   ((isa '+Person V) (; V nm))

** Case 3: The CAR is a list of +Person objects.**

Now this one is a little bit tricky. Let's say the CAR is a list. In this case, we want to loop over every list item and return the name. We can do this with mapcar and an anonymous function which takes an object and returns it name property:

(mapcar '((This) (: nm)) V)

But how can we check if V is a list? For this purpose, we can use the pair function which checks if the argument is a cons pair and returns it if true. Technically, a "normal" list is also a cons pair, while numbers, strings and and objects (i. e. all the other cases) are not. So we can expand our cond condition list with the following line:

(cond
   ((num? V) (datStr V))
   ((isa '+Person V) (; V nm))
   ((pair V) (mapcar '((This) (: nm)) V) )

** Case 4: The CAR is a string.**

Lastly, if the CAR is a string or any other case we didn't consider, we do nothing with it:

(cond
   ((num? V) (datStr V))
   ((isa '+Person V) (; V nm))
   ((pair V) (mapcar '((This) (: nm)) V) )
   (T V) )

Bringing it together

Now we combine everything in one function:

1. get the list with (getl This)
2. apply mapcar to build a new cons pair
3. before we set the car, we modify it depending on the result of cond.

(mapcar 
   '((X)
      (cons (cdr X) 
         (let V (car X)
            (cond
               ((pair V)
               (mapcar '((This) (: dat)) V) )
               ((num? V) (datStr V))  # Can only be date
               ((isa '+Person V) (; V nm))
               (T V) ) ) ) )
   (getl This) ) 

Convert to JSON and return it!

Now that we have our list, all we have to do is converting it and modify our HTTP-Header so that it returns Content-Type: application/json instead of text/html. Instead of our standard html function, we call the function httpHead with the arguments "application/json" for the content-type and 0 for the cache-control. Let's test it in the REPL:

family: (httpHead "application/json" 0)

HTTP/1.0 200 OK
Server: PicoLisp
Date: Sun, 17 Oct 2021 14:44:31 GMT
Cache-Control: max-age=0
Cache-Control: private, no-store, no-cache
Content-Type: application/json

The function returns a complete header which informs the browser that the received data format is JSON.

Now we have to send it. In the HTTP protocol, there are two possibilities to upload data: Either the content-length is communicated in the header (which is difficult, i. e. expensive in a dynamically created page), or the upload is sent in chunks. We can create chunked upload and send the content via the ht:Out function from the htlibrary.

   (httpHead "application/json" 0)
   (ht:Out *Chunked
      (printJson
         ...

As final step, we wrap it in a function called person.json. The .json is not a requirement, but it makes clear that the output of this format is a json-format. Accordingly, we modify the server and the allowed-function.

(allowed NIL
   "@lib.css" "!person.json" )

...

(de person.json (This)
   (httpHead "application/json" 0)
   (ht:Out *Chunked
      (printJson
         ...
   ...

Then we can start the program with

$ pil family-rest.l -family~main -go +

If we now point the server towards http://localhost:8080/?-A67, we see the following JSON-formatted output:

This output can be read by any application that calls a GET request to the URL.

That's it! The complete source code to this example can be found here.

Sources

• https://software-lab.de/doc/index.html
• https://software-lab.de/doc/app.html