Since my last post on reddit asking for some help regarding Emacs and TiddlyWikis REST API I gained some elisp
knowledge I’d like to share.
TiddlyWiki 5
For those of you who haven’t heard of TiddlyWiki yet:
TiddlyWiki is a personal wiki and a non-linear notebook for organising and sharing complex information. It is an open-source single page application wiki in the form of a single HTML file that includes CSS, JavaScript, and the content. It is designed to be easy to customize and re-shape depending on application. It facilitates re-use of content by dividing it into small pieces called Tiddlers. – Wikipedia
You use the wiki as a single HTML page or via nodejs
. With nodejs
we can
talk to Tiddlywiki via its REST API.Every single page inside the wiki is called tiddler
.
On the philosophy of tiddlers: “The purpose of recording and organising information is so that it can be used again. The value of recorded information is directly proportional to the ease with which it can be re-used.”
A tiddler
has following format:
Code Snippet 1: Tiddler JSON format
Next I’ll show you how to setup your TiddlyWiki instance.
Basic setup
I use node.js
to run my TiddlyWiki instance. For isolation reasons I use Docker
to run it. Here is my Dockerfile
:
FROM mhart/alpine-node
# Create a group and user
RUN addgroup -g 984 -S appgroup
RUN adduser -h /DATA/wiki -u 1000 -S appuser -G appgroup
# Tell docker that all future commands should run as the appuser user
ENV TW_BASE=/DATA TW_NAME=wiki TW_USER="xxx" TW_PASSWORD="xxx" TW_LAZY=""
ENV TW_PATH ${TW_BASE}/${TW_NAME}
WORKDIR ${TW_BASE}
RUN npm install -g npm@8.10.0
RUN npm install -g tiddlywiki http-server
# COPY plugins/felixhayashi /usr/lib/node_modules/tiddlywiki/plugins/felixhayashi
# RUN ls -la /usr/lib/node_modules/tiddlywiki/plugins
COPY start.sh ${TW_BASE}
# Change ownership
RUN chown appuser:appgroup /DATA/start.sh
EXPOSE 8181
USER appuser
ENTRYPOINT ["/DATA/start.sh"]
CMD ["/DATA/start.sh"]
Code Snippet 2: Dockerfile for running TiddlyWiki 5 using alpine
And as for start.sh
:
#!/usr/bin/env sh
# Start image server
http-server -p 82 /DATA/wiki/images &
# Start tiddlywiki server
tiddlywiki /DATA/wiki --listen port=8181 host=0.0.0.0 csrf-disable=yes
Code Snippet 3: Bash script to start a simple http-server (for uploading images) and the tiddlywiki server instance (node.js)
Now you should be able to call the API (via curl
for example):
curl http://127.0.0.1:8181/recipes/default/tiddlers/Emacs | jq
Code Snippet 4: Now you should be able to call the API (via curl
for example).
{
"title": "Emacs",
"created": "20210623082136326",
"modified": "20210623082138258",
"tags": "Topics",
"type": "text/vnd.tiddlywiki",
"revision": 0,
"bag": "default"
}
Code Snippet 5: The REST API will send back a JSON response.
request.el
I use request.el for crafting and sending HTTP requests. So what is request.el
all about?
Request.el is a HTTP request library with multiple backends. It supports url.el which is shipped with Emacs and curl command line program. User can use curl when s/he has it, as curl is more reliable than url.el. Library author can use request.el to avoid imposing external dependencies such as curl to users while giving richer experience for users who have curl. – Source
GET
Let’s have a look how a simple (GET) API call looks like:
(let*
((httpRequest
(request "https://api.chucknorris.io/jokes/random"
:parser 'json-read
:sync t
:success (cl-function
(lambda (&key data &allow-other-keys)
(message "I sent: %S" data)))))
(data (request-response-data httpRequest)))
;; Print information
(cl-loop for (key . value) in data
collect (cons key value)))
Code Snippet 6: Get a random Chuck Norris joke
((categories .
[])
(created_at . "2020-01-05 13:42:19.576875")
(icon_url . "https://assets.chucknorris.host/img/avatar/chuck-norris.png")
(id . "YNmylryESKCeA5-TJKm_9g")
(updated_at . "2020-01-05 13:42:19.576875")
(url . "https://api.chucknorris.io/jokes/YNmylryESKCeA5-TJKm_9g")
(value . "The descendents of Chuck Norris have divided into two widely known cultures: New Jersey and New York."))
POST
Sending a POST
request is also an easy task:
(let*
((httpRequest
(request "http://httpbin.org/post"
:type "POST"
:data '(("key" . "value") ("key2" . "value2"))
:parser 'json-read
:sync t
:success (cl-function
(lambda (&key data &allow-other-keys)
(message "I sent: %S" data)))))
(data (request-response-data httpRequest))
(err (request-response-error-thrown httpRequest))
(status (request-response-status-code httpRequest)))
;; Print information
(cl-loop for (key . value) in data
collect (cons key value)))
Code Snippet 7: POST request with data
And here is the result:
((args)
(data . "")
(files)
(form
(key . "value")
(key2 . "value2"))
(headers
(Accept . "*/*")
(Accept-Encoding . "deflate, gzip, br, zstd")
(Content-Length . "21")
(Content-Type . "application/x-www-form-urlencoded")
(Host . "httpbin.org")
(User-Agent . "curl/7.83.1")
(X-Amzn-Trace-Id . "Root=1-62cdbc5c-52d3ad32436c1cb8778808e5"))
(json)
(origin . "127.0.0.1")
(url . "http://httpbin.org/post"))
Code Snippet 8: POST response as list of Elisp cons cells
Emacs
;; default tiddlywiki base path
(setq tiddlywiki-base-path "http://127.0.0.1:8181/recipes/default/tiddlers/")
GET tiddler
Let’s GET a tiddler:
(let*
((httpRequest
(request (concat tiddlywiki-base-path "Emacs")
:parser 'json-read
:sync t
:success (cl-function
(lambda (&key data &allow-other-keys)
(message "I sent: %S" data)))))
(data (request-response-data httpRequest))
(err (request-response-error-thrown httpRequest))
(status (request-response-status-code httpRequest)))
;; Print information
(cl-loop for (key . value) in data
collect (cons key value)))
Code Snippet 9: Get a tiddler by name ("Emacs")
((title . "Emacs")
(created . "20210623082136326")
(modified . "20210623082138258")
(tags . "Topics")
(type . "text/vnd.tiddlywiki")
(revision . 0)
(bag . "default"))
Code Snippet 10: Response as list of Elisp cons cells
PUT a new tiddler
Creating a new tiddler is also simple. Using ob-verb let’s add a PUT
request to the API:
PUT http://127.0.0.1:8181/recipes/default/tiddlers/I%20love%20Elisp
x-requested-with: TiddlyWiki
Content-Type: application/json; charset=utf-8
{
"title": "I love Elisp",
"tags": "Emacs [[I Love]]",
"send-with": "verb",
"text": "This rocks!"
}
Code Snippet 11: Sample request for creating a new tiddler
Check if tiddler was indeed created:
GET http://127.0.0.1:8181/recipes/default/tiddlers/I%20love%20Elisp
x-requested-with: TiddlyWiki
Accept: application/json; charset=utf-8
Code Snippet 12: GET request using verb
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 13 Jul 2022 10:03:27 GMT
Connection: keep-alive
Keep-Alive: timeout=5
Transfer-Encoding: chunked
{
"title": "I love Elisp",
"tags": "Emacs [[I Love]]",
"fields": {
"send-with": "verb"
},
"text": "This rocks!",
"revision": 1,
"bag": "default",
"type": "text/vnd.tiddlywiki"
}
Code Snippet 13: A new tiddler was created
Now let’s translate that to request.el
code. This I’ll some extra complexity: I’ll add a function (defun
) to PUT
a new tiddler for us, where name , tags and body of the tiddler are variable.
;; Define function for inserting new tiddlers
(defun insert-tiddler(name tags body)
(let*
(
(tiddler-title name)
(url-path (url-hexify-string tiddler-title))
(tiddler-tags tags)
(tiddler-body body)
(httpRequest
(request (concat tiddlywiki-base-path url-path)
:type "PUT"
:data (json-encode
`(
("title" . ,tiddler-title)
("created" . ,(format-time-string "%Y%m%d%H%M%S%3N"))
("modified" . ,(format-time-string "%Y%m%d%H%M%S%3N"))
("tags" . ,tiddler-tags)
("text" . ,tiddler-body)
("type" . "text/vnd.tiddlywiki")))
:headers '(
("Content-Type" . "application/json")
("X-Requested-With" . "Tiddlywiki")
("Accept" . "application/json"))
:encoding 'utf-8
:sync t
:complete
(function*
(lambda (&key data &allow-other-keys)
(message "Inside function: %s" data)
(when data
(with-current-buffer (get-buffer-create "*request demo*")
(erase-buffer)
(insert (request-response-data data))
(pop-to-buffer (current-buffer))))))
:error
(function* (lambda (&key error-thrown &allow-other-keys&rest _)
(message "Got error: %S" error-thrown)))
)))
(format "%s:%s"
(request-response-headers httpRequest)
(request-response-status-code httpRequest)
)))
;; Insert 2 tiddlers
(insert-tiddler "I love Elisp" "Elisp [[I Love]]" "This rocks!")
Code Snippet 14: Create new function for inserting new tiddlers
"((etag . \"default/I%20love%20Elisp/61:\") (content-type . text/plain) (date . Wed, 13 Jul 2022 12:30:33 GMT) (connection . keep-alive) (keep-alive . timeout=5)):204"
Code Snippet 15: New tiddler was created
Some explanations:
- in line 6 I URL encode the
tiddler-title
-
I love Elisp
should becomeI%20love%20Elisp
-
- in line 21 some headers are set
-
X-Requested-With
is required to be set toTiddlyWiki
-
Content-Type
should bejson
- we also accept
json
as a response
-
- in line 13 we specify the
data
to be sent to the API- each field (key, value sets) is set accordingly (see 10)
- I set the
created
andmodified
fields usingformat-time-string
Now let’s check again if tiddler really exists:
GET http://127.0.0.1:8181/recipes/default/tiddlers/I%20love%20Elisp
x-requested-with: TiddlyWiki
Accept: application/json; charset=utf-8
Code Snippet 16: Check if new tiddler exists
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 13 Jul 2022 12:40:22 GMT
Connection: keep-alive
Keep-Alive: timeout=5
Transfer-Encoding: chunked
{
"title": "I love Elisp",
"created": "20220713143033566",
"modified": "20220713143033566",
"tags": "Elisp [[I Love]]",
"text": "This rocks!",
"type": "text/vnd.tiddlywiki",
"revision": 61,
"bag": "default"
}
Code Snippet 17: It does exist!
Use cases
Now what can you do with this little custom functions? Let me share my use cases.
Add bookmark
A bookmark in my TiddlyWiki represents a tiddler of following format:
GET http://127.0.0.1:8181/recipes/default/tiddlers/chashell
Accept: application/json; charset=utf-8
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 13 Jul 2022 12:49:58 GMT
Connection: keep-alive
Keep-Alive: timeout=5
Transfer-Encoding: chunked
{
"title": "chashell",
"created": "20210519103441485",
"modified": "20210519103528982",
"fields": {
"name": "chashell",
"note": "Chashell is a Go reverse shell that communicates over DNS. It can be used to bypass firewalls or tightly restricted networks.",
"url": "https://github.com/sysdream/chashell"
},
"tags": "Golang Security Tool Bookmark",
"type": "text/vnd.tiddlywiki",
"revision": 0,
"bag": "default"
}
Every bookmarks consists of a name , a note and an url. Every tiddler supposed to be a bookmark is tagged by Bookmark
. In this chashell
is a tiddler and at the same time a bookmark in my wiki. As part of my daily routine, I go through my pocket entries and decide which ones I should bookmark in Tiddlywiki. These are my keybindings for the getpocket major mode:
(map! :map pocket-reader-mode-map
:after pocket-reader
:nm "d" #'pocket-reader-delete
:nm "SD" #'dorneanu/pocket-reader-send-to-dropbox
:nm "a" #'pocket-reader-toggle-archived
:nm "B" #'pocket-reader-open-in-external-browser
:nm "e" #'pocket-reader-excerpt
:nm "G" #'pocket-reader-more
:nm "TAB" #'pocket-reader-open-url
:nm "tr" #'pocket-reader-remove-tags
:nm "tN" #'dorneanu/pocket-reader-remove-next
:nm "C-b" #'dorneanu/tiddlywiki-add-bookmark
:nm "ta" #'pocket-reader-add-tags
:nm "gr" #'pocket-reader-refresh
:nm "p" #'pocket-reader-search
:nm "U" #'pocket-reader-unmark-all
:nm "y" #'pocket-reader-copy-url
:nm "Y" #'dorneanu/pocket-reader-copy-to-scratch)
Let’s have a look at dorneanu/tiddlywiki-add-bookmark
:
(defun dorneanu/tiddlywiki-add-bookmark ()
"Adds a new bookmark to tiddlywiki. The URL is fetched from clipboard or killring"
(require 'url-util)
(interactive)
(pocket-reader-copy-url)
(setq my-url (org-web-tools--get-first-url))
(setq url-html (org-web-tools--get-url my-url))
(setq url-title (org-web-tools--html-title url-html))
(setq url-title-mod (read-string "Title: " url-title))
(setq url-path (url-hexify-string url-title-mod))
(setq url-note (read-string (concat "Note for " my-url ":")))
(setq url-tags (concat "Bookmark "(read-string "Additional tags: ")))
(request (concat tiddlywiki-base-path url-path)
:type "PUT"
:data (json-encode `(("name" . ,url-title-mod) ("note" . ,url-note) ("url" . ,my-url) ("tags" . ,url-tags)))
:headers '(("Content-Type" . "application/json") ("X-Requested-With" . "TiddlyWiki") ("Accept" . "application/json"))
:parser 'json-read
:success
(cl-function
(lambda (&key data &allow-other-keys)
(message "I sent: %S" (assoc-default 'args data))))
:complete (lambda (&rest _) (message "Added %s" (symbol-value 'url-title-mod)))
:error (lambda (&rest _) (message "Some error"))
:status-code '((400 . (lambda (&rest _) (message "Got 400.")))
(418 . (lambda (&rest _) (message "Got 418.")))
(204 . (lambda (&rest _) (message "Got 202."))))
)
)
Code Snippet 18: Bookmark entries from getpocket directly into Tiddlywiki
Add quote
After reading each book I usually do some post-reading/post-processing. While I could use the Tiddlywiki web interface to add new tiddlers, I’d rather do it from Emacs directly.
Often I need to insert new quotes from book (or web articles). How to I do this:
(defun dorneanu/tiddlywiki-add-quote ()
"Adds a new quote"
(interactive)
(setq quote-title (read-string "Quote title: " quote-title))
(setq url-path (url-hexify-string quote-title))
(setq quote-source (read-string (concat "Source for " quote-title ": ") quote-source))
(setq quote-body (read-string (concat "Text for " quote-title ": ")))
(setq quote-tags (concat "quote "(read-string "Additional tags: ")))
(request (concat tiddlywiki-base-path url-path)
:type "PUT"
:data (json-encode `(
("title" . ,quote-title)
("created" . ,(format-time-string "%Y%m%d%H%M%S%3N"))
("modified" . ,(format-time-string "%Y%m%d%H%M%S%3N"))
("source" . ,quote-source)
("tags" . ,quote-tags)
("text" . ,quote-body)
("type" . "text/vnd.tiddlywiki")))
:headers '(("Content-Type" . "application/json") ("X-Requested-With" . "TiddlyWiki") ("Accept" . "application/json"))
:parser 'json-read
:success
(cl-function
(lambda (&key data &allow-other-keys)
(message "I sent: %S" (assoc-default 'args data))))
:complete (lambda (&rest _) (message "Added quote <%s>" (symbol-value 'quote-title)))
:error (lambda (&rest _) (message "Some error"))
:status-code '((400 . (lambda (&rest _) (message "Got 400.")))
(418 . (lambda (&rest _) (message "Got 418.")))
(204 . (lambda (&rest _) (message "Got 202."))))
)
)
Code Snippet 19: Directly add new quotes from Emacs
I simply invoke M-x dorneanu/tiddlywiki-add-quote
and read-string
will ask for a quote title, some source of the quote (e.g. a book) and of course the actual text.
Hydra
I’ve recently discovered hydra and I came up with some hydra also for TiddlyWiki:
(defhydra hydra-tiddlywiki (:color blue :hint nil)
"
Tiddlywiki commands^
---------------------------------------------------------
_b_ Add new bookmark
_j_ Add new journal entry
_t_ Add new tiddler
_q_ Add new quote
"
("b" dorneanu/tiddlywiki-add-bookmark)
("j" vd/tw5-journal-file-by-date)
("q" dorneanu/tiddlywiki-add-quote)
("t" dorneanu/tiddlywiki-add-tiddler))
;; Keybindings
(my-leader-def
:infix "m w"
"h" '(hydra-tiddlywiki/body :which-key "Open Tiddlywiki hydra")
"j" '(vd/tw5-journal-file-by-date :which-key "Create/Open TW5 Journal file")
"s" '(my/rg-tiddlywiki-directory :which-key "Search in TW5 directory"))
Code Snippet 20: Hydra for Tiddlywiki
This way I press M m w h
and the TiddlyWiki hydra will pop up.
Conclusion
I hope some day there will be a full (elisp) package for TiddlyWiki combining some of the functionalities/ideas mentioned here. If you have anything to add/share, please let me know.
Top comments (0)