Stupid J Jupyter Tricks

Cross posted from my blog Analyze the Data not the Drivel

Continuing with my software irritant dispatching theme: today's
irritant - including syntax colored code in blogs, LaTeX, Word, and other documents. This one is a persistent pain in the ass; especially for users of idiosyncratic programming languages. We all have our favorite hacks and workarounds; today I am sharing one of mine: using jupyter to syntax color J code.

A few years ago Martin Saurer made an outstanding contribution to the J programming language community. He created a jupyter kernel for J. If you don't know what jupyter is please crawl out from under your rock and join the rest of us in the 21^st century.

As I have said before jupyter notebooks are simply the correct way to present complex "executable" technical documents that contain heavy doses of mathematics, program code, and graphics. I'm not the only one that holds this opinion: many concur. It would be an understatement to say that jupyter has been a success. It's now a dominant tool in many disciplines. If you're not using it you are missing out on one of the most useful open-source programming tools available.

I came to jupyter from the pythonverse. The JSON based ipynb
notebook format originated there and initially stood for Interactive PYthon Note*Book, but in one of the best software design decisions ever made, the creators of jupyter decided to make it *programming language agnostic. This means any programming language can plug into jupyter if it implements a well-defined kernel interface. This is what Martin implemented. Now, many programming languages support jupyter. It's become an expected facility, and if it's missing, well that's a good reason to use another programming language.

Martin's J kernel works very well and I use it all the time, but there are a few ways the J kernel, and jupyter kernels in general, could be improved.

Debugging notebook code

Debugging notebook code is not as polished as it could be. Kernels are typically "invisible" executables that respond to requests sent from the notebook server. Most kernels stay in the background and do not provide standard language-specific programming environments. Usually, this isn't much of a problem, but when debugging complex programs it would be handy to push a button and pop up a standard debugger. In J's case, a JQt or JHS session would appear containing all the code loaded or created by
the notebook. You could then debug and modify your code, then pop over to the notebook session to try it out.

Right now the J kernel does not "pop up" visible J sessions but you can achieve something similar by extracting J notebook code as a J script and executing it in another JQt or JHS session. jupyter makes this easy!

Stupid J Jupyter Trick #1 - export notebook code as a J script

jupyter can download notebooks in a variety of formats. One of the
formats provided by the J kernel is ijs or J script. Go to the
File menu, select Download as, then pick J (.ijs). jupyter will extract all the J code cells in the notebook in cell order and write a script file to wherever you want it. I usually save such scripts in the J temp directory jpath ~temp where they can be conveniently loaded and run.

When you run a downloaded J notebook script in another J session it essentially¹ recreates the notebook kernel state.

Inserting J code into notebooks

Extracting J code from notebooks for debugging is useful but oddly the inverse, inserting J code into notebooks, is even more useful. As previously noted, jupyter can export notebooks in many formats. In particular, it can generate HTML, LaTeX, Markdown, and PDF versions of notebooks. I've found the generated LaTeX, HTML, and Markdown formats extremely handy.

Stupid J Jupyter Trick #2 - J code notebook insertion

If you're looking for a quick and dirty way to syntax color J code do the following:

1. Create a blank jupyter notebook.

2. Open a J script in your favorite editor.

3. Cut and Paste the script into a jupyter code cell.

4. Download the notebook as HTML or PDF.

jupyter will nicely syntax color your code. It even does a decent job of breaking lines. The GitHub files brandxmp.ijs.pdf and brandxmp.ijs.html are examples of what you get right out of the box.

Stupid J Jupyter Trick #3 - smarter J code notebook insertion

The quick and dirty method is simple and handy but it's of limited use when preparing complex LaTeX documents. With LaTeX, you want more than just code. You want cross-references, web links, footnotes, mathematical expressions, graphics, indexes, appendices, and other literate programming goodies. It's much easier to achieve such ends if your J code is split over many cells. The following verb does just that.

ipynbfrjod=:3 : 0

NB.*ipynbfrjod v-- extract J  words from JOD  and insert in blank
NB. jupyter notebook.
NB.
NB. monad:  clIpynb =. ipynbfrjod blclNames
NB.
NB.   NB. examples use docs and utils
NB.   require 'general/jod'
NB.   od ;:'docs utils'
NB.
NB.   nbj=: ipynbfrjod ;:'sha1 sha1dir'
NB.   nbj write 'C:\Users\baker\jupyter_notebooks\test0.ipynb'
NB.
NB.   nbj=: ipynbfrjod }. grp 'ipynb'
NB.   nbj write 'C:\Users\baker\jupyter_notebooks\ipynb_onself.ipynb'

NB. require 'general/jod' !(*)=. disp
jc=. disp&.> y

NB. markdown sections with word name
sec=. dblquote (<MDSECTION,JWORDMARK) ,&.> y
sec=. (<NBJCELLBEGst) ,&.> sec ,&.> <NBJCELLBEGen

NB. j code to quoted list of python strings notebook format
nbj=. <;._2@(REVPYESCAPECHRS&changestr)@tlf&.> jc
nbj=. ;&.> '"' , L: 0 (<'\n",',LF) ,~ L: 0 nbj
nbj=. ,&'"'&.> '\n",'&beforelaststr&.> nbj
nbj=. sec ,&.> nbj ,&.> <NBJCELLEND
toJ NBHEADER , (LF ,~ ','&beforelaststr ;nbj) , NBTRAILER
)

After loading J words into cells all the jupyter export facilities are immediately available. Examples of jupyter outputs exploiting this hack are available on GitHub: see ipynb_onself.ipynb and ipynb_onself.pdf.

To use ipynbfrjod download ipynb.ijs and the corresponding document ipynb.pdf. All these files are on GitHub. Help yourself!

Postscript: If you already have a recent version of J installed you can directly install this script as an addon by typing the following command in a JQt or JHS session.

install 'github:bakerjd99/jackshacks'

NB. load script
load '~addons/jacks/ipynb.ijs'

1. Some care must be taken when developing J jupyter notebooks to guarantee exported scripts accurately recreate kernel states. ↩