Updating a Bokeh Graph in a Django Website

With our first simple graph on an existing Django page, the challenge is to update it when the data on the page updates.

To recap: I have a Django page, on it is a table of data, and I have added a Bokeh histogram that summarises the frequency distribution of one of that table's columns. To get there we:

1. Added some Bokeh JavaScript and css links to our Django template
2. Installed Bokeh on the webserver, added the BokehApp to Django's INSTALLED_APPS.
3. In the Django view (which loads and renders the template) created the graph (first a figure, then vbars, then the div and script components to add to template,
4. Added the div and script components to our Django template.

And we have a simple histogram on the page. Nice.

But this page, allows users to specify a number of filters, and click a refresh button, which requests new data (with these filters applied, using an AJAX request) and updates the table.

It now needs to update the graph as well!

Updating the data in a Bokeh graph (the options)

Turns out, there is no easy, or obvious way to do that. Bokeh is full of features, and can update all right, any number of ways, but not this way, not easily, not in any way I could find in the documentation. What options does it provide? At least these:

1. Run a Bokeh server. This is a server you can run which can run python code of your wanting. It's easy to run: bokeh serve --show myapp.py (though on a live webserver would need to be installed as a service and whatever port(s) - port 5006 by default - it uses to communicate with Bokeh in the client must be opened on the firewall, the gateway and any reverse proxy etc.). In short, this is a very nice prospect and feature as it communicates over a websocket which has a number of advantages over AJAX. But, all the same, it's overkill for this trial. And parked. What I want is a way for my client side JavaScript, when the AJAX call completes and a response is received, to update the Bokeh Graph. I don't want a whole new piece of infrastructure (just yet, not on the first date).
2. Using BokehJS. Looks very promising. Basically a JavaScript interface to Bokeh (✔️). But the documentation is discouraging for this trial. The documentation boldly warns: The BokehJS APIs is still in development and may undergo changes in future releases. and all the examples it provides basically see the three key Python steps we used (creating a figure, vbars and components) moving entirely client side into JavaScript. That has some appeal too. Alas it totally does not explain if and how one might, using JavaScript, provide new data to a Bokeh graph created and provided by server side Bokeh. So this too we park for now, and bookmark the idea for later. For this trial, I am seeking a way to update the graph we created in part 1.
3. Configuring the graph to use an AjaxDataSource. Rather than providing two lists of numbers as a data source for the graph, we can provide it with a URL and a polling interval. And it will pool back checking for new data to that URL. But no, oh so close, but not what I want. I don't want polled AJAX calls, I already have a user-requested AJAX call (filter criteria changed, new data requested). So this too gets parked. Not a bad feature for live updating of data (though the Bokeh Server is probably better) but not good for this trial.
4. Using a CustomJS callback. This is another brilliant feature. Server side we can write some custom JS and attach it to a client side element to call back to, on given events, like for example data changing! Yes. And no .... Alas It can be attached to a whole range of Bokeh events, but not to our event (the click-handler for my button), the one that triggers an AJAX call to get new data and update the table. That button! I can't attach custom JS to that button and its click event, only to Bokeh widgets that have the js_on_change method (with which the callback is registered with that widget). Again, ever so close, but no prize. That said, we will come to this one later in this trial, it does have a use, even in this trial (clue: we can use it to register a JavaScript callback that adjusts the axes, ticks and labels, which is called when the data on the vbars is changed - but we still have to find a way to change that data, client-side).

And with that (four exhaustively research options parked), we throw our hands up in the air, file a Stack Overflow plea for help, and hope for the best. But, no answers arriving in any hurry and wanting to soldier on, we are left with no options but to reverse engineer something (which is technical jargon for working out how to do something in lieu of documentation or mentorship)

Reverse engineering Bokeh (client-side)

Scratching my head, I was wondering "if, client-side, I have new data from AJAX call to my server, and I have a Bokeh histogram, how can I tell that histogram that I have new data and give it that data?" That was the crux of the problem that, in all my reading, and research, to that point, I had not found an answer to.

The first, baby-step, in reverse engineering, of course, is to take a look at the code. Fortunately, in nay modern browser, I can now press F12 and up propose the source code for the page and a wonderful suite of debugging tools. My, how easy things have become since the 1980s when I first started reverse engineering games by disassembling them to strip the copy protection from them (looks furtively around the room for any disapproving glares, and hopes the statute of limitations is in full swing).

Now if you recall, a web page with a Bokeh graph on it has a div and script provided by bokeh.embed.components and injected into the page using our Django template.

Step 1: That div

Here's what the div looks like:

<div class="bk-root" id="591aa428-0ac3-4085-938e-481caf339309" data-root-id="1233">

wherein, we see our first two clues. Clearly an id that the Bokeh JavaScript library will use to find the div (and I'm betting at this point the script contains a reference to it) and a data-root-id that sounds a lot like a lead in our hunt for a way to furbish this div with new data!

Searching that page for that id surprise, surprise it crops in the script, under:

const render_items = [{"docid":"e71600f9-95b3-4b84-9210-eff6b3036e17","root_ids":["1233"],"roots":{"1233":"591aa428-0ac3-4085-938e-481caf339309"}}];

Where we see the data-root-id and id bound once again.

Step 2: That script

The script itself yields none more clue. It starts with:

<script type="text/javascript">
    (function() {
          const fn = function() {
            Bokeh.safely(function() {

which tells us, there's a JavaScript object called Bokeh, that we might interface with. And what better way to explore that object than to use the browsers debugger.

Step 3: That Bokeh object

Perhaps the most straightforward way to catch JavaScript in action, after the whole page has loaded and everything has rendered and, it's in a clear run state is to set an Event Listener Breakpoint on the mouse-click event.

Doing that, and then clicking on the Bokeh graph (I elected to click on the Refresh Tool button that the graph provides) breaks in the debugger, and there's a wonderful list of the variables, and under Global we see the Bokeh object.

By expanding it and looking at all its many many properties with patience a small clue might be found. But we might find faster clues by looking at and following the code.

We broke in an event handler:

function documentClick(e) {

to be exact. F9 is the debugger's Step key, so pressing F9 a few times and watching where we go may yield fruit. In fact, 11 F9 presses and we find outselves here:

this.throttled_paint = (0, throttle_1.throttle)(() => this.repaint(), 1000 / 60);

And that, is why I thought clicking the refresh Tool button might be a fine idea. There it is. this.repaint. But what is this?

Easy, we just type this into the console and see that it's:

PlotView {removed: Signal0, _ready: Promise, _slots: WeakMap, _idle_notified: true, model: Plot, …}

Bingo!

Step 4: That PlotView

Now PlotView sounds good. In the console in fact I type this. and after teh . the debugger intellisense predictor throws up a list of all the attributes of that object. I flick down them, and there's one that catches my eye model.

So this.model. and voila it has an id, this.model.id and it is ... 1233! yes, that is the data-root-id of our div.

Getting warm now.

Looking at the Bokeh object I play hte same game. Bokeh. throws up a long list of attributes again, and I page down, and one catches my eye again. It is documents. Bokeh.documents seems to be an Array with one entry so Bokeh.documents[0]. lets us walk through the attributes of a document and voila, there is Bokeh.documents[0]._all_models and it has an interesting value:

Map(32) {"1233" => Plot, "1256" => Toolbar, "1250" => PanTool, "1251" => WheelZoomTool, "1252" => BoxZoomTool, …}

32 entries and the first one is our plot. So let's try:

Bokeh.documents[0].get_model_by_id(1233)

no luck. How about:

Bokeh.documents[0].get_model_by_id("1233")

and that looks good, it's a Plot object so lets try:

Bokeh.documents[0].get_model_by_id(1233) == this.model

which is true! So we have a way of getting models in Javascript!

Step 5: Finding the data source

Alas, the Plot object has no eaisly found data source. Poking around inside of it I do find it has renderers again a list with one entry, and so exploring this.renderers[0] I see it has a data_source property, which in turn has a data property and, you get the idea ... this.model.data_source.data is on the money:

this.model.renderers[0].data_source.data
{top: Array(12), x: Array(12)}

And top and x are the very thinhgs I provided to figure.vbar at the server end and each of these 12 value Arrays are the very lists I provided. Found!

Step 6: That data-root-id, again

Recall, that in my Django view I built the graph as follows:

from bokeh.plotting import figure
from bokeh.embed import components

def view_Events(request):
    # Collect the categories and values
    (players, frequency) = Event.frequency("players", events)

    # Create the figure
    plot = figure(height=350,
                  x_axis_label="Count of Players",
                  y_axis_label="Number of Events",
                  background_fill_alpha=0,
                  border_fill_alpha=0,
                  tools="pan,wheel_zoom,box_zoom,save,reset")

    # And the bars
    bars = plot.vbar(x=players, top=frequency, width=0.9)

    # Build the context variables
    graph_script, graph_div = components(plot)

    # Add them to context
    context = {"graph_script": graph_script,
               "graph_div": graph_div}

    # Render the view
    return render(request, 'events.html', context=context)

I checked plot and sure enough it has an id. And plot.id here is exaclty the same value that appears in the data-root-id of the div and is the id of the Plot object that was repainting itself when we walked through the click handler. The threads come together.

Changing the data source

Sure enough if I pass plot.id into context of the template and use client side it can find the Plot using that id.

And so on the client side I have:

const plot = Bokeh.documents[0].get_model_by_id(plotid)
const source = plot.renderers[0].data_source
source.data.x = players;
source.data.top = frequency; 

where plotid came in during page load via the Django template context, and players and frequency are delivered by the AJAX call.

And that sort of works. The values are set.

But the graph does not change. Hmmmm

Step 7: What's wrong?

I can see the data source is changed. Why hasn't the graph changed? I'm missing something.

Scratching my head a little, I think, the missing link must be that repaint command. That is, we have given the Plot new data, but we need to tell it that it has new data, tell it that it needs to redraw itself somehow.

So I break in the code above and inspect the source variable for clues. And the very first attribute on its list is change, which rings a loud bell. It has the value:

Signal0 {sender: ColumnDataSource, name: "change"}

And so is a singal of some sort, that we might send to the Plot. Inspecting the attributes of the signal we find it has (4th on its list), an emit method.

So I add:

source.change.emit(); 

to code above to get:

const plot = Bokeh.documents[0].get_model_by_id(plotid)
const source = plot.renderers[0].data_source
source.data.x = players;
source.data.top = frequency; 
source.change.emit(); 

And voila, now the graph updates. We hit paydirt!

Step 8: Do we really need renderers?

Not real comfortable with the use of renderers and at this stage seeing that there are a lot of models ...:

Bokeh.documents[0]._all_models
Map(32) {"1233" => Plot, "1256" => Toolbar, "1250" => PanTool, "1251" => WheelZoomTool, "1252" => BoxZoomTool, …}

32 models and among them when I expand the list are Vbars!

So I try passing not plot.id into the context, but bars.id and the AJAX response code becomes:

const bars = Bokeh.documents[0].get_model_by_id(barsid)
const source = bars.data_source
source.data.x = players;
source.data.top = frequency;
source.change.emit();

And it works beautifully. No renderers needed. The connection is clear now too. At the python side I pass the x/y data to vbar() and at thye JavaScript side I get the vbar and change it's data source and emit a change signal ...all works wonderfully.

The only thing I remain mildly uncomfortable about is the documents[0]. I'm stil not clear what documents are, when there might be more than one and so on.

Car vectors graciously created and made available by macrovector - www.freepik.com