Developing a keyboard heatmap and customizing keys with Karabiner

For the last 5 years, I've been using an Apple Magic Keyboard. There wasn't any major reason to be using it, except having the same layout of my laptop, but it did the job pretty nicely.

After joining Finiam, I started looking more into how I used my keyboard, and how I could become more efficient with it. A thing that was bothering me for some time was the fact that my keyboard had a Portuguese layout, and every time I wanted to make {, ] characters, Finger Gymnastics 101, would be required. I know I could always change the input language, but it wasn't the same thing. This was just the excuse I needed to get a new keyboard and with a US layout. So I went by and bought a Keychron k2v2 and set myself up to start looking for more ways of having it optimized for me.

The first thing that I did was installing Karabiner-Elements. It's a keyboard customizer for macOS, where you can easily swap a key with another, use a base set of complex modifications developed by the community or create your own rules. I started investigating and trying new combinations... But then it came to my mind:

"Wouldn't it be nice if I could make a heatmap of my keystrokes and then create aliases or easier configurations for things that I identify as the most used?"

Well, of course, it would, and I can do it as a nice excuse to play with Elixir again. So let's get our hands dirty and get into some code.

Developing the Heatmapper

First, we need to collect our keystrokes. This would be a nice challenge to do, but I'll leave it on the side for now. So, I started looking into existing keyloggers in Github (Yeah, I know this sounds a bit weird because the chosen one could be malicious - I'm aware of that possibility). After a few possibilities, I found out caseyscarborough/keylogger, went through the code and it seemed simple to use and make changes to adjust it for my specific needs. Now let's dive into the heatmap and drawing part.

From the beginning I wanted the heatmapper to draw the keyboard and not use a base SVG layout (or something similar). Zamith's mogrify_draw enables me to draw keyboard keys in a simple way using mogrify as the base.

Another thing was that since the code will handle drawing a keyboard, it would be an even cooler solution to be able to draw any keyboard, given the provided structure of the keys.

The first step is to parse the log file of keystrokes provided. As I mentioned earlier, I made some changes to the keylogger and this version can be found here. You can then follow the instructions in the README on how to use it, but it outputs to the file the following on each keystroke read:

<keycode>::<keyname>

I'll probably remove keyname in the future, as I'm not using it in the application itself (just for debugging purposes), and therefore reduce the size of the log file.

Looking, at the code, we first open the file and then parse its content to generate a map of frequencies. In the parse_line/1 function, it uses a regex to get the keycode in each line.

{:ok, contents} <- File.read(path)
frequencies = parse(contents)

def parse(contents) do
  contents
  |> String.split("\n", trim: true)
  |> Enum.map(&parse_line(&1))
  |> Enum.reject(&is_nil(&1))
  |> Enum.frequencies()
  |> normalize_frequencies
end

The normalize_frequencies/1 function, calculates the maximum value of keystrokes (the max frequency), and then for each value in the frequencies map, returns a normalized value: value/max. See the example below:

# log file contents
31::o
45::n
8::c
31::o

# frequencies map after parse
frequencies = %{ 31: 2, 45: 1, 8: 1 }

# frequencies normalized
max_frequency = 2
normalized = %{ 31: 2/2, 45: 1/2, 8: 1/2 }

Now that we have normalized values, we can jump to the drawing process. The first thing to do here is to implement the logic for several keyboard layouts to be drawn, so together with the path to the log file, it's also necessary to pass which keyboard type to draw.

For organizational purposes, I decided to divide the layouts by keyboard brand first, and then by type. For example, below we have the types :keychron, :macbook and :niz.

def get_keyboard_layout(model, keyboard) do
  case model do
    :keychron -> 
      keyboard_layout(Layouts.Keychron, keyboard)
    :macbook -> 
      keyboard_layout(Layouts.Macbook, keyboard)
    :niz -> 
      keyboard_layout(Layouts.Niz, keyboard)
    _ -> {:error, :no_model}
  end
end

$> get_keyboard_layout(:keychron, :k2v2)

Calling the function with a valid model and type returns the keyboard layout. Each layout is under layouts/<model>.ex, and consists of a function with the name of the keyboard type and the following structure:

# file: layouts/keychron.ex
def k2v2 do
  [
    [
      %{size: 1, keycode: 53, name: "esc"},
      ...
    ],
    [
      %{size: 1, keycode: 50, name: "`~"},
      ...
    ],
    ...
  ]
end

It's basically a list of rows of the keyboard, where each row, the list of keyboard keys is defined. A keyboard key can be defined with keycode, name, and size (assuming that a size of 2, will draw a key with a width of 2 * DEFAULT_KEY_PX_SIZE and a fixed height of DEFAULT_KEY_PX_SIZE). It is also possible to define a height and width instead of size if we wish to draw keys with different heights.

Now let's see how everything is drawn.

The first step is using mogrify_draw to paint the base image where everything will be laid next. It'll create a png with the respective width and height of the keyboard we want to draw.

defp draw_base_image(keyboard) do
  %Mogrify.Image{path: "heatmap.png", ext: "png"}
  |> custom("size", "#{95 * keyboard_width}x#{95 * keyboard_height}")
  |> canvas("white")
end

To draw each keyboard key, we go through each row, and subsequently across all row's keys, keeping track of the xx_position and yy_position that we are at the moment. With this information, we know the exact place in pixels to start drawing the key in the base image. But before actually drawing, we get the respective frequency for that key and the RGB color calculated accordingly to the frequency passed. Then, we draw a rounded rectangle (the key) and a white text over it (the key name).

defp draw_key(key, xx_position, yy_position, image, frequencies) do
    freq = frequencies[to_string(key.keycode)] || 0
    {red, green, blue} = get_rgb_color(freq)

    image
    |> custom("fill", "rgb(#{red},#{green},#{blue})")
    |> rounded_rectangle(
      xx_position * @key_size + 10,
      yy_position * @key_size + 10,
      xx_position * @key_size + @key_size * key_width(key),
      yy_position * @key_size + @key_size * key_height(key),
      10,
      10
    )
    |> custom("fill", "white")
    |> Mogrify.Draw.text(
      xx_position * @key_size + 45,
      yy_position * @key_size + 45,
      key.name
    )
  end

Looking at the function rounded_rectangle it's derived from the ones enabled by mogrify_draw, but using the roundRectangle primitive from the -draw given by the mogrify command-line tools (check here all the primitives from mogrify if you need something more custom). For this primitive, it's necessary to pass the base image, the upper left and bottom right point, together with the border-radius information.

defp rounded_rectangle(
       image,
       upper_left_x,
       upper_left_y,
       lower_right_x,
       lower_right_y,
       border_w,
       border_h
     ) do
  image
  |> custom(
    "draw",
    "roundRectangle #{
      to_string(
        :io_lib.format("~g,~g ~g,~g ~g,~g", [
          upper_left_x / 1,
          upper_left_y / 1,
          lower_right_x / 1,
          lower_right_y / 1,
          border_w / 1,
          border_h / 1
        ])
      )
    }"
  )
end

After recursively going through all the rows and keys, the image heatmap.png is created.

• Grey keys are keys that weren't used (or the keycode is not being detected by the keylogger.)
• Stronger blue keys are the ones used with less frequency.
• Mix of blue and red are the keys with mid-frequency.
• Red keys the most used ones.

You can check the complete code and more detailed instructions on how to run it, in this Github repo.

Looking at the final result of a week of work we can identify some keys that are heavily used, such as the arrow keys, left command, left shift, left control, and even the backspace. It's somewhat a good representation of my usage as the referred keys take part in most of the keybindings that I use for work (except the backspace - not sure of what I delete so much 😅).

Creating custom key modifications with Karabiner-Elements

As I mentioned earlier, initially I made some simple modifications with Karabiner, but now we are in the condition of making some more complex ones, that are not provided by the community.

To create custom modification rules, we need to create a file under ~/.config/karabiner/assets/complex_modifications/ where you can have only one file with all the rules or group them in different files. After having valid rules they will appear as available to enable in the Complex modifications tab.

As an example, below, I'm creating a simple rule to change the caps_lock key to be a control key and just be a caps_lock when pressed together with shift.

karabiner/assets/complex_modifications/capsToControl.json

{
  "title": "Change caps key",
  "rules": [
    {
      "description": "Change caps_lock key to control. (Use shift+caps_lock as caps_lock)",
      "manipulators": [
        {
          "type": "basic",
          "from": {
            "key_code": "caps_lock",
            "modifiers": {
              "mandatory": [
                "shift"
              ],
              "optional": [
                "caps_lock"
              ]
            }
          },
          "to": [
            {
              "key_code": "caps_lock"
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "caps_lock",
            "modifiers": {
              "optional": [
                "any"
              ]
            }
          },
          "to": [
            {
              "key_code": "right_control"
            }
          ]
        }
      ]
    },
  ]
}

To see more on how to write these rules, you can the Karabiner Configuration Reference Manual or use this online editor to generate them.

Wrapping up

There's still some stuff to do to make the heatmap more accurate and robust, such as making it work for more complex keyboard layouts, like split keyboards or with vertical gaps between keys. But I'll leave it for later when I have the time.

This was a fun and simple project to try some ideas and make something generic that could be used by other people. The experiments with Karabiner-Elements weren't all described here, but they were about trying more multiple key combinations, using only keys that I don't find necessary for my daily use, such as the home, end, page down and page up.

Hope you also liked reading it and that you give it a try.

Have a nice one and see you in the next post 👋