Managing Windows Within mruby Part final: full encapsulation of state

The color is mauve

One trivial realization I had regarding the bike shed. Before I had the pattern of naming C functions bound to Ruby methods like class_name_method_name_method. While this makes sense I think it's better naming the functions like ClassName_method_name instead. This make it more visually distinct and easier searching for it. I've changed to that naming scheme here.

Also a little debugging trick when loading code externally into the Ruby VM with functions like mrb_load_string. After the function call you can check for failure by checking if the VM has an unhandled exception, like so

mrb_load_string(mrb, "foo");
if(mrb->exc) {
mrb_print_error(mrb);
}

Now that I've established that the bike shed is mauve today, lets establish where we're at and what we're doing to finish off this exploratory code. Right now we have an instance of the mruby VM running and can create Windows windows calling into the VM. The Ruby methods implemented in C use the Win32 API to create new windows. These are remarkable achievements and we still have some polish left. We want everything possible within the VM. Enumerated, here's the steps we'll take

1. Change the create method to a constructor
2. Add add_window and remove_window class methods on W32Window. These keep track of the number of open windows. Set g_should_quit after the last window closes.
3. In the new W32Window constructor register itself to the open windows using the new W32Window.add_window method.
4. Perform a three step procedure registering arbitrary data with a Windows window. We'll assign the object_id of the W32Window instances to the Windows window. This is the most complicated part of the program
5. Change the callback function for processing Windows messages to call W32Window.remove_window when the window closes. When there are no more windows left, the callback function requests the program quits.

Creating a constructor method in C with mruby

This is straightforward. Just as if we're working entirely within Ruby we define an initialize instance method which the VM calls when we call new. Lets rename the original w32_window_create_method to W32Window_initialize and change mrb_define_class_method to mrb_define_method appropriately

RClass *W32Window = mrb_define_class(mrb, "W32Window", mrb->object_class);
mrb_define_method(mrb, W32Window, "initialize",
                        W32Window_initialize, MRB_ARGS_REQ(2));

Then change mrb_load_string to create an instance instead of calling create

mrb_load_string(mrb, "5.times { |n| W32Window.new \"Window #{n}\", width: n * 100, height: n * 100 }");
if(mrb->exc) {
  mrb_print_error(mrb);
}

Running this works exactly as before. The difference is we now have an instance backing each window. This allows us in the near future to match a Windows window with a Ruby object instance later.

The function prototypes for instance variable manipulation are in the variable.h header. Add that include at the top of the file. We're using the mrb_iv_set function, which is the equivalent of instance_variable_set :@foo, 'bar'. It's definition

mrb_iv_set(mrb_state *mrb, mrb_value obj, mrb_sym sym, mrb_value v)

The only new thing here is the mrb_sym type. It's actually a typedef of uint32_t. Every symbol in Ruby is an index to a global hashed symbol table. The integer value is there result of the hash.

There are handy C functions for creating and finding symbols. The easiest one for our case is mrb_intern_cstr. The documentation implies that it creates a symbol. This function actually finds an existing symbol based upon the C string provided or creates a new symbol in the VM and returns a new symbol.

Putting it all together, right after extracting the arguments

mrb_iv_set(state, self, mrb_intern_cstr(state, "@window_name"), window_name);

Lets validate that this works by changing up the mrb_load_string line a bit

mrb_load_string(mrb, "1.upto 5 { |n| puts W32Window.new(\"Window #{n}\", width: n * 100, height: n * 100).inspect }");

One special thing about creating an instance. Currently we are returning mrb_nil_value(). While this is currently working the new method actually returns nil instead of the new instance. Change return mrb_nil_value() to return self.

Populating a Win32 window with a Ruby Object ID

Now we create instances of objects in the VM for each Win32 window we create. There is a disconnect between our instance and the window callback function. When Windows send a message to our window our callback function main_window_callback runs with the message passed into it. What we don't know is which Ruby object that window belongs to. We do know the HWND of that window. With that we could create a table in the VM of HWND objects to W32Window instances. However we have a shorter path available. Windows supports passing an arbitrary pointer upon creating the window. You can also set an arbitrary pointer on the window after it's finished creating. Microsoft has a detailed article explaining the procedure.

Notice the important distinction. CreateWindowEx initiates the window creation. We can pass an arbitrary pointer to CreateWindowEx as the last parameter, but we cannot set the arbitrary pointer on the window yet. For that, when Windows finishes constructing the window and just before presenting it, it sends the window the WM_CREATE message, with the LPARAM in the callback function set as the pointer we specified in CreateWindowEx. This means setting a pointer to our Ruby object on the window is a two step process. First, get the C pointer to self and pass it as the last parameter to CreateWindowEx. Then, in our callback function, intercept the WM_CREATE message. Cast the LPARAM back to the Ruby pointer, then use SetWindowLongPtr to store the pointer along with the window so we can get it back later.

This is the gnarliest bit of code we'll ever get into so I want to be sure we're all understanding the different components involved. We cannot have full control over the window and events, Windows controls that. Windows notifies us when things happen through sending messages to our callback function. Our task is matching up the window messages with data inside the VM. In particular we want the VM to maintain a list of the open windows. When the list is empty, we want to notify Windows to terminate the program. Each time we create a window within the VM we add to the list. When Windows notifies us it destroyed a window we remove it from our list.

When Windows calls our callback function we're in the C. What we need now is a pattern in C for locating objects in the VM from outside the VM. The first intuition is get a C pointer to the live objects inside the VM. It's not recommended we keep a C pointer to VM objects. That pointer can become invalid for reasons outside our control resulting in a hard crash following a wild pointer. The object it points to can get garbage collected if we're not careful. The VM may move data around in memory, like the Ruby 2.7 heap compacting feature. What we do have that's guaranteed stable and unique for identifying objects within the VM is the object_id of every object. We'll use the object_id, a 64 bit unsigned integer.

Keeping track of all open windows within Ruby

To get started lets add a bit of Ruby code. We'll have the W32Window class keep track of all of it's instances.

RClass *W32Window = mrb_define_class(mrb, "W32Window", mrb->object_class);
const char* ruby_str = R"(
class W32Window
  @open_windows = []

  def self.add_window obj
    @open_windows.push obj
  end

  def self.remove_window oid
    @open_windows.delete_if { |e| e.object_id == oid }
  end
end)";
mrb_load_string(mrb, ruby_str);

This sets us up for success later. Although iterating using Ruby code isn't as efficient as writing it in C this isn't called on a hot path.

In the W32Window_initialize function, lets add to the @open_windows upon creation using the method. We can call methods on Ruby objects from C using mrb_funcall.

mrb_value mrb_funcall(mrb_state *mrb, mrb_value val, const char *name, mrb_int argc, ...);

This is equivalent to obj.send :method_name, arg1, arg2 .. ,argN. The second parameter is the object we're sending the message to, third is a const char * of the name of the method, fourth is the number of method arguments we're sending, and then the va_args of the mrb_values as method parameters.

mrb_funcall(state, mrb_obj_value(mrb_obj_ptr(self)->c),
            "add_window", 1, self);

(There is a slightly more type-safe way using mrb_funcall_argv which takes an mrb_symbol for the method name instead of a literal C string)

The strange bit is mrb_obj_value(mrb_obj_ptr(self)->c). What we're doing here is the Ruby equivalent of self.class.add_window self. The construction mrb_obj_value(mrb_obj_ptr()) is like self.class. mrb_obj_ptr is a macro which takes any mrb_value type and casts it's void * value.p to an RObject *. (Recall that every mrb_value is a wrapper with a void *, and the tt member of the mrb_value informs how to interpret the void * With an RObject * we can get the objects RClass * by following the ->c pointer. However, mrb_funcall doesn't want an RClass * or RObject *, it wants an mrb_value. So we have to convert the RClass * back to an mrb_value using the mrb_obj_value function.

Next we need a pointer to a bundle of data that we'll store with the window that we get back when Windows sends the window messages. We need two things, the mrb_state and the object_id of the W32Window instance tied to that window. Lets add that just underneath all the #includes

struct W32Window_cb_data {
  mrb_state* state;
  uint64_t object_id;
};

And then allocate a bit of memory on the heap to contain the data. I 100% would rather not allocate memory for such a situation. I did commit to working in Rome by working with a dynamic language with lots of small heap memory allocations, and will therefore work as the Romans do.

W32Window_cb_data *cb_data = reinterpret_cast<W32Window_cb_data *>
(malloc(sizeof(W32Window_cb_data)));
cb_data->state = state;
cb_data->object_id = mrb_obj_id(self);

Now finally pass that along to CreateWindowEx as the last parameter.

HWND main_window = CreateWindowEx(0, wclass.lpszClassName, RSTRING_PTR(window_name),
                                  main_window_style, CW_USEDEFAULT, CW_USEDEFAULT,
                                  mrb_fixnum(kw_values[0]),
                                  mrb_fixnum(kw_values[1]),
                                  0, 0,
                                  GetModuleHandle(nullptr),
                                  cb_data);

Attaching the Ruby Object ID to the window

Great, that's the first part done. We allocated the information we need later and passed it along to the next step. The next step is extracting the pointer and saving it to the window's user data when Windows sends our window the WM_CREATE message. At the top of the main_window_callback we'll do that using SetWindowLongPtr

if(message == WM_CREATE) {
  CREATESTRUCT* create = reinterpret_cast<CREATESTRUCT *>(lparam);
  LONG_PTR objectid_ptr = reinterpret_cast<LONG_PTR>(create->lpCreateParams);
  SetWindowLongPtr(window, GWLP_USERDATA, objectid_ptr);
}

When windows sends the WM_CREATE messages it populates the lparam as a CREATESTRUCT. That type encapsulates all the parameters passed into CreateWindowEx. The member of the CREATESTRUCT we care about is lpCreateParams. That's the W32Window_cb_data we passed in into CreateWindowEx. We relay it one last time, setting the GWLP_USERDATA pointer to the W32Window_cb_data window pointer. We won't return after this condition, we'll let the rest of the function continue as normal.

Deleting a Ruby object on window close

Now one last bit. We'll catch message sent to our window when Windows deletes the window. This is the WM_DESTROY message. What we'll do is extract the GWLP_USERDATA pointer from the closing window, cast it back to the W32Window_cb_data pointer, call W32Window.remove_window with the object_id. Then we'll check if there are less than 1 open windows. If so, we'll tell Windows to terminate the program just as before with PostQuitMessage. Since we used malloc to create a small struct on the heap containing W32Window_cb_data, we also have to free it to avoid a memory leak. The whole procedure in code, replacing WM_CLOSE with WM_DESTROY. There isn't anything here we haven't seen previously.

case(WM_DESTROY): {
  LONG_PTR instance_pointer = GetWindowLongPtr(window, GWLP_USERDATA);
  W32Window_cb_data *cb_data = reinterpret_cast<W32Window_cb_data*>(instance_pointer);
  RClass *klass = mrb_class_get(cb_data->state, "W32Window");
  assert(klass != nullptr);

  mrb_value oid;
  SET_INT_VALUE(oid, cb_data->object_id);
  mrb_funcall(cb_data->state, mrb_obj_value(klass), "remove_window", 1, oid);

  mrb_value open_windows = mrb_obj_iv_get(cb_data->state,
                                          (RObject *)klass,
                                          mrb_intern_cstr(cb_data->state, "@open_windows"));
  if(RARRAY_LEN(open_windows) < 1) {
    PostQuitMessage(0);
  }

  free(cb_data);
  break;
}

Compiling and running now, it works as expected! The program continues running until all the windows close.

This is a pretty large achievement. We integrated with Win32 while pushing most of the state of the program into the Ruby VM. Most of it. There's one last thing left, the main program loop is still in C. Lets put that into Ruby as well.

Moving the main program loop to Ruby

Lets create an Application module with four static methods on it start, loop, quitand process_windows_messages. The first three are basic Ruby methods. The last one is a method we'll implement in C. We'll push our current mrb_load_string call creating W32Windows into start, call process_windows_messages in the loop, and then set a flag when we call quit that the program should terminate.

First create the Application module in C so we have an easy reference to it later.

RClass *Application = mrb_define_module(mrb, "Application");

Adding to the current ruby_str string.

module Application
  @is_running = true

  def self.start
    1.upto 5 do |n|
      W32Window.new("Window #{n}", width: n * 100, height: n * 100)
    end
    loop
  end;

  def self.loop
    while @is_running
      process_windows_messages
    end
  end

  def self.quit
    @is_running = false
  end
end

Define the Application.process_windows_messages in C, moving the while(PeekMessage) loop into it. When we receive the WM_QUIT message, we call the Application.quit method instead of setting g_should_quit instead.

mrb_value
Application_process_windows_messages(mrb_state* state, mrb_value self) {
  MSG message = {};
  while(PeekMessage(&message, nullptr, 0, 0, PM_REMOVE)) {
    switch (message.message) {
    case (WM_QUIT): {
      mrb_funcall(state, self, "quit", 0);
      break;
    }
    default: {
      TranslateMessage(&message);
      DispatchMessage(&message);
      break;
    }
    }
  }
  return mrb_nil_value();
}

Then add the process_windows_messages method on Application using mrb_define_module_function

mrb_define_module_function(mrb, Application, "process_windows_messages",
                            Application_process_windows_messages, MRB_ARGS_NONE());

Now we can delete the original loop and g_should_quit as these are entirely within the Ruby VM state.

The last thing we have to do is call start on Application to kick off the program loop written entirely in Ruby

mrb_funcall(mrb, mrb_obj_value(Application), "start", 0);

Compile and run again, it still works exactly as before. However, the amazing thing here is now the VM controls absolutely everything about the program. All of the program's functionality sets up the VM state and then kicks off the VM in a program loop. We implement Ruby methods in C whenever we must interface with something external to the VM. It's memory stable, though in it's current state it dominates the CPU. It runs the program loop as fast as possible. A quick though inelegant fix is add the mruby-sleep gem in the build_config.rb and rebuild mruby. This provides a sleep method for throttling the while loop.

More polish

While this all works and we could wrap up our exploratory program there's one last little thing that bothers me. We call RegisterClassEx with the same information and window class name. While this doesn't crash the program it is an error. Lets fix this little error so that we create one window class on program start and use that instead.

Lets create a W32Window.register_window_class and call it within Application.start. We'll use the class name of W32Window as the Windows window class name. Move all of the code regarding registering a Windows window class into it.

mrb_value
W32Window_register_window_class(mrb_state *state, mrb_value self) {
  mrb_value klass_name = mrb_class_path(state, mrb_class_ptr(self));
  WNDCLASSEX wclass = {};
  wclass.cbSize = sizeof(WNDCLASSEX);
  wclass.lpszClassName = RSTRING_PTR(klass_name);
  wclass.lpfnWndProc = main_window_callback;
  wclass.hInstance = GetModuleHandle(nullptr);
  wclass.style = CS_OWNDC | CS_VREDRAW | CS_HREDRAW;
  RegisterClassEx(&wclass);

  return mrb_nil_value();
}

Only thing new here is mrb_class_path which returns an mrb_value containing the string name of the class. It's the same as calling W32Window.name within Ruby. Register this as a class method

mrb_define_class_method(mrb, W32Window, "register_window_class",
                        W32Window_register_window_class, MRB_ARGS_NONE());

Then within W32Window_initialize use the Ruby class name for registering the Windows window class

mrb_value klass_name = mrb_class_path(state, mrb_class(state, self));

CreateWindowEx(0, RSTRING_PTR(klass_name), RSTRING_PTR(window_name),

Lastly call W32Window.register_window_class at the top of the Application.start method. Compile and run and it works exactly as before, the only difference here is that it's more 'correct' as we're not repeatedly creating the same window class every time we create a new window.

The end of a mini series

That's it for this exploratory code. I've proved to myself that it is possible encapsulating state entirely within the Ruby VM while working with a complicated external system in C. The size of the program by line count doubled. There is an increased mental overhead of passing data back and forth out to Windows and receiving it back in the message callback.

There are other approaches to this. We could keep some state in C like we did previously or keep tables of mapping from C to the Ruby VM in C and translate them ourselves when we need. Those approaches is how the current Ruby integrated environment I'm working on currently functions. I run into issues with this approach, where the right Ruby incantation can bring the whole program down in a hard to debug fashion. I've found that keeping as much within the Ruby VM as possible dramatically decreases complexity and increases the amount of debugging tools available especially as I add more functionality to the environment.

Two things I did not get a chance to touch on are mrb_data types, which allow you to set instance variables wrapping C structures, and the MRB_TT_CPTR class type encapsulating a C pointer.

I'm now taking the techniques I learned here and bringing them back to my still unnamed project. It will take a while encapsulating it's current state into Ruby fully. A lot of uninteresting error prone grunt work not worth documenting.

The entire final exploratory program

#include <windows.h>
#include <stdio.h>
#include <assert.h>
#include "mruby.h"
#include "mruby/compile.h"
#include "mruby/string.h"
#include "mruby/variable.h"
#include "mruby/array.h"
#include "mruby/class.h"

struct W32Window_cb_data {
  mrb_state* state;
  uint64_t object_id;
};

LRESULT CALLBACK
main_window_callback(HWND window, UINT message, WPARAM wparam, LPARAM lparam) {
  if(message == WM_CREATE) {
    CREATESTRUCT* create = reinterpret_cast<CREATESTRUCT *>(lparam);
    LONG_PTR objectid_ptr = reinterpret_cast<LONG_PTR>(create->lpCreateParams);
    SetWindowLongPtr(window, GWLP_USERDATA, objectid_ptr);
  }

  LRESULT result = 0;
  switch(message) {
  case(WM_DESTROY): {
    LONG_PTR instance_pointer = GetWindowLongPtr(window, GWLP_USERDATA);
    W32Window_cb_data *cb_data = reinterpret_cast<W32Window_cb_data*>(instance_pointer);
    RClass *klass = mrb_class_get(cb_data->state, "W32Window");
    assert(klass != nullptr);

    mrb_value oid;
    SET_INT_VALUE(oid, cb_data->object_id);
    mrb_funcall(cb_data->state, mrb_obj_value(klass), "remove_window", 1, oid);

    mrb_value open_windows = mrb_obj_iv_get(cb_data->state,
                                            (RObject *)klass,
                                            mrb_intern_cstr(cb_data->state, "@open_windows"));
    if(RARRAY_LEN(open_windows) < 1) {
      PostQuitMessage(0);
    }

    free(cb_data);
    break;
  }
  default: {
    result = DefWindowProc(window, message, wparam, lparam);
  }
  }
  return result;
}

mrb_value
W32Window_register_window_class(mrb_state *state, mrb_value self) {
  mrb_value klass_name = mrb_class_path(state, mrb_class_ptr(self));
  WNDCLASSEX wclass = {};
  wclass.cbSize = sizeof(WNDCLASSEX);
  wclass.lpszClassName = RSTRING_PTR(klass_name);
  wclass.lpfnWndProc = main_window_callback;
  wclass.hInstance = GetModuleHandle(nullptr);
  wclass.style = CS_OWNDC | CS_VREDRAW | CS_HREDRAW;
  RegisterClassEx(&wclass);

  return mrb_nil_value();
}

mrb_value
W32Window_initialize(mrb_state* state, mrb_value self) {
  mrb_value window_name;
  mrb_int width;
  mrb_int height;
  const char* kw_names[2] = { "width", "height" };
  mrb_value kw_values[2] = {};
  const mrb_kwargs kwargs = { 2, kw_values, kw_names, 2, nullptr };
  mrb_get_args(state, "S:", &window_name, &kwargs);
  mrb_funcall(state, mrb_obj_value(mrb_obj_ptr(self)->c),
              "add_window", 1, self);
  mrb_iv_set(state, self, mrb_intern_cstr(state, "@window_name"), window_name);

  DWORD main_window_style = (WS_OVERLAPPEDWINDOW | WS_VISIBLE);

  W32Window_cb_data *cb_data = reinterpret_cast<W32Window_cb_data *>(malloc(sizeof(W32Window_cb_data)));
  cb_data->state = state;
  cb_data->object_id = mrb_obj_id(self);

  mrb_value klass_name = mrb_class_path(state, mrb_class(state, self));

  CreateWindowEx(0, RSTRING_PTR(klass_name), RSTRING_PTR(window_name),
                 main_window_style, CW_USEDEFAULT, CW_USEDEFAULT,
                 mrb_fixnum(kw_values[0]),
                 mrb_fixnum(kw_values[1]),
                 0, 0,
                 GetModuleHandle(nullptr),
                 cb_data);

  return self;
}

mrb_value
Application_process_windows_messages(mrb_state* state, mrb_value self) {
  MSG message = {};
  while(PeekMessage(&message, nullptr, 0, 0, PM_REMOVE)) {
    switch (message.message) {
    case (WM_QUIT): {
      mrb_funcall(state, self, "quit", 0);
      break;
    }
    default: {
      TranslateMessage(&message);
      DispatchMessage(&message);
      break;
    }
    }
  }

  return mrb_nil_value();
}

int main() {
  mrb_state *mrb = mrb_open();
  RClass *W32Window = mrb_define_class(mrb, "W32Window", mrb->object_class);
  RClass *Application = mrb_define_module(mrb, "Application");

  const char* ruby_str = R"(
class W32Window
  @open_windows = []

  def self.add_window obj
    @open_windows.push obj
  end

  def self.remove_window oid
    @open_windows.delete_if { |e| e.object_id == oid }
  end
end

module Application
  @is_running = true

  def self.start
    W32Window.register_window_class

    1.upto 5 do |n|
      W32Window.new("Window #{n}", width: n * 100, height: n * 100)
    end

    loop
  end;

  def self.loop
    while @is_running
      process_windows_messages
    end
  end

  def self.quit
    @is_running = false
  end
end
)";

  mrb_load_string(mrb, ruby_str);

  mrb_define_class_method(mrb, W32Window, "register_window_class",
                          W32Window_register_window_class, MRB_ARGS_NONE());

  mrb_define_method(mrb, W32Window, "initialize",
                    W32Window_initialize, MRB_ARGS_REQ(2));

  mrb_define_module_function(mrb, Application, "process_windows_messages",
                             Application_process_windows_messages, MRB_ARGS_NONE());

  mrb_funcall(mrb, mrb_obj_value(Application), "start", 0);

  mrb_close(mrb);

  return 0;
}