Après Midi

This is part 4 in a series on adding .midi persistence and depersistence to an open source project written in C++ using the JUCE framework. You can check out part 3 here.

In the final installment of my efforts to bring midi import and export to BespokeSynth I'm working my way through the details of handling midi in Juce, with the added layer of having some Juce functionalities handled by Bespoke.

The first part of my endeavours has centered on getting midi import working, and to that end I've started with a few changes.

void NoteCanvas::LoadMidi()
{
    using namespace juce;
    FileChooser chooser("Load midi", File(ofToDataPath("")), "*.mid", true, false, TheSynth->GetFileChooserParent());
    if (chooser.browseForFileToOpen())
    {
        juce::File file = chooser.getResult();

        //get fileinputstream from midi file
        juce::FileInputStream inputStream(file);

        juce::MidiFile midifile;

        midifile.readFrom(inputStream);

This is the LoadMidi function so far, which for now just gets a midi file with FileChooser, makes a FileInputStream with it and then reads from it into a MidiFile.

From what I understand, the midi notes that populate the grid are contained in a vector of NoteCanvasElement pointers called mCurrentNotes. The function that adds these elements is as follows:

NoteCanvasElement* NoteCanvas::AddNote(double measurePos, int pitch, int velocity, double length, int voiceIdx/*=-1*/, ModulationParameters modulation/* = ModulationParameters()*/)

This tells me that to make each note I'll need to extract the following information:

• position
• pitch
• velocity
• length
• voice index
• ModulationParameters

Most of these make sense to me, I'll need to do some looking into voice index and ModulationParameters and figure out what the relevance of those is but regardless now I know what I need to extract from the MidiFile.

So things are getting pretty deep now with midi standards and parsing out individual parameters from bits in midi messages so I thought I'd stop and take stock of what I've learned so far before I dive deeper in. I'm also noticing a dearth of information on midi import in the Juce community so hopefully having it laid out here will help someone in the future. Not to bash their forum at all though, it's an awesome resource and I've had about 20 tabs of threads on it open for the past few weeks. Aside from the Bespoke Discord, The Audio Programmer's discord has been a great resource as well and I've seen a lot of overlap between the members of those 3 communities. But I digress, on to the scoop on midi, starting with a brief overview.

Midi allows musicians to sequence notes and have them played back by some sort of electronic instrument, be it a software synthesizer, analogue synthesizer, a sampler, or any number of different devices created over the years to take advantage of this standard. It can also be used to accept input from one device to another, having a midi keyboard to play notes on a software synthesizer is a common application. These notes are relayed in "messages", which are sequences of "events".

The most common events used, such as the ones you'd receive from a midi keyboard into a digital audio workstation, are Note On and Note Off messages.

These are composed of 3 bytes:

1. a status byte - 1001 CCCC
2. a data byte - 0PPP PPPP
3. a data byte - 0VVV VVVV

Where:

"CCCC" is the MIDI channel (from 0 to 15)
"PPP PPPP" is the pitch value (from 0 to 127)
"VVV VVVV" is the velocity value (from 0 to 127)

The midi channel determines which of up to 16 discrete instruments or channels will be played from the input. The pitch value determines the frequency of the note to be played from 0 to 127. The velocity (generally) determines how loud a note is, with a higher value being a louder note.

For every Note On message, a corresponding Note Off message must be received as well or the note is sustained forever. Here's a page with more info that was very helpful in figuring all this out!

So far I've been able to get the note number (key) and velocity from the messages but still have to work out the time and length of the events, which may pose a challenge with a grid that can change in size. Regardless, once I get the notes created and displayed I'll be able to debug the specifics of getting them aligned right in the note canvas and determine how (or if it's desirable to) set the global tempo based on midi.

This morning I was able to get all the information I needed from the midi messages, using Juce's MidiMessageSequence::MidiEventHolder I was able to use accessor methods to get the relevant node, velocity, and start time, and length. I was wondering how to find the relevant Note Off object for a given Note On object, but MidiEventHolder keeps track of that for you. By comparing timestamps from the Note On and Note Off objects I was able to get the note length. I passed all of these to the NoteCanvas::AddNote function and it looks like everything was added properly - but when I try loading a midi file nothing happens. I'm going to look into AddNote some more and try to discern how what I'm doing differs from its normal behaviour and hopefully I can get to the bottom of this.

I was calling the clear function after loading the midi instead of before, that was the problem. Having moved that it's starting to come together! The timing is off, the tempo seems to be maybe half what it should be, but the notes are showing up in the right places relative to one another and the UI is updating properly, playing the notes properly etc.

While waiting for input from the other devs on the load midi function, I got started on the save midi function and have... something now. I initially thought this would be the easy part of this process, but it seems to come with all sorts of other problems. Essentially because Bespoke has its own way of dealing with midi rather than using Juce's built in MidiBuffer class, I have to set the properties of those notes on import, and then recreate midi data from them afterward on export. Getting and setting note number (key), velocity and starting timestamps was easy enough, but I need to now figure out 1. setting metadata on the first midi track with info about tempo and 2. creating matching Note On and Note Off pairs from a single note event object from Bespoke. I think I'm almost there with the latter, and the former I need to dive deeper into midi tempos to discern. From what I've read midi tempos are measured differently, the tempo note meta event in Juce only accepts tempo in a microseconds per quarter note int format. Thus I need to determine what that calculation is, though it seems like there may be something to do with PPQ as well which I don't know where to get.

First, let me give a quick update on where I'm at with midi import and then I'll move to my ongoing struggles with midi export. So far midi import is in good shape, I'm able to import a midi type 0 file with 1 track and have the note placement, velocity and lengths all line up properly. A type 1 midi file will only import the first track pending a UI solution from Bespoke's maintainer to have a modal window prompt the user for track selection, so that's taken care of for now. The maintainer's also decided against setting the transport tempo/time signature based on the imported midi file so I left that alone. I added logic to update the measures slider maximum when the midi file exceeds it, and that about covers it for midi import.

As for export, I've given up trying to set the file's tempo for now in favor of just getting something serviceable - but still haven't gotten there yet. I'm now generating corresponding Note Off events for my Note On events but I'm not 100% sure they're being loaded in properly, possibly due to not being matched up. After several hours going back and forth tweaking loadMidi and saveMidi and looking for problems in the data I downloaded this open source tool MIDIopsy to take a look at the binary up close and personal. A simple 1 note midi file I exported resulted in this binary representation:

This looks to me like the timestamps aren't being set properly as the time fields are all blank. I'm going to look at the format for how that's set and hopefully get some working midi files saved soon!

After a lot of trial and a lot of error, I've got something worth submitting a PR for finally and just submitted one. Yesterday I accidentally deleted my project trying to create a new branch and it snowballed into me redownloading and rebuilding the app several times trying to remember how I did it the first time. Luckily I had my first Hacktoberfest blog to fall back on to remember the cmake commands to build for visual studio, so I got it up and running again. After that my problems with midi continued, I gave up trying to write the tempo to the track but still ended up having PPQN and time units throw me for a loop when I tried to sort out my timestamp issues. Someone from the Bespoke discord named baconpaul was kind enough to help walk me through the midi timestamp issue and this how I arrived at the proper timestamps - first of all it's worth noting a lot of this is Bespoke-specific so if you were to recreate these steps you'd have to adapt them for however your sequencer treats notes. For Bespoke the steps were as follows:

1. Add the note column (position) and offset
2. Multiply by PPQN
3. Multiply by the difference between the interval and a quarter note

And this is what it looks like in the save function:

float noteStart = (element->mCol + element->mOffset) * tppq *
                    TheTransport->GetMeasureFraction(mInterval) / TheTransport->GetMeasureFraction(kInterval_4n);

For the corresponding Note Off event, I just added the length before multiplying as the the two events are treated as one note object with a length value in Bespoke's sequencer.

I debugged and tested these changes as I went along by exporting a midi file from Ableton so I knew it was set up properly, importing the file into MIDIopsy to see what the correct values would be, then importing and exporting the file in note canvas to see the changes. Likewise after fixing the note start and stop times, I realized the note numbers/keys were off when exporting. I reversed the operations being done in AddNote to get the key, and suddenly the keys were fixed as well.

Here's import:

And here's export - (yes I suck at keys, this is why I need midi files)

With that pull request my job is done for now, having been in contact with the maintainer and the other developers to shape what I was working on throughout I doubt there will be many changes requested in the review so I'll wrap this up here. What an experience this has been. I've always wanted to work on music software but there's always been technological barriers in my way, be they lack of information for Windows versions of software, missing SDKs, libraries or other dev tools set up wrong, or just a dearth of perspective that meant I had no idea where to start on something. There were some of those for this project, but after the work I've done in this class they seemed like a joke. Setting up Juce and Bespoke was painless, and though there's always room for improvement I think the Juce documentation was good at relaying what all the tools did. I definitely would have had a tough time getting this done without the Bespoke community on discord and the Juce forum, so I'm very grateful such supportive communities exist to help programmers trying to get into music software. Looking forward to this getting merged and adding to the knowledge base out there on using midi with Juce.