A REPL for Fat-Finger Friendly Typing

My Python interpreter, Memphis, has a REPL (read-eval-print loop)!

This is old news. As long as you made zero mistakes while interacting with the wise old owl 🦉, you could interpret to your heart’s content. Assuming you never wanted to evaluate the same statement twice, or if you did, didn’t mind retyping it. Also with zero mistakes.

I was perfectly content with this REPL. Thrilled even. I had written home about this REPL. But my bosses’s bosses’ bossi demanded we improve the REPL for the bottom line for the people. They called me into their office and nodded me into the chair across from their mahogany desk. “Some users are expecting the backspace key to work.” TO HELL WITH THE USERS! “The up arrow should bring up their last command.” ARROW KEY SUPPORT IS SO NINETIES! “Can you have this done by the end of Q5?” I QUIT!

So I went back to my desk and improved the REPL.

I improved it so much that all the keys worked. The backspace key, the up arrow, the down arrow, the left arrow, and last, but not least, the backspace arrow. An accountant could have a field day with the new REPL. tick tick tick tick tick tick tick tick tick. That’s the accountant typing numbers, not a bomb slowly diffusing.

I sent the REPL down to the lab and told my main machinist to put a rush job on this order. It’s the REPL I said and from the look in their eyes I could tell they understood. 750ms later the build was complete and we had arrow key support. I took the product back to the big wigs, begged for my job back, and asked them what they thought. They ran a few commands, printed some prints, and added some adds. They made a mistake and hit the backspace key. I rolled my eyes because seriously who makes mistakes but they seemed satisfied. They realized they didn’t want to run a long command they had already typed out and this is where my life went to hell in a handbasket. They. hit. Ctrl. C. Seriously, who does that?! You know that ends the current process, right? RIGHT???

“We need Ctrl-C support by the end of next year.” These people and their demands. I would add Ctrl-C support. But it absolutely would not be within the next two years.

So I went back to my desk and added Ctrl-C support.

What made this REPL worthy of people with fat-fingered tendencies?

Would I be a tool?

I have staked my entire professional and financial future on building things “from scratch,” so I faced a quandary on day 1 of this project. I chose to use crossterm for the key detection primarily because of the cross-platform support. Honestly though, crossterm was very, very good. The API is intuitive and I was especially pleased with KeyModifiers (which we needed to handle Ctrl-C, which I thought was unnecessary, see above).

Raw mode is a pain

We needed it so that the terminal wouldn't handle special keys for us. But damn, I didn't realize it would turn our screen into a malfunctioning typewriter. Anyway, I had to normalize all strings to add a carriage return before any newline characters. Which worked fine and I'm THRILLED about it.

/// When the terminal is in raw mode, we must emit a carriage return in addition to a newline,
/// because that does not happen automatically.
fn normalize<T: Display>(err: T) -> String {
    let formatted = format!("{}", err);
    if terminal::is_raw_mode_enabled().expect("Failed to query terminal raw mode") {
        formatted.replace("\n", "\n\r")
    } else {
        formatted.to_string()
    }
}

/// Print command which will normalize newlines + carriage returns before printing.
fn print_raw<T: Display>(val: T) {
    print!("{}", normalize(val));
    io::stdout().flush().expect("Failed to flush stdout");
}

Also! If you do not disable raw mode on an unexpected panic, your terminal session is rendered unusable. I installed custom panic handler to catch this and play nice.

panic::set_hook(Box::new(|info| {
    // This line is critical!! The rest of this function is just debug info, but without this
    // line, your shell will become unusable on an unexpected panic.
    let _ = terminal::disable_raw_mode();

    if let Some(s) = info.payload().downcast_ref::<&str>() {
        eprintln!("\nPanic: {s:?}");
    } else if let Some(s) = info.payload().downcast_ref::<String>() {
        eprintln!("\nPanic: {s:?}");
    } else {
        eprintln!("\nPanic occurred!");
    }

    if let Some(location) = info.location() {
        eprintln!(
            "  in file '{}' at line {}",
            location.file(),
            location.line()
        );
    } else {
        eprintln!("  in an unknown location.");
    }

    process::exit(1);
}));

Integration testing was fun

Under my old REPL (which I preferred, see above), I could test it integration-ally by just running the binary and passing in some Python code to stdin. That stopped working when using crossterm I think because of a contract dispute. I honestly can’t explain it fully, but event::read() would timeout and fail in the integration test provided with stdin input. So I mocked it.

pub trait TerminalIO {
    fn read_event(&mut self) -> Result<Event, io::Error>;
    fn write<T: Display>(&mut self, output: T) -> io::Result<()>;
    fn writeln<T: Display>(&mut self, output: T) -> io::Result<()>;
}

/// A mock for testing that doesn't use `crossterm`.
struct MockTerminalIO {                                                        
    /// Predefined events for testing
    events: Vec<Event>,

    /// Captured output for assertions
    output: Vec<String>,
}

impl TerminalIO for MockTerminalIO {
    fn read_event(&mut self) -> Result<Event, io::Error> {
        if self.events.is_empty() {
            Err(io::Error::new(io::ErrorKind::Other, "No more events"))
        } else {
            // remove from the front (semantically similar to VecDequeue::pop_front).
            Ok(self.events.remove(0))
        }
    }

    fn write<T: Display>(&mut self, output: T) -> io::Result<()> {
        self.output.push(format!("{}", output));
        Ok(())
    }

    fn writeln<T: Display>(&mut self, output: T) -> io::Result<()> {
        self.write(output)?;
        self.write("\n")?;
        Ok(())
    }
}

Which resulted in the whole thing becoming a unit test? Honestly I don’t know. At this point, I call it an integration test if I either a) call a binary inside another binary, or 2) launch a server / open a port / listen on a socket inside a test. If you have another definition you’d like to leave in the comments, please don’t because that sounds annoying TBH.

I created two utility functions to get started.

/// Run the complete flow, from input code string to return value string. If you need any Ctrl
/// modifiers, do not use this!
fn run_and_return(input: &str) -> String {
    let mut terminal = MockTerminalIO::from_str(input);
    Repl::new().run(&mut terminal);
    terminal.return_val()
}

/// Turn an input string into a list of crossterm events so we don't have to
/// hand-compile our test.
fn string_to_events(input: &str) -> Vec<Event> {
    input
        .chars()
        .map(|c| {
            let key_code = match c {
                '\n' => KeyCode::Enter,
                _ => KeyCode::Char(c),
            };
            Event::Key(KeyEvent::new(key_code, KeyModifiers::NONE))
        })
        .collect()
}

Using these, we can now test these common scenarios with fairly little boilerplate.

#[test]
fn test_repl_name_error() {
    let return_val = run_and_return("e\n");
    assert!(return_val.contains("NameError: name 'e' is not defined"));
}

#[test]
fn test_repl_expr() {
    let third_from_last = run_and_return("12345\n");
    assert_eq!(third_from_last, "12345");
}

#[test]
fn test_repl_statement() {
    let return_val = run_and_return("a = 5.5\n");

    // empty string because a statement does not have a return value
    assert_eq!(return_val, "");
}

#[test]
fn test_repl_function() {
    let code = r#"
def foo():
    a = 10
    return 2 * a

foo()
"#;
    let return_val = run_and_return(code);
    assert_eq!(return_val, "20");
}

#[test]
fn test_repl_ctrl_c() {
    let mut events = string_to_events("123456789\n");
    let ctrl_c = Event::Key(KeyEvent::new(KeyCode::Char('c'), KeyModifiers::CONTROL));
    events.insert(4, ctrl_c);
    let mut terminal = MockTerminalIO::new(events);

    Repl::new().run(&mut terminal);
    assert_eq!(terminal.return_val(), "56789");
}

Code entrypoints get me out of bed in the morning

One of my motivations in adding a REPL at all was because I believe you make your code better when you add a second entrypoint. You are essentially becoming the second user for your library, which helps you get closer to understanding The One Perfect Abstraction we are all poking at our keyboards in search of. I mean this point earnestly.

“Zero Dependencies” 😉

The REPL is now behind a feature flag as a way to get back at management. I am keeping alive the ability to interpret Python code with the help of zero third-party crates, which means crossterm would either need to be an exception or I would introduce a feature flag. Now, if you compile without the REPL enabled and run “memphis”, it will politely tell you “wrong build, dumbass.”

Goodbye

The REPL is here. You can run it like this. If you want to buy it, that sounds like a scam. Be well & Talk soon.

If you’d like to get more posts like this directly to your inbox, you can subscribe here!

Elsewhere

In addition to mentoring software engineers, I also write about my experience as an adult-diagnosed autistic person. Less code and the same number of jokes.

• Britain in moderation - From Scratch dot org