Who Are You

A junior data person that does everything through JupyterNotebooks JN. Maybe you just started your first job or about to start and long past the point of online courses and tutorials. So this is NOT for BEGINNERS in anyway, actually if you are a beginner you are better starting of with JN!

The spirit of JN

Before going through why you should leave JN, I want to give a big thanks to the JN development team on making my learning experience and many others a smooth and simple welcoming experience as we were starting with data science. But what I feel like thanking them on more, is what I call the spirit of JN. Reading about functional programming, a lot of the concepts and best practices were already familiar to me simply because of the JN style. The style of JN is like: you are making small blocks of code (functions)such that, the output of one block can feed as an input to others, and the whole thing runs in a sequential way. This kinda forces you into dividing your code into smaller simpler parts, testing them, moving them around becomes supper easy, tracking errors and bugs is much easier and modifications effects are limited to a smaller area of the total code. Expand this style further and you will start creating helper files and modules without realizing, it grows naturally on you, at least that was my case. This kind of style or spirit is what makes me really grateful to the JN team.

What's A JN

JNs are like a save space, very friendly and useful for experimenting and creating quick results, models and EDA using using a GUI run by Python. Going into the wild, things start to become less friendly, you move files and code all over the place, you may work on other machines, use the command line to run code, inspect the machine and so on.

What JN does is abstracting a lot of things so you can experiment without worrying about them, but this abstractions comes with a price tag, you carry a lot of baggage and limit yourself as a programmer. The wild requires you to be fast and lean, learning new tricks and utilizing the full potential of the beautiful Python language.

So Should I leave JN for good?

The short answer is yes, but not totally. The main reason I left the JN was because I could not use Vim, scrolling and copy and pasting through large notebooks became a nightmare.

I tried looking for a way to use vim with JN but with no hope. Then I found my salvation, vscode integrated JN last year, so you could open and edit and even transform the notebooks to other files natively inside vscode, and as a bounce you can add the vim plugin(or any other plugin) to do your vi magic.

This seemed promising at first, but I was still not able to use vim commands on the cells; The notebooks inside vscode where just like the ones on the browser, you could not jump cells or use vim inside of them. The only solution was to work with pure .py files. This seemed intimidating at first as I am used to the cell layout and running blocks of code individually, working with the normal python files was not like what I used to do for 2 years of online courses.

This seemed like a problem, but because of it, I discovered a new world that I would not have dared to approach if I was sticking to the notebooks.

All You Need Is #%%

This is not a mask for a swear word, rather it's a special comment in python files that treats what is beneath it as a "cell". I found this when I converted my notebooks to .py files, this comment replaced the cell structure, and with vscode, when you add it to your .py file, you get the usual cell options, run, run above debug cell as shown.

With this weird comment, you have now the structure of JN inside python files, and you can use vim and other great vscode plugins if you want. This is why I said you will not leave JN totally as it can still be with you in spirt and style.

Hacking Is The Way to Go

Adding this comment whenever I wanted to create a new cell was too much for me, could I make a short-key/key-biding that can write it down for me? Turns out I could. I had to read about keybidings in vscode and how to configure them, this drove me to dig around and get my hands dirty, something I wouldn't have done if I was sticking to JN. From there on, I started to play with different things like: the development environment, configuration files, libraries and so on.

A Mental Shift

Relying on JN will limit you, at least that was my case. Removing the JN from the equation forced me to try and tweak things more, read more documentations and learn how stuff work under the hood. This simple shift of mentality has improved and added to my skills greatly, I understand this might not be your cup of tea, but trying will not hurt as you are getting almost exactly the same experience as using JN the normal way.