TIL: Python virtual environments and venv

I'm learning Python. The reason is, well, Python is the lingua franca of the Data Science world, and I intend to work with it, eventually. I'm also an old school programmer, so I'm not convinced that "just use X" is the right approach. I refuse to just rely on a Jupyter notebook like Google Collab, etc., so I got to work with Python in real world (which means CLI, Python packages/dependencies, and virtual environments).

At first sight, virtual environments were super confusing! The name "virtual environment" doesn't help understanding what it is. Plus, there are so many ways to set up virtual environments that I was quite lost between all the names and projects, libraries and techniques, etc. Yeah, there was some friction to truly understand it – and I'm still on my way to learn it all.

With it said, I'm sticking with Python's built-in virtual environment tool for now, venv. So the rest of this post assumes venv is being used to work with virtual environments.

WTF are virtual environments anyway?

So far, a virtual environment seem to be a command-line trick to isolate a project from each other. The Python packaging doesn't seem to assume the concept of local package installation (eg.: Node.js' node_modules), it keeps packages into either a global, system-wide scope, or a local, user-wide scope (each system user ending up with its own set of Python package installs).

The problem is that to work on multiple Python projects using your own OS user, you would have conflicts with package versions, just like how you would with a global package scope. :(

So, virtual environments solve this issue. It tricks both Python and pip to install and load packages into a local directory (aka.: your project's directory) instead of global-wide or user-wide directories. It allows you to work on different projects with the same dependencies but different versions without conflicts.

That's it. A quite common approach of pretty much any modern programming language today.

How to use venv

venv is a built-in Python module that does exactly this: manage virtual environments. It is a subset of virtualenv, a Python module that used to be the standard way to set up virtual environments but now compete with other options, like venv.

Using it is really very simple. venv is built-in, so you don't even need to install it, it's already available for you if you're using Python v3.3 or newer.

To create a virtual environment, run:

python3 -m venv .venv

It will create a virtual environment into the .venv directory. You can name it however you like – but .venv seems to be a popular choice, so I'm sticking to it.

Then, in order to use it, you must activate it first:

source .venv/bin/activate

Here's the magic. When you activate the virtual environment, it will trick python, python3, pip, and pip3 to install and load packages from/to .venv. So, each project will have this separation of dependencies. :-)

After you've activated your virtual environment, you can install your local, project-specific dependencies with pip, possibly with the help of a requirements.txt:

pip3 install -r requirements.txt

And when you're done working with this project for the day, you can deactivate (exit from) the virtual environment with:

deactivate

TILs

venv is quite easy to use and work with. It does not isolate runtime versions, though. It seem there are some other projects to do it, but I didn't get to them just yet.

Also, there were a few things that really confused me along the way. For instance, Python literacy talks about virtual environments as something that "isolates Python projects with everything – including Python, pip, etc.", and from it I understood that it somehow could embed a specific Python runtime too. After reading a ton of things online, discussing with people in IRC and Telegram chats, I finally understood that venv does not embed specific Python runtime versions, just dependencies.

The next thing I want to try is virtualenv, which seem more feature-rich. I'll post my findings here.

Thanks for reading!