How to create a PIP package hosted on private Github repo

I have few projects with few classes I've developed to integrate multiples services and servers. Those classes are private to my business and now I need an easy way to allow all projects use it and download it and all its updates.

What is the best way? Well, we all use pip everytime to download thousands of packages to our projects and is awesome.

So what can we do with our own private repos? Lets host its project on Github and see what happens.

This articles I'm writing and will share my experience on how to do this. Until the day of this article, I have no idea like you, on HOW it will works.

Lets start our study case.

First steps

Based on this tutorial: https://python-packaging.readthedocs.io/en/latest/minimal.html I'll create my first ever package ready to go to my private pip repo.

Creating root path and start a git

First, I'll create an path, called ptolemaios wich is my main class name and start a new git on it.

$ mkdir ptolemaios
$ cd ptolemaios
$ git init
> Initialized empty Git repository in '...ptolemaios/.git/'

Folder structure

On the tutorial it says I need this folder structure:

ptolemaios/
    ptolemaios/
        __init__.py
    setup.py

so, lets create again one more folder called ptolemaios inside my ptolemaios folder... sounds a little confusing.

$ mkdir ptolemaios

Creating basic files

and lets create the files setup.py and __init__.py

$ touch setup.py
$ touch ptolemaios/__init__.py

ok, now I have the exactly structure on the tutorial. Let's commit it and go to next steps.

$ git add .
$ git commit -am "created basic package structure files and folders"
> 2 files changed, 0 insertions(+), 0 deletions(-)
> create mode 100644 ptolemaios/__init__.py
> create mode 100644 setup.py

MVP Package

As I want to learn how to do it, and check if is working, I'll start by creating a function that just prints something on my screen.

lets open our __init__.py file and edit it

def my_cool_test_method():
    print('It works!')

and also lets open our setup.py and configure the basics about the package

from setuptools import setup

setup(
    name='ptolemaios',
    version='0.0.1',
    description='My private package from private github repo',
    url='git@github.com:rfschubert/ptolemaios-sdk-package.git',
    author='Raphael Schubert',
    author_email='raphael.schubert@digitalbankscorp.com',
    license='unlicense',
    packages=['ptolemaios'],
    zip_safe=False
)

you can customize those infos with all infos you want.

lets commit it

$ git commit -am "updated setup.py and created my_cool_test_method"
> 2 files changed, 19 insertions(+)

I've created my private repo on github already, so now, I'll add it to project and push all commits to my repo

$ git remote add origin git@github.com:rfschubert/ptolemaios-sdk-package.git
$ git push -u origin master
> Counting objects: 9, done.
> Delta compression using up to 8 threads.
> Compressing objects: 100% (6/6), done.
> Writing objects: 100% (9/9), 1.00 KiB | 514.00 KiB/s, done.
> Total 9 (delta 0), reused 0 (delta 0)
> To github.com:rfschubert/ptolemaios-sdk-package.git
>  * [new branch]      master -> master
> Branch 'master' set up to track remote branch 'master' from 'origin'.

now, I want to add it on my running project to see if will work, I'll open any project I have on my machine and try installl it from GitHub using pip.

Installing with pip

lets open our terminal inside the project we want to install it and run this command

$ pip install git+ssh://git@github.com/rfschubert/ptolemaios-sdk-package.git
> Collecting git+ssh://git@github.com/rfschubert/ptolemaios-sdk-package.git
>   Cloning ssh://git@github.com/rfschubert/ptolemaios-sdk-package.git to > /private/var/folders/7z/....
> Installing collected packages: ptolemaios
>   Running setup.py install for ptolemaios ... done
> Successfully installed ptolemaios-0.0.1

Here we have few important things. Where I finded this URL?

Well, it is the basic clone url with few modifications.

My ssh clone url is: git@github.com:rfschubert/ptolemaios-sdk-package.git, look at : on ssh url, change it to / and add git+ssh:// on your ssh url

Now you should have git+ssh://git@github.com/rfschubert/ptolemaios-sdk-package.git, that's it! just do pip install git+ssh://git@github.com/rfschubert/ptolemaios-sdk-package.git and be happy!

Lets test it on my project.

I'll go on Django console to test it easy.

import ptolemaios

ptolemaios.my_cool_test_method()
# It works!

AWESOME! I can have now private packages on my project.

Step back

Well, on my computer, all works like a charm, but that's why I have the ssh keys to fetch data from private repository. How my Heroku app will download it ?

first I'll run the command to update my requirements.txt file

$ pip freeze > 'requirements.txt'

now, if I look inside of it, it only has my package name, and not the GIT url

...
ptolemaios==0.0.1
...

you will need change this line to work like expected to:

...
git+ssh://git@github.com/rfschubert/ptolemaios-sdk-package.git
...

so, when your server deploy, it will fetch from repository the current version.

I've founded an answer that can be usefull to some of you, but not to me, I'll need do some more research and update this file later.

You can create a Machine User on GitHub add it to your private repos, generate an ssh key on your server and add it to Machine User, every time you deploy, your server will have the ssh key and will be allowed to download it.

How to deploy to Heroku?

I dont know yet, but I'll find out and I'll be back... let's do some more research...