UPDATE - the updated version of this project is here - https://github.com/unfor19/python-project/wiki
By the end of this blog post, you will be able to:
- Create a well-structured Python project
- Use relative and absolute imports in a Python project
- Invoke a specific module from the command-line (terminal)
Link to GitHub repository - unfor19/python-project
python-project
Python project structure, relative imports, absolute imports, packages, and modules. Let's make it simpler.
Getting Started
Executing modules from the project's root directory (top-level package)
meirgabay@~/python-project (master)$ python main.py
My Path: python-project/main.py
Insert your name: willy
Hello willy
meirgabay@~/python-project (master)$ python -m appy
My Path: python-project/appy/__main__.py
Insert your name: willy
Hello willy
# Contains relative imports - `..utils.message`
meirgabay@~/python-project (master)$ python appy/core/app.py
Traceback (most recent call last):
File "appy/core/app.py", line 1, in <module>
from ..utils import message
ImportError: attempted relative import with no known parent package
meirgabay@~/python-project (master)$ python -m appy.core.app
My Path: python-project/appy/core/app.py
Insert your name: willy
Hello willy
# Doesn't contain relative imports, so no exceptions were raised
meirgabay@~/python-project (master)$ python appy/utils/message.py
My Path: python-project/appy/utils/message.py
meirgabay@~/python-project (master)$ python -m appy.utils.message
My Path: python-project/appy/utils/message.py
Questions and Answers (Q&A)
Project, Packages, Modules and Scripts, what are they?
- Project - a directory, also known as the top-level package, which contains packages and modules
- Package (in a project) - a directory which contains modules and/or packages (sub-directories)
- Script - a Python script (
.py
) which can be exected from the terminal - Module - a Python script (
.py
) which can be imported withimport
andfrom
What about Packages which are not part of a project?
- Package (built-in) - a package which is shipped with Python and can be imported with
import
andfrom
- Package (pip) - a package which is installed with pip and can be imported with
import
andfrom
. Think about it, pip stands for Package Installer for Python
How do I import Packages and Modules that I've created?
Python project's packages and modules can be imported with relative paths from any module which is part of the same project. An example is available in appy/core/app.py
If you intend to import a package or a module which is not part of the same project, you'll have to use absolute paths. This can be done with importlib, see this StackOverflow answer.
Do I need the __init__.py file?
- Short answer - no
- In previous versions of Python, you had to create the
__init__.py
file in each directory that you want to import as a package, they were called regular packages. From version 3.3+ it is not required anymore - Implicit Namespace Packages, packages without an__init__.py
file are called namespace packages.
Why do relative imports raise a problem in pylint?
The error - Attempted relative import beyond top-level packagepylint(relative-beyond-top-level)
- Short answer - I don't know
- All I can say is that it doesn't happen with flake8
Is it possible to invoke a function from the terminal?
Short answer - it depends
Trying to invoke a function from the terminal, such as
appy.core.app.main()
, will raise the ModuleNotFound exception. A package must be imported before invoking one of its functions.
meirgabay@~/python-project (master)$ python -m appy.core.app.main
/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/bin/python: Error while finding module specification for 'appy.core.app.main' (ModuleNotFoundError: __path__ attribute not found on 'appy.core.app' while trying to find 'appy.core.app.main')
- Since you can't invoke
main()
directly from the terminal, calling it from theif __main__
block enables executing it from the terminal. It's possible to pass arguments, but it's a bit ugly, read the docs to learn how. The following example attempts to execute the moduleappy.core.app
, which in turn call itsif __main__
block
meirgabay@~/python-project (master)$ python -m appy.core.app
My Path: python-project/appy/core/app.py
Insert your name: willy
Hello willy
- If the
PWD
is a subdirectory of the project, such aspython-project/appy
, an attempt to execute a module which contains relative imports, will raise the exception below. Remember, yourPWD
should always be the project's root directory, in this case it'spython-project
.
# PWD is `appy`
meirgabay@~/python-project/appy (master)$ python -m core.app
Traceback (most recent call last):
File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/Users/meirgabay/python-project/appy/core/app.py", line 1, in <module>
from ..utils import message
ValueError: attempted relative import beyond top-level package
- It doesn't happen when invoking
message
, sincemessage
doesn't use relative imports
meirgabay@~/python-project/appy (master)$ python utils/message.py
My Path: python-project/appy/utils/message.py
meirgabay@~/python-project/appy (master)$ python -m utils.message
My Path: python-project/appy/utils/message.py
- Invoking a function from the terminal is also possible by using the
-c
flag. Surprise, it's possible to pass arguments in a more intuitive way, for exampleapp.main(my_arg1, my_arg2)
meirgabay@~/python-project (master)$ python -c "import appy.core.app as app; app.main()"
Insert your name: willy
Hello willy
What are the available command-line flags in Python?
- Read the docs - using cmdline
- In this tutorial, we used both
-c
and-m
flags
Why is it possible to execute python -m appy
?
The appy/__main__.py file acts like the if __main__
code snippet, but on packages. This enables the appy
package to be executed with python -m
or with runpy
meirgabay@~/python-project (master)$ python -m appy
My Path: python-project/appy/__main__.py
Insert your name: willy
Hello willy
What's runpy
and why do you use it in main.py
?
The runpy
package provides the ability to run modules from a module (Python script).
import runpy
def main():
# import a package, and pass all current global variables to it
appy_package = runpy.run_module(mod_name="appy", init_globals=globals())
# import the function script_path() from the submodule message, and execute it
appy_package['message'].script_path(__file__)
# execute the function main(), which is located in appy/__main__.py
appy_package['main']()
if __name__ == "__main__":
main()
What's globals()
?
The official definition from the docs
Return a dictionary representing the current global symbol table. This is always the dictionary of the current module (inside a function or method, this is the module where it is defined, not the module from which it is called).
Example - Expand/Collapse
meirgabay@~/python-project (master)$ python
Python 3.8.2 (default, Jun 30 2020, 19:04:41)
[Clang 11.0.3 (clang-1103.0.32.59)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> globals()
{'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': }
Why do you have a weird path with pyenv
when you run Python?
In some of the examples you might have seen that my Python binary is located in
/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/bin/python
This is because I'm using pyenv, the official definition from the docs
pyenv lets you easily switch between multiple versions of Python. It's simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.
pyenv great for checking backwards compatibility
Switching to a different version
- Install relevant version -
pyenv install 3.7.7
- Run
export PYENV_VERSION=3.7.7
- For a day to day use
- Install relevant version -
pyenv install 3.8.2
- Add
export PYENV_VERSION=3.8.2
to your terminal'src
or_profile
($HOME/.bashrc
,$HOME/.bash_profile
,$HOME/.zshrc
)
Examples - Expand/Collapse
meirgabay@~/python-project (master)$ export PYENV_VERSION=
meirgabay@~/python-project (master)$ pyenv versions
* system (set by /Users/meirgabay/.pyenv/version) # default OS Python
3.7.7
3.8.2
meirgabay@~/python-project (master)$ python --version
Python 2.7.16
# switching to a different version
meirgabay@~/python-project (master)$ export PYENV_VERSION=3.7.7
meirgabay@~/python-project (master)$ python --version
Python 3.7.7
# day to day use
meirgabay@~/python-project (master)$ source ~/.bash_profile
meirgabay@~/python-project (master)$ python --version
Python 3.8.2
Is there a good framework for creating a Python CLI?
- Short answer - yes! more than a few!
- The Click Framework is amazing! I used it in githubsecrets and frigga
NOTE: you'll discover that I have
__init__.py
files in my projects, that's a nasty old habit, I promise I will drop them, I promise - Read more about the available Python CLI frameworks in this great blog post - Building Beautiful Command Line Interfaces with Python
Where can I find a well-structured Python project?
- sampleproject - the official Python sample project
- Django - a complex Python project
- requests - a non-complex Python project
Final words
I hope that this blog post helped you in understanding how to create a well-structured Python project, and if it did, then Heart, Clap, Star and share!
Top comments (0)