We use Python packages to structure and organize our code. When talking about Python packages, people generally mean one of the following:
- Packages that are installable with tools like pip and pipenv, most often distributed through the Python Package Index.
- Packages in your own code base, used to structure and organize your code.
If you’re looking for how to install packages, you should read the article on installing packages with pip install instead. In addition, I can also recommend the article on virtual environments.
This article is about creating your own packages and modules. We’ll look at what packages are, how they are structured, and how to create a Python package. You’ll also discover how packages and modules work together to organize and structure your codebase.
If you are not familiar with modules, read my article on Python modules first and then come back here. These two subjects are strongly related to each other.
Table of contents
What are Python packages?
A Python package is a directory that contains Python modules. A Python package can contain sub-packages, which are also directories containing modules. Python packages always contain a special file, named __init__.py
. You’ll learn exactly what this mysterious file is for, and how you can use it to make your package easier to import from.
Structure of a Python package
So a Python package is a folder that contains Python modules and an __init__.py
file. The structure of a simple Python package with two modules is as follows:
. └── package_name ├── __init__.py ├── module1.py └── module2.py
As mentioned, packages can contain sub-packages. We can use sub-packages to further organize our code. I’ll show you how to do that in one of the sections below. Let’s first take a look at the structure of a package with sub-packages:
. └── package_name ├── __init__.py ├── subpackage1 ├── __init__.py ├── module1.py └── subpackage2 ├── __init__.py ├── module2.py
As you can see, packages are hierarchical, just like directories.
What is __init__.py in a Python package?
The __init__.py
file is a special file that is always executed when the package is imported. When importing the package from above with import package_name
, the __init__.py
file is executed.
When importing the nested package from above, with import package_name.subpackage1
, the __init__.py
file of both package_name
and subpackage1
are executed. The order is as follows:
- first the
__init__.py
file ofpackage_name
is executed, - then the
__init__.py
file ofsubpackage1
.
I’ve added simple print statements to all the __init__.py
files to demonstrate. We can create a main.py
file in the package_name
folder with the following contents:
import package_name.subpackage1
If we run this program, the output will be:
$ python3 main.py Hello from package_name Hello from subpackage1
The print statements in both __init__.py
files are executed due to the import of our sub-package.
Organize your code in Python packages
We now have the following tools in our belt to properly organize our Python code:
- Packages
- Sub-packages
- Modules
It is advisable that you use sub-packages to group related modules together. The use of sub-packages also helps you to keep package and module names short and concise. They are often a good alternative when you find yourself using underscores in package names.
An example Python package
Let’s create a package named http
in the current directory. Our aim is that this imaginary package, one day, contains all the tools one might need to work with the HTTP protocol. We start simple though, with just HTTP GET and POST requests and a simple HTTP server.
Our package contains two sub-packages: client
and server
. An initial package layout might look like this:
http/ __init__.py client/ __init__.py get.py post.py server/ __init__.py run.py
Some things to notice:
- The names are short and descriptive. Because
client
andserver
are subpackages ofhttp
, it’s obvious to everyone that these are an HTTP client and server. We don’t need to call themhttp_client
andhttp_server
. - We’ve grouped similar functionality in sub-packages:
- client code in a client package,
- and server code in a server package.
- We’ve grouped closely related code into modules. E.g., all we need to do HTTP get requests goes in the
get.py
module.
Extending out Python package
Now imagine that we want to jump on the async programming model. While our original server used to be synchronous, we now also offer an async variant.
There are three ways to add this to our codebase:
- create a new package named
http_async
- create new subpackages named
sync
andasync
in thehttp
package - create new modules with names like
async_get
andasync_post
and async_run.
Which option would you choose?
Option 1: Create a new package
Option 1 is the simplest and most straightforward. We can create a new package named http_async
and copy the contents of the http
package into it. Next, we adapt all the code to be async. A user of our original package might get away with changing a single line of code: change import http
to import http_async
. Since asynchronous programming is a completely different paradigm, this simple swap would probably not be enough though.
Option 2: Create new subpackages
Option two means that existing users of our library, even those that don’t want to use async, need to change their code. It could be a good option when starting fresh, but it’s not such a good option for an existing library.
Option 3: Create new modules
And then there’s option three: creating new modules with async appended to the module names. Although existing users would not need to change their code, I still would not recommend this. It’s more logical to bundle the async modules in their own package, if only because you wouldn’t have to repeat the async
prefix/postfix all the time when doing imports. It fits in the general objective of keeping package names and module names short and concise.
Importing modules in __init__.py
Our previous HTTP package can be improved even more, by importing the important functions inside of __init__.py
. The __init__.py
file is often a good place to import other modules. Let’s take our http
package from above, for example. Our get.py module provides a function get()
that returns a response. We can import this function into our __init__.py
file:
from http.client.get import get
This is a common pattern in Python. It’s a good idea to import all the modules you need in the __init__.py
file. If we use the import as described above, we can import the get function like this:
from http.client import get
Depending on the situation, and also depending on taste, you could also do this:
from http import client client.get(...)
Without the import in __init__.py
, users of our package would need to do what we did ourselves to use the get function:
from http.client.get import get
Absolute or relative imports
We’ve imported the get()
function from the http.client.get
module using absolute imports. You can also use relative import. The advantage of relative imports is that you can import the module you want to use without having to know the full path. The package names might even change, without breaking the code. So relative imports make your code more robust to change.
A relative import in the __init__.py
file of http.client
looks like this:
from .get import get
This is also one of the cases where wildcards are useful. We can import all the elements in the http.client.get
module using a wildcard:
from .get import *
This is not so bad, because we know exactly what we’ve put in get.py
. In addition, we can change function names in get.py
without needing to change import statements. This again makes the code more flexible.
Modules vs Packages
Let’s conclude by summarizing all of the above information about Python packages and the things we learned in the previous article on Python modules. A question that often pops up, is this: what’s the difference between a module and a package?
- A modules is always a single file, while a package can contain many modules.
- A module is used to bundle Python functions, classes, constants, and anything else that you want to make reuseable. A package in turn bundles modules.
- Modules can live on their own and don’t need to be part of a package, while a package needs modules to be useful.
- Packages and modules, together, form a powerful way of organizing our code.