We use Python packages to structure and organize our code. When talking about Python packages, people generally mean one of the following:
- Packages that are installable with tools like pip and pipenv, most often distributed through the Python Package Index.
- Packages in your own code base, used to structure and organize your code.
This article is about creating your own packages and modules. We’ll look at what packages are, how they are structured, and how to create a Python package. You’ll also discover how packages and modules work together to organize and structure your codebase.
If you are not familiar with modules, read my article on Python modules first and then come back here. These two subjects are strongly related to each other.
Table of contents
What are Python packages?
A Python package is a directory that contains Python modules. A Python package can contain sub-packages, which are also directories containing modules. Python packages always contain a special file, named
__init__.py. You’ll learn exactly what this mysterious file is for, and how you can use it to make your package easier to import from.
Structure of a Python package
So a Python package is a folder that contains Python modules and an
__init__.py file. The structure of a simple Python package with two modules is as follows:
. └── package_name ├── __init__.py ├── module1.py └── module2.py
As mentioned, packages can contain sub-packages. We can use sub-packages to further organize our code. I’ll show you how to do that in one of the sections below. Let’s first take a look at the structure of a package with sub-packages:
. └── package_name ├── __init__.py ├── subpackage1 ├── __init__.py ├── module1.py └── subpackage2 ├── __init__.py ├── module2.py
As you can see, packages are hierarchical, just like directories.
What is __init__.py in a Python package?
__init__.py file is a special file that is always executed when the package is imported. When importing the package from above with
import package_name, the
__init__.py file is executed.
When importing the nested package from above, with
import package_name.subpackage1, the
__init__.py file of both
subpackage1 are executed. The order is as follows:
- first the
- then the
I’ve added simple print statements to all the
__init__.py files to demonstrate. We can create a
main.py file in the
package_name folder with the following contents:
If we run this program, the output will be:
$ python3 main.py Hello from package_name Hello from subpackage1
The print statements in both
__init__.py files are executed due to the import of our sub-package.
Organize your code in Python packages
We now have the following tools in our belt to properly organize our Python code:
It is advisable that you use sub-packages to group related modules together. The use of sub-packages also helps you to keep package and module names short and concise. They are often a good alternative when you find yourself using underscores in package names.
An example Python package
Let’s create a package named
http in the current directory. Our aim is that this imaginary package, one day, contains all the tools one might need to work with the HTTP protocol. We start simple though, with just HTTP GET and POST requests and a simple HTTP server.
Our package contains two sub-packages:
server. An initial package layout might look like this:
http/ __init__.py client/ __init__.py get.py post.py server/ __init__.py run.py
Some things to notice:
- The names are short and descriptive. Because
serverare subpackages of
http, it’s obvious to everyone that these are an HTTP client and server. We don’t need to call them
- We’ve grouped similar functionality in sub-packages:
- client code in a client package,
- and server code in a server package.
- We’ve grouped closely related code into modules. E.g., all we need to do HTTP get requests goes in the
Extending out Python package
Now imagine that we want to jump on the async programming model. While our original server used to be synchronous, we now also offer an async variant.
There are three ways to add this to our codebase:
- create a new package named
- create new subpackages named
- create new modules with names like
Which option would you choose?
Option 1: Create a new package
Option 1 is the simplest and most straightforward. We can create a new package named
http_async and copy the contents of the
http package into it. Next, we adapt all the code to be async. A user of our original package might get away with changing a single line of code: change
import http to
import http_async. Since asynchronous programming is a completely different paradigm, this simple swap would probably not be enough though.
Option 2: Create new subpackages
Option two means that existing users of our library, even those that don’t want to use async, need to change their code. It could be a good option when starting fresh, but it’s not such a good option for an existing library.
Option 3: Create new modules
And then there’s option three: creating new modules with async appended to the module names. Although existing users would not need to change their code, I still would not recommend this. It’s more logical to bundle the async modules in their own package, if only because you wouldn’t have to repeat the
async prefix/postfix all the time when doing imports. It fits in the general objective of keeping package names and module names short and concise.
Importing modules in __init__.py
Our previous HTTP package can be improved even more, by importing the important functions inside of
__init__.py file is often a good place to import other modules. Let’s take our
http package from above, for example. Our get.py module provides a function
get() that returns a response. We can import this function into our
from http.client.get import get
This is a common pattern in Python. It’s a good idea to import all the modules you need in the
__init__.py file. If we use the import as described above, we can import the get function like this:
from http.client import get
Depending on the situation, and also depending on taste, you could also do this:
from http import client client.get(...)
Without the import in
__init__.py, users of our package would need to do what we did ourselves to use the get function:
from http.client.get import get
Absolute or relative imports
We’ve imported the
get() function from the
http.client.get module using absolute imports. You can also use relative import. The advantage of relative imports is that you can import the module you want to use without having to know the full path. The package names might even change, without breaking the code. So relative imports make your code more robust to change.
A relative import in the
__init__.py file of
http.client looks like this:
from .get import get
This is also one of the cases where wildcards are useful. We can import all the elements in the
http.client.get module using a wildcard:
from .get import *
This is not so bad, because we know exactly what we’ve put in
get.py. In addition, we can change function names in
get.py without needing to change import statements. This again makes the code more flexible.
Modules vs Packages
Let’s conclude by summarizing all of the above information about Python packages and the things we learned in the previous article on Python modules. A question that often pops up, is this: what’s the difference between a module and a package?
- A modules is always a single file, while a package can contain many modules.
- A module is used to bundle Python functions, classes, constants, and anything else that you want to make reuseable. A package in turn bundles modules.
- Modules can live on their own and don’t need to be part of a package, while a package needs modules to be useful.
- Packages and modules, together, form a powerful way of organizing our code.