Python Packages: How To Create And Use With Examples

We use Python packages to structure and organize our code. When talking about Python packages, people generally mean one of the following:

1. Packages that are installable with tools like pip and pipenv, most often distributed through the Python Package Index.
2. Packages in your own code base, used to structure and organize your code.

If you’re looking for how to install packages, you should read the article on installing packages with pip install instead. In addition, I can also recommend the article on virtual environments.

This article is about creating your own packages and modules. We’ll look at what packages are, how they are structured, and how to create a Python package. You’ll also discover how packages and modules work together to organize and structure your codebase.

If you are not familiar with modules, read my article on Python modules first and then come back here. These two subjects are strongly related to each other.

What are Python packages?

A Python package is a directory that contains Python modules. A Python package can contain sub-packages, which are also directories containing modules. Python packages always contain a special file, named __init__.py. You’ll learn exactly what this mysterious file is for, and how you can use it to make your package easier to import from.

Structure of a Python package

So a Python package is a folder that contains Python modules and an __init__.py file. The structure of a simple Python package with two modules is as follows:

.
└── package_name
    ├── __init__.py
    ├── module1.py
    └── module2.py

As mentioned, packages can contain sub-packages. We can use sub-packages to further organize our code. I’ll show you how to do that in one of the sections below. Let’s first take a look at the structure of a package with sub-packages:

.
└── package_name
    ├── __init__.py
    ├── subpackage1
        ├── __init__.py
        ├── module1.py
    └── subpackage2
        ├── __init__.py
        ├── module2.py

As you can see, packages are hierarchical, just like directories.

What is __init__.py in a Python package?

The __init__.py file is a special file that is always executed when the package is imported. When importing the package from above with import package_name, the __init__.py file is executed.

When importing the nested package from above, with import package_name.subpackage1, the __init__.py file of both package_name and subpackage1 are executed. The order is as follows:

1. first the __init__.py file of package_name is executed,
2. then the __init__.py file of subpackage1.

I’ve added simple print statements to all the __init__.py files to demonstrate. We can create a main.py file in the package_name folder with the following contents:

import package_name.subpackage1

If we run this program, the output will be:

$ python3 main.py
Hello from package_name
Hello from subpackage1

The print statements in both __init__.py files are executed due to the import of our sub-package.

Organize your code in Python packages

We now have the following tools in our belt to properly organize our Python code:

• Packages
• Sub-packages
• Modules

It is advisable that you use sub-packages to group related modules together. The use of sub-packages also helps you to keep package and module names short and concise. They are often a good alternative when you find yourself using underscores in package names.

An example Python package

Let’s create a package named http in the current directory. Our aim is that this imaginary package, one day, contains all the tools one might need to work with the HTTP protocol. We start simple though, with just HTTP GET and POST requests and a simple HTTP server.

Our package contains two sub-packages: client and server. An initial package layout might look like this:

http/
    __init__.py
    client/
        __init__.py
        get.py
        post.py
    server/
        __init__.py
        run.py

Some things to notice:

• The names are short and descriptive. Because client and server are subpackages of http, it’s obvious to everyone that these are an HTTP client and server. We don’t need to call them http_client and http_server.
• We’ve grouped similar functionality in sub-packages:
  • client code in a client package,
  • and server code in a server package.
• We’ve grouped closely related code into modules. E.g., all we need to do HTTP get requests goes in the get.py module.

Extending out Python package

Now imagine that we want to jump on the async programming model. While our original server used to be synchronous, we now also offer an async variant.

There are three ways to add this to our codebase:

1. create a new package named http_async
2. create new subpackages named sync and async in the http package
3. create new modules with names like async_get and async_post and async_run.

Which option would you choose?

Option 1: Create a new package

Option 1 is the simplest and most straightforward. We can create a new package named http_async and copy the contents of the http package into it. Next, we adapt all the code to be async. A user of our original package might get away with changing a single line of code: change import http to import http_async. Since asynchronous programming is a completely different paradigm, this simple swap would probably not be enough though.

Option 2: Create new subpackages

Option two means that existing users of our library, even those that don’t want to use async, need to change their code. It could be a good option when starting fresh, but it’s not such a good option for an existing library.

Option 3: Create new modules

And then there’s option three: creating new modules with async appended to the module names. Although existing users would not need to change their code, I still would not recommend this. It’s more logical to bundle the async modules in their own package, if only because you wouldn’t have to repeat the async prefix/postfix all the time when doing imports. It fits in the general objective of keeping package names and module names short and concise.

Importing modules in __init__.py

Our previous HTTP package can be improved even more, by importing the important functions inside of __init__.py. The __init__.py file is often a good place to import other modules. Let’s take our http package from above, for example. Our get.py module provides a function get() that returns a response. We can import this function into our __init__.py file:

from http.client.get import get

This is a common pattern in Python. It’s a good idea to import all the modules you need in the __init__.py file. If we use the import as described above, we can import the get function like this:

from http.client import get

Depending on the situation, and also depending on taste, you could also do this:

from http import client

client.get(...)

Without the import in __init__.py, users of our package would need to do what we did ourselves to use the get function:

from http.client.get import get

Absolute or relative imports

We’ve imported the get() function from the http.client.get module using absolute imports. You can also use relative import. The advantage of relative imports is that you can import the module you want to use without having to know the full path. The package names might even change, without breaking the code. So relative imports make your code more robust to change.

A relative import in the __init__.py file of http.client looks like this:

from .get import get

This is also one of the cases where wildcards are useful. We can import all the elements in the http.client.get module using a wildcard:

from .get import *

This is not so bad, because we know exactly what we’ve put in get.py. In addition, we can change function names in get.py without needing to change import statements. This again makes the code more flexible.

Modules vs Packages

Let’s conclude by summarizing all of the above information about Python packages and the things we learned in the previous article on Python modules. A question that often pops up, is this: what’s the difference between a module and a package?

• A modules is always a single file, while a package can contain many modules.
• A module is used to bundle Python functions, classes, constants, and anything else that you want to make reuseable. A package in turn bundles modules.
• Modules can live on their own and don’t need to be part of a package, while a package needs modules to be useful.
• Packages and modules, together, form a powerful way of organizing our code.