Poetry is a package manager for Python. It’s a tool that serves multiple purposes. It:
- Conveniently manages your virtual environments (no need to create them manually)
- Is a great tool for installing Python packages (you don’t need pip install anymore)
- Helps you to manage your dependencies
- Can be used for building Python packages that you can share with the world
This article takes a close look at Python Poetry. We’ll first install it properly, and then I’ll show you how to create projects with Poetry and manage your virtual environment and project dependencies.
Table of contents
Poetry has several advantages over using pip or other package managers. For starters, it’s an intuitive tool for installing Python packages in a virtual environment. In addition, it conveniently manages your virtual environments as well. Poetry also manages your dependencies: it will try to find a combination of dependencies that work together nicely and store this combination in a so-called lock file.
Once you get to a point where you want to distribute your software as a Python package, Poetry greatly simplifies the building and uploading of Python packages as well!
Installing Python Poetry
Poetry itself recommends that you use their installer. This installer will isolate poetry from the rest of your system by vendorizing it, so it won’t clash with other dependencies you might have installed through
Using the installer
Poetry just recently updated its installer. In this tutorial, I will show you the most up-to-date method of using it. You’ll often find other tutorials referencing another URL. I recommend not using that one. To use their latest and greatest installer, you can use this command on Windows:
(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -
And on a Mac or Linux system:
curl -sSL https://install.python-poetry.org | python3 -
Like most Python packages, you can install Python Poetry with
pip too if you prefer. It’s not the recommended method, though:
pip install --user poetry
If you use pip, I recommend installing it in your user account. These days, that’s probably the default behavior for your Python installation already. However, we can ensure this by explicitly using the
--user flag as demonstrated above.
Test if your poetry installation works
Finally, test if the installation worked with the following command:
If you see something like
Poetry 0.13.0, it means the installation was successful.
Keeping Python Poetry up-to-date
If you used the Poetry installer script, you can use the following command to update Poetry:
poetry self update
If you decide to use pip, you’ll need to use pip to update poetry:
pip install --upgrade poetry
Starting a project with Python Poetry
When you start a project with Poetry, quite a few things will happen. Let’s begin by creating a new project:
poetry new demo
This command created a new directory called
demo in the current directory. When we inspect the directory, we’ll see a few files:
demo ├── demo │ └── __init__.py ├── pyproject.toml ├── README.rst └── tests ├── __init__.py └── test_demo.py
demo directory is where your project resides. In it, another directory called
demo was created. This is your main package, containing an
__init__.py file for your convenience. You’re free to rename or remove this package, it’s just there for your convenience.
The command also created a
pyproject.toml file, containing the project’s metadata. We’ll take a closer look at that file shortly. Then there’s a
README.rst file, which should contain a short description of the project. It’s similar to the more common Markdown README.md file. I personally always rename this file right off the bat to README.md.
Finally, we see a
tests directory, a package that will contain your unit tests, if any.
You are free to remove any of these files and directories, except for the
pyproject.toml file. E.g., when you don’t want to work with packages and modules in your project, you can remove the
demo directory. When you don't want to work with tests, you can remove the tests directory. The basic structure is just there as a suggestion and to entice people to adhere to some best practices.
The pyproject.toml file
Now let’s take a closer look at the
pyproject.toml file. It’s a TOML file that contains metadata for the project. TOML is a simple format that is easy to read and write. The format will look familiar if you ever worked with .ini files, but TOML has an official specification and is more expressive. Our
pyproject.toml file looks like this:
[tool.poetry] name = "demo" version = "0.1.0" description = "" authors = ["Your name <email@example.com>"] [tool.poetry.dependencies] python = "^3.10" [tool.poetry.dev-dependencies] pytest = "^5.2" [build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api"
As you can see, the file is divided in sections:
tool.poetrysection contains metadata for the project like its name, version, description, and author(s).
tool.poetry.dependenciescontains the dependencies for the project. These dependencies are required to run the software in this project.
- tool.poetry.dev-dependencies contains dependencies that are required for developers working on this project. These dependencies are not needed for the final product but only for the development and testing of the application.
- Finally, the fourth section is the
build-systemsection, which contains settings for the build system. We’ll ignore this for now.
Install and remove packages with Python Poetry
To add and install packages (your project dependencies), you can either edit the
pyproject.toml file, or use the
poetry add <package> command. I strongly suggest you use the
poetry add command since it does more than simply edit the file. It instantly:
- Looks for a suitable version that does not conflict with other dependencies
- Installs the package in the accompanying virtual environment
- Creates or updates a lock file called
Let’s see this in action. We’ll install the latest version of the popular Python requests package:
poetry add requests
The output should look something like this:
Creating virtualenv demo-IUjJzrPZ-py3.10 in C:\Users\erik\AppData\Local\pypoetry\Cache\virtualenvs Using version ^2.28.0 for requests Updating dependencies Resolving dependencies... Writing lock file Package operations: 15 installs, 0 updates, 0 removals • Installing pyparsing (3.0.9) • Installing atomicwrites (1.4.0) • Installing attrs (21.4.0) • Installing certifi (2022.6.15) • Installing charset-normalizer (2.0.12) • Installing colorama (0.4.5) • Installing idna (3.3) • Installing packaging (21.3) • Installing more-itertools (8.13.0) • Installing pluggy (0.13.1) • Installing py (1.11.0) • Installing urllib3 (1.26.9) • Installing wcwidth (0.2.5) • Installing pytest (5.4.3) • Installing requests (2.28.0)
First of all: this command triggered the creation of a virtual environment because this is the first time we actually used the project. The virtual environment is not created inside the project directory but in a directory in your user account. In my case, that directory is:
Separating the virtual environment from your project makes it easier to exclude the virtual environment from version control. If you prefer to have the virtual environment in the project directory, you can set the
virtualenvs.in-project=true option with the following command:
poetry config virtualenvs.in-project true
This setting will apply to all future projects.
Next, a bunch of packages gets installed, together with our requested package. These are all dependencies of the
requests package. We can inspect the
pyproject.toml file to see that the package was added there as well:
[tool.poetry.dependencies] python = "^3.10" requests = "^2.28.0"
It only lists the package we requested and not all its dependencies. After all, these dependencies will be sorted out by the package manager at installation time and the optimal versions will be selected based on all the requirements you specify. However, the dependencies will be stored in the lock file. In the output, we can see a message, Writing lock file. Let’s find out what this lock file is for.
requests, a new file was created as well, called
poetry.lock. The file is a bit too large to be included on this page, but if you inspect it yourself, you’ll see many entries like this:
[[package]] name = "idna" version = "3.3" description = "Internationalized Domain Names in Applications (IDNA)" category = "main" optional = false python-versions = ">=3.5"
These are all packages that are requirements of the
requests package, including
requests itself. Locking the versions of these packages in a
poetry.lock file ensures that the versions of these packages are always the same when recreating the virtual environment.
This is a powerful feature because it ensures that our project builds are deterministic. This is important for CI/CD, and it also helps a lot for developers to have a consistent environment that is easy to create and recreate. In addition, it serves as a cache. When reinstalling the dependencies when there’s a lock file, Poetry doesn’t need to resolve all the dependencies since they are already stored in the lock file.
Adding developer dependencies
As mentioned, there’s also a section called
tool.poetry.dev-dependencies that lists all the dependencies required to build and test the project. These dependencies are not needed to use and run the project (as a regular user); hence, they won’t get installed automatically either.
To add a dev dependency, we can add the
--dev option to the add command, like so:
poetry add --dev <package name>
To remove a package from your project, use the following command:
poetry remove <package name>
This will remove the package and all of its dependencies unless those dependencies are also required by another package listed in your
pyproject.toml file. Similarly to adding dev dependencies, we can also remove them with the extra
poetry remove --dev <package name>
Install the dependencies of an existing Python Poetry project
When you have an existing project that is based on Poetry, you can install all the dependencies listed in
pyproject.toml at once with the following command:
If there is a
poetry.lock file present, this will install all the packages from that lock file. If not, Poetry will resolve the dependencies, install the packages, and generate a new lock file.
Using your project’s virtual environment
Poetry created a virtual environment for us as well. We’ll now learn how we can use this virtual environment.
Running your script
You can run a script in your project’s virtual environment by using the
poetry run command. If you created a file called
main.py, for example, you can run it with:
poetry run python main.py
In the same way, you can also run commands that got installed in your virtual environment. E.g., to run
pytest, which is installed as a developer dependency, you can use
poetry run pytest.
Activating the virtual environment
You can use the
poetry shell command if you want to use the virtual environment interactively. This command will activate the virtual environment in a newly launched shell. Once in there, you don’t need to use poetry run to run your scripts. You can use the virtual environment like you would with any other virtual environment.
Updating your dependencies
There are two scenarios when it comes to updating packages:
- Updating packages according to the
pyproject.tomldefined version constraints
- Updating a package to its latest (or a specific) version
Because the package versions get locked in the
poetry.lock file, they won’t be updated automatically. This is a good thing: it ensures that your project keeps working as intended, even when you try to rebuild it a year from now. However, we all know how important it is to keep your software and dependencies up-to-date, so we do need a way to update them.
To update the dependencies, you can use the
poetry update command. This command updates the dependencies in the virtual environment and then updates the
poetry.lock file. It will still adhere to the constraints of the
pyproject.toml file though. E.g., if you have defined a dependency that you want to keep above version 3.0.0 and below 4.0.0, poetry will try to update it to the latest
3.x.x version that is still below
We can also update a specific package with
poetry update package_name.
Note that using
poetry update is equivalent to removing the
poetry.lock file and running
poetry install again.
Update packages to the latest version
If you want to be more rigorous and want to try the latest version, you need to use the
poetry add command. The following commands demonstrate how to update the
requests package to its latest version or to a specific version:
# Install the latest version poetry add requests@latest # Install a specific version poetry add firstname.lastname@example.org
Building and publishing packages
Poetry makes it easy to build packages. I’ll only provide a quick intro to this. For full instructions, I recommend you to read the official documentation.
In essence, to build a package, you can use the
poetry build command. This command will create two files in a newly created directory called
dist. The first file is a wheel file (.whl), a compiled package, and the second is a tar.gz file containing the package’s source code.
With these two files ready, you can publish them to a repository. By default, that will be the public PyPI repository. You can also publish to a private repository, though. No matter where you publish, you’ll probably need to set up credentials for the repository first so poetry can access it.
For PyPI, it is recommended to create an access token instead of the more old-fashioned username and password. Once you have created such an access token, you can configure it with:
poetry config pypi-token.pypi <token>
Now you’re ready to publish your package with:
Poetry is interoperable
Poetry has a few features that make it play nicely with existing projects. It also has an export feature to convert your project back to an old-school pip-based one.
Converting an existing project to Poetry
To use Poetry in a pre-existing project, you can use the
poetry init command. This command will create a
pyproject.toml file in the current directory:
cd my-project poetry init
This will start an interactive wizard that helps you create a
pyproject.toml file for your existing project!
Exporting to a regular requirements.txt file
You can export the list of dependencies you created with Poetry to a
requirements.txt file. This can come in handy when you need to work with others that don’t use Poetry, for example. Where I use this more is when creating Docker images. As part of the build steps, I export my dependencies to a
requirements.txt file, so I don’t need to install and use Poetry inside the
To export your dependencies to a
requirements.txt file, use the following command:
poetry export -f requirements.txt > requirements.txt
A cheat sheet of Poetry commands
The following table lists the most-used poetry commands, including a short description of what they do.
|Shows the version of your poetry installation|
|Create a new poetry project|
|Start a wizard that helps you convert an existing project to a Poetry project|
|Add package to |
|Remove package from your project (including its dependencies)|
|List the installed packages|
|Export the list of dependencies (currently only to requirements.txt format)|
|Install all dependencies of the current poetry project. Uses |
|Run the command inside the project’s virtual environment|
|Start a new shell with the project’s virtual environment activated|
Poetry vs Pipenv
Although I’ve written about an alternative too, called Pipenv, I’d highly recommend Poetry. Even though Poetry and Pipenv have a lot of overlap. I feel most people will agree poetry is, by now, the better alternative. Pipenv paved the way, but development on it stalled a little in the past years. Meanwhile, poetry gained a lot of traction and users.
A couple of advantages Poetry offers over Pipenv are:
- It uses the now standardized
- Many say it is faster in resolving dependencies
- Poetry also helps you build your project and publish it on PyPI or a private repository.
- It’s subjective, but I like the (colorized) command-line interface more
If you want to learn more about this great tool, you can try the following resources:
- The Poetry introduction here has detailed installation instructions, including instructions to enable auto-completion on the command line.
- If you want to learn how to define version ranges exactly, the Poetry site has an extensive section on dependency specification.
- You can learn more about creating and publishing your own packages on this page.
You learned what Poetry has to offer. By now, you probably can see how a tool like Poetry can make your life a lot easier than using pip install and manually creating virtual environments. We’ve looked at how a Poetry project is structured, what the
poetry.lock files are for, and how to install and update your project’s dependencies. We also took a quick look at creating your own package and how to share it with others through PyPI.