Python Poetry: Package and venv Management Made Easy

Poetry is a package manager for Python. It’s a tool that serves multiple purposes. It:

• Conveniently manages your virtual environments (no need to create them manually)
• Is a great tool for installing Python packages (you don’t need pip install anymore)
• Helps you to manage your dependencies
• Can be used for building Python packages that you can share with the world

This article takes a close look at Python Poetry. We’ll first install it properly, and then I’ll show you how to create projects with Poetry and manage your virtual environment and project dependencies.

Why Poetry?

Poetry has several advantages over using pip or other package managers. For starters, it’s an intuitive tool for installing Python packages in a virtual environment. In addition, it conveniently manages your virtual environments as well. Poetry also manages your dependencies: it will try to find a combination of dependencies that work together nicely and store this combination in a so-called lock file.

Once you get to a point where you want to distribute your software as a Python package, Poetry greatly simplifies the building and uploading of Python packages as well!

Installing Python Poetry

Poetry itself recommends that you use their installer. This installer will isolate poetry from the rest of your system by vendorizing it so that it won’t clash with other dependencies you might have installed through pip.

Using the installer

Poetry just recently updated its installer. In this tutorial, I will show you the most up-to-date method of using it. You’ll often find other tutorials referencing another URL. I recommend not using that one. To use their latest and greatest installer, you can use this command on Windows:

(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -

And on a Mac or Linux system:

curl -sSL https://install.python-poetry.org | python3 -

Using pip

Like most Python packages, you can install Python Poetry with pip too if you prefer. It’s not the recommended method, though:

pip install --user poetry

If you use pip, I recommend installing it in your user account. These days, that’s probably the default behavior for your Python installation already. However, we can ensure this by explicitly using the --user flag as demonstrated above.

Test if your poetry installation works

Finally, test if the installation worked with the following command:

poetry --version

If you see something like Poetry 0.13.0, it means the installation was successful.

Keeping Python Poetry up-to-date

If you used the Poetry installer script, you can use the following command to update Poetry:

poetry self update

If you decide to use pip, you’ll need to use pip to update poetry:

pip install --upgrade poetry

Starting a project with Python Poetry

When you start a project with Poetry, quite a few things will happen. Let’s begin by creating a new project:

poetry new demo

This command created a new directory called demo in the current directory. When we inspect the directory, we’ll see a few files:

demo
├── demo
│   └── __init__.py
├── pyproject.toml
├── README.rst
└── tests
    ├── __init__.py
    └── test_demo.py

The demo directory is where your project resides. In it, another directory called demo was created. This is your main package, containing an __init__.py file for your convenience. You’re free to rename or remove this package, it’s just there for your convenience.

The command also created a pyproject.toml file, containing the project’s metadata. We’ll take a closer look at that file shortly. Then there’s a README.rst file, which should contain a short description of the project. It’s similar to the more common Markdown README.md file. I personally always rename this file right off the bat to README.md.

Finally, we see a tests directory, a package that will contain your unit tests, if any.

You are free to remove any of these files and directories, except for the pyproject.toml file. E.g., when you don’t want to work with packages and modules in your project, you can remove the demo directory. When you don't want to work with tests, you can remove the tests directory. The basic structure is just there as a suggestion and to entice people to adhere to some best practices.

The pyproject.toml file

Now let’s take a closer look at the pyproject.toml file. It’s a TOML file that contains metadata for the project. TOML is a simple format that is easy to read and write. The format will look familiar if you ever worked with .ini files, but TOML has an official specification and is more expressive. Our pyproject.toml file looks like this:

[tool.poetry]
name = "demo"
version = "0.1.0"
description = ""
authors = ["Your name <your@e-mail.address>"]

[tool.poetry.dependencies]
python = "^3.10"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

As you can see, the file is divided in sections:

1. The tool.poetry section contains metadata for the project like its name, version, description, and author(s).
2. tool.poetry.dependencies contains the dependencies for the project. These dependencies are required to run the software in this project.
3. tool.poetry.dev-dependencies contains dependencies that are required for developers working on this project. These dependencies are not needed for the final product but only for the development and testing of the application.
4. Finally, the fourth section is the build-system section, which contains settings for the build system. We’ll ignore this for now.

Install and remove packages with Python Poetry

To add and install packages (your project dependencies), you can either edit the pyproject.toml file, or use the poetry add <package> command. I strongly suggest you use the poetry add command since it does more than simply edit the file. It instantly:

• Looks for a suitable version that does not conflict with other dependencies
• Installs the package in the accompanying virtual environment
• Creates or updates a lock file called poetry.lock

Let’s see this in action. We’ll install the latest version of the popular Python requests package:

poetry add requests

The output should look something like this:

Creating virtualenv demo-IUjJzrPZ-py3.10 in C:\Users\erik\AppData\Local\pypoetry\Cache\virtualenvs
Using version ^2.28.0 for requests

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 15 installs, 0 updates, 0 removals

  • Installing pyparsing (3.0.9)
  • Installing atomicwrites (1.4.0)
  • Installing attrs (21.4.0)
  • Installing certifi (2022.6.15)
  • Installing charset-normalizer (2.0.12)
  • Installing colorama (0.4.5)
  • Installing idna (3.3)
  • Installing packaging (21.3)
  • Installing more-itertools (8.13.0)
  • Installing pluggy (0.13.1)
  • Installing py (1.11.0)
  • Installing urllib3 (1.26.9)
  • Installing wcwidth (0.2.5)
  • Installing pytest (5.4.3)
  • Installing requests (2.28.0)

What happened?

First of all: this command triggered the creation of a virtual environment because this is the first time we actually used the project. The virtual environment is not created inside the project directory but in a directory in your user account. In my case, that directory is: C:\Users\erik\AppData\Local\pypoetry\Cache\virtualenvs.

Separating the virtual environment from your project makes it easier to exclude the virtual environment from version control. If you prefer to have the virtual environment in the project directory, you can set the virtualenvs.in-project=true option with the following command:

poetry config virtualenvs.in-project true

This setting will apply to all future projects.

Next, a bunch of packages gets installed, together with our requested package. These are all dependencies of the requests package. We can inspect the pyproject.toml file to see that the package was added there as well:

[tool.poetry.dependencies]
python = "^3.10"
requests = "^2.28.0"

It only lists the package we requested and not all its dependencies. After all, these dependencies will be sorted out by the package manager at installation time and the optimal versions will be selected based on all the requirements you specify. However, the dependencies will be stored in the lock file. In the output, we can see a message, Writing lock file. Let’s find out what this lock file is for.

Locking packages

After installing requests, a new file was created as well, called poetry.lock. The file is a bit too large to be included on this page, but if you inspect it yourself, you’ll see many entries like this:

[[package]]
name = "idna"
version = "3.3"
description = "Internationalized Domain Names in Applications (IDNA)"
category = "main"
optional = false
python-versions = ">=3.5"

These are all packages that are requirements of the requests package, including requests itself. Locking the versions of these packages in a poetry.lock file ensures that the versions of these packages are always the same when recreating the virtual environment.

This is a powerful feature because it ensures that our project builds are deterministic. This is important for CI/CD, and it also helps a lot for developers to have a consistent environment that is easy to create and recreate. In addition, it serves as a cache. When reinstalling the dependencies when there’s a lock file, Poetry doesn’t need to resolve all the dependencies since they are already stored in the lock file.

Adding developer dependencies

As mentioned, there’s also a section called tool.poetry.dev-dependencies that lists all the dependencies required to build and test the project. These dependencies are not needed to use and run the project (as a regular user); hence, they won’t get installed automatically either.

To add a dev dependency, we can add the --dev option to the add command, like so:

poetry add --dev <package name>

Removing packages

To remove a package from your project, use the following command:

poetry remove <package name>

This will remove the package and all of its dependencies unless those dependencies are also required by another package listed in your pyproject.toml file. Similarly to adding dev dependencies, we can also remove them with the extra --dev option:

poetry remove --dev <package name>

Install the dependencies of an existing Python Poetry project

When you have an existing project that is based on Poetry, you can install all the dependencies listed in pyproject.toml at once with the following command:

poetry install

If there is a poetry.lock file present, this will install all the packages from that lock file. If not, Poetry will resolve the dependencies, install the packages, and generate a new lock file.

Using your project’s virtual environment

Poetry created a virtual environment for us as well. We’ll now learn how we can use this virtual environment.

Running your script

You can run a script in your project’s virtual environment by using the poetry run command. If you created a file called main.py, for example, you can run it with:

poetry run python main.py

In the same way, you can also run commands that got installed in your virtual environment. E.g., to run pytest, which is installed as a developer dependency, you can use poetry run pytest.

Activating the virtual environment

You can use the poetry shell command if you want to use the virtual environment interactively. This command will activate the virtual environment in a newly launched shell. Once in there, you don’t need to use poetry run to run your scripts. You can use the virtual environment like you would with any other virtual environment.

Updating your dependencies

There are two scenarios when it comes to updating packages:

1. Updating packages according to the pyproject.toml defined version constraints
2. Updating a package to its latest (or a specific) version

Because the package versions get locked in the poetry.lock file, they won’t be updated automatically. This is a good thing: it ensures that your project keeps working as intended, even when you try to rebuild it a year from now. However, we all know how important it is to keep your software and dependencies up-to-date, so we do need a way to update them.

To update the dependencies, you can use the poetry update command. This command updates the dependencies in the virtual environment and then updates the poetry.lock file. It will still adhere to the constraints of the pyproject.toml file though. E.g., if you have defined a dependency that you want to keep above version 3.0.0 and below 4.0.0, poetry will try to update it to the latest 3.x.x version that is still below 4.0.0.

We can also update a specific package with poetry update package_name.

Note that using poetry update is equivalent to removing the poetry.lock file and running poetry install again.

Update packages to the latest version

If you want to be more rigorous and want to try the latest version, you need to use the poetry add command. The following commands demonstrate how to update the requests package to its latest version or to a specific version:

# Install the latest version
poetry add requests@latest

# Install a specific version
poetry add requests@2.28.0

Building and publishing packages

Poetry makes it easy to build packages. I’ll only provide a quick intro to this. For full instructions, I recommend you to read the official documentation.

In essence, to build a package, you can use the poetry build command. This command will create two files in a newly created directory called dist. The first file is a wheel file (.whl), a compiled package, and the second is a tar.gz file containing the package’s source code.

With these two files ready, you can publish them to a repository. By default, that will be the public PyPI repository. You can also publish to a private repository, though. No matter where you publish, you’ll probably need to set up credentials for the repository first so poetry can access it.

For PyPI, it is recommended to create an access token instead of the more old-fashioned username and password. Once you have created such an access token, you can configure it with:

poetry config pypi-token.pypi <token>

Now you’re ready to publish your package with:

poetry publish

Poetry is interoperable

Poetry has a few features that make it play nicely with existing projects. It also has an export feature to convert your project back to an old-school pip-based one.

Converting an existing project to Poetry

To use Poetry in a pre-existing project, you can use the poetry init command. This command will create a pyproject.toml file in the current directory:

cd my-project
poetry init

This will start an interactive wizard that helps you create a pyproject.toml file for your existing project!

Exporting to a regular requirements.txt file

You can export the list of dependencies you created with Poetry to a requirements.txt file. This can come in handy when you need to work with others that don’t use Poetry, for example. Where I use this more is when creating Docker images. As part of the build steps, I export my dependencies to a requirements.txt file, so I don’t need to install and use Poetry inside the Dockerfile.

Somewhat recently, poetry moved this functionality to the Poetry export plugin that you need to install separately. So if this is not working out of the box for you, you might need to install that plugin with the command:

poetry self add poetry-plugin-export

# Or with pip:
pip install poetry-plugin-exportCode language: plaintext (plaintext)

To export your dependencies to a requirements.txt file, use the following command:

poetry export -f requirements.txt -o requirements.txt

And because requirements.txt is the default format, you can leave out the -f requirements.txt part if you want.

A cheat sheet of Poetry commands

The following table lists the most-used poetry commands, including a short description of what they do.

Poetry vs Pipenv

Although I’ve written about an alternative too, called Pipenv, I’d highly recommend Poetry. Even though Poetry and Pipenv have a lot of overlap. I feel most people will agree poetry is, by now, the better alternative. Pipenv paved the way, but development on it stalled a little in the past years. Meanwhile, poetry gained a lot of traction and users.

A couple of advantages Poetry offers over Pipenv are:

• It uses the now standardized pyproject.toml
• Many say it is faster in resolving dependencies
• Poetry also helps you build your project and publish it on PyPI or a private repository.
• It’s subjective, but I like the (colorized) command-line interface more

Learn more

This article is part of a free Python Tutorial. You can browse the tutorial with the navigation buttons at the top and bottom of the article or use the navigation menu. If you want to learn more about Poetry, you can try the following resources:

• The Poetry introduction here has detailed installation instructions, including instructions to enable auto-completion on the command line.
• If you want to learn how to define version ranges exactly, the Poetry site has an extensive section on dependency specification.
• You can learn more about creating and publishing your own packages on this page.

Conclusion

You learned what Poetry has to offer. By now, you probably can see how a tool like Poetry can make your life a lot easier than using pip install and manually creating virtual environments. We’ve looked at how a Poetry project is structured, what the pyproject.toml and poetry.lock files are for, and how to install and update your project’s dependencies. We also looked at creating your own package and how to share it with others through PyPI.