Poetry Build and Publish

Last time, I walked through packaging a simple Python module using setuptools and setup.py to generate distributions and publish them to PyPI.

Now in Part 2, I’ll show you how the Poetry dependency manager and build system simplifies parts of this process. We’ll use the same merge sort example code from Part 1, but package and distribute it with Poetry instead of setuptools.

Like in Part 1, we’ll test the package locally, but we’ll also use TestPyPI before publishing to the main package index. Poetry streamlines building and uploading distributions, but the overall workflow remains similar.

By the end, you’ll see how both setuptools and Poetry can accomplish the task of packaging and publishing Python projects. But’s it’s easier with Poetry. To quickly package a Poetry project, just run poetry publish --build on a properly configured project. But let’s go through the details step-by-step.

Code

To move our code to Poetry, first we need the same example code from Part 1 that we want to package:

def merge(list1, list2):
    merged_list = []
    i, j = 0, 0
    
    while i < len(list1) and j < len(list2):
        if list1[i] < list2[j]:
            merged_list.append(list1[i])
            i += 1
        else:
            merged_list.append(list2[j])
            j += 1
    
    # Add any remaining elements from list1 or list2
    while i < len(list1):
        merged_list.append(list1[i])
        i += 1
        
    while j < len(list2):
        merged_list.append(list2[j])
        j += 1
    
    return merged_list

To create a poetry project for this:

> pip install poetry
...
> poetry new mergefast 
Created package mergefast in mergefast

Doing that creates the structure for my package. Compared to using setuptools directly, Poetry has already initialized the configuration needed to build distributions.

.
├── README.md
├── mergefast
│   └── __init__.py
├── pyproject.toml
└── tests
    └── __init__.py

For there I just need to copy my code into core.py and add my test.py into the project.

Let’s look at the pyproject.toml:

[tool.poetry]
name = "mergefast"
version = "0.1.1"
description = ""
authors = ["Adam Gordon Bell <adam@earthly.dev>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

You can see that name, version, and other details traditionally configured in a setup.py are already here. That makes building the distribution simple.

Poetry Build

To build a package with poetry:

> poetry build 
Building mergefast (0.1.1)
  - Building sdist
  - Built mergefast-0.1.1.tar.gz
  - Building wheel
  - Built mergefast-0.1.1-py3-none-any.whl

This will build both a source distribution and a wheel. To build each separately use poetry build --format sdist and poetry build --format wheel.

Testing the Package

Because of Poetry’s focus on virtual environments, possible to test package without even building it.

> poetry shell
Spawning shell within /Users/adam/Library/Caches/pypoetry/virtualenvs/mergefast.
> pip list
Package            Version   Editable project location
------------------ --------- ---------------------------------------
certifi            2023.7.22
charset-normalizer 3.3.0
docutils           0.20.1
idna               3.4
importlib-metadata 6.8.0
jaraco.classes     3.3.0
keyring            24.2.0
markdown-it-py     3.0.0
mdurl              0.1.2
mergefast          0.1.0     /Users/adam/sandbox/mergefast/mergefast

We can also test the packages the same way as in the setuptools based article, using pip install mergefast-0.1.1.tar.gz and using an Earthfile to test installation in a clean container.

That’s a great practice. But another testing method available to us it is using https://test.pypi.org/ to test the publish process end to end.

Using Test PyPi

Before publishing your package directly to PyPI, it’s a good idea to test the entire packaging and distribution process on TestPyPI first.

TestPyPI is a separate instance of the Python Package Index (PyPI) designed specifically for testing and experimentation. It allows us to practice the process of packaging and publishing our code without affecting the main PyPI repository.

It’s basically a staging release location we can use to test things out.

To use TestPyPI, we go through the same registration and key creation process as on PyPI.

• Create an Account
• Setup 2-Factor Auth
• Create an API Key

Once you have your API key, poetry can take your API token in POETRY_PYPI_TOKEN_TESTPYPI.

export POETRY_PYPI_TOKEN_TESTPYPI=pypi-redacted

Or it can be passed as a parameter to poetry publish, if you indicate --repository testpypi :

poetry publish  --repository testpypi -u __token__ -p your_generated_token

Either way, the key is to tell poetry publish to use --repository testpypi and your package will publish to TestPyPi.

> poetry publish --build --repository testpypi -n
There are 2 files ready for publishing. Build anyway? (yes/no) [no] y 
Building mergefast (0.1.1)
  - Building sdist
  - Built mergefast-0.1.1.tar.gz
  - Building wheel
  - Built mergefast-0.1.1-py3-none-any.whl

Publishing mergefast (0.1.1) to PyPI
 - Uploading mergefast-0.1.1-py3-none-any.whl 100%
 - Uploading mergefast-0.1.1.tar.gz 100%

Once that package is up on test PyPI, you can test it end to end with pip install --index-url https://test.pypi.org/simple/ mergefast. Or as I’m fond of doing wrapping the whole thing up in a Earthfile target, so I can test it end to end.

poetry-test-publish:
    FROM +build
    RUN poetry config repositories.testpypi https://test.pypi.org/legacy/
    RUN poetry publish --build --repository testpypi -n

test-pypi-install:
    FROM python:3.11-buster
    RUN pip install --index-url https://test.pypi.org/simple/ mergefast
    COPY tests .
    RUN python test.py

Then I can always test the latest test published package on a clean container like this:

> earthly +test-pypi-install
+test-pypi-install | --> COPY +build/dist dist
+test-pypi-install | --> expandargs ls ./dist/*.tar.gz
+test-pypi-install | --> RUN pip install --index-url https://test.pypi.org/simple/ mergefast
...
+test-pypi-install| --> COPY tests .
+test-pypi-install | --> RUN python test.py
+test-pypi-install | timsort took 6.349711754999589 seconds
+test-pypi-install | mergefast took 27.499190239999734 seconds

The Final Poetry Publish

And now that we have tested our package end to end, we can publish it onto PyPi.org with a simple poetry publish --build

> poetry publish --build
There are 2 files ready for publishing. Build anyway? (yes/no) [no] y 
Building mergefast (0.1.1)
  - Building sdist
  - Built mergefast-0.1.1.tar.gz
  - Building wheel
  - Built mergefast-0.1.1-py3-none-any.whl

Publishing mergefast (0.1.1) to PyPI
 - Uploading mergefast-0.1.1-py3-none-any.whl 100%
 - Uploading mergefast-0.1.1.tar.gz 100%

And there you go, the package is published and you have a repeatable process for doing so. For Python only packages, poetry publish is a simple way to go. No need for setuptools and setup.py at all.

For a simple package like this, this whole testing workflow might be overkill. But if your package has users and you want to make sure you don’t break their workflow I think it makes sense to sanity test your packages.

Next Up: C Extensions

Next up, let’s tackle the c version of this code and publish a python c extension on to PyPi. Publishing a native extension is a bit trickier, but we now have the skills to easily tackle this problem. The testing setup we’ve established here, with Earthly and test.pypy.org and our knowledge of poetry and setup tools will all come together in part three

There is not a lot of ways for packaging to wrong with a single file package, but in the next article, testing end to end is will really pay off.