In this Series
Table of contents
Last time, I walked through packaging a simple Python module using setuptools and setup.py to generate distributions and publish them to PyPI.
Now in Part 2, I’ll show you how the Poetry dependency manager and build system simplifies parts of this process. We’ll use the same merge sort example code from Part 1, but package and distribute it with Poetry instead of setuptools.
Like in Part 1, we’ll test the package locally, but we’ll also use TestPyPI before publishing to the main package index. Poetry streamlines building and uploading distributions, but the overall workflow remains similar.
By the end, you’ll see how both setuptools and Poetry can accomplish the task of packaging and publishing Python projects. But’s it’s easier with Poetry. To quickly package a Poetry project, just run
poetry publish --build on a properly configured project. But let’s go through the details step-by-step.
To move our code to Poetry, first we need the same example code from Part 1 that we want to package:
def merge(list1, list2): =  merged_list = 0, 0 i, j while i < len(list1) and j < len(list2): if list1[i] < list2[j]: merged_list.append(list1[i])+= 1 i else: merged_list.append(list2[j])+= 1 j # Add any remaining elements from list1 or list2 while i < len(list1): merged_list.append(list1[i])+= 1 i while j < len(list2): merged_list.append(list2[j])+= 1 j return merged_list
To create a poetry project for this:
> pip install poetry ... > poetry new mergefast Created package mergefast in mergefast
Doing that creates the structure for my package. Compared to using setuptools directly, Poetry has already initialized the configuration needed to build distributions.
. ├── README.md ├── mergefast │ └── __init__.py ├── pyproject.toml └── tests └── __init__.py
For there I just need to copy my code into
core.py and add my test.py into the project.
Let’s look at the
[tool.poetry] name = "mergefast" version = "0.1.1" description = "" authors = ["Adam Gordon Bell <email@example.com>"] readme = "README.md" [tool.poetry.dependencies] python = "^3.11" [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api"
You can see that name, version, and other details traditionally configured in a setup.py are already here. That makes building the distribution simple.
To build a package with poetry:
> poetry build Building mergefast (0.1.1) - Building sdist - Built mergefast-0.1.1.tar.gz - Building wheel - Built mergefast-0.1.1-py3-none-any.whl
This will build both a source distribution and a wheel. To build each separately use
poetry build --format sdist and
poetry build --format wheel.
Testing the Package
Because of Poetry’s focus on virtual environments, possible to test package without even building it.
> poetry shell Spawning shell within /Users/adam/Library/Caches/pypoetry/virtualenvs/mergefast. > pip list Package Version Editable project location ------------------ --------- --------------------------------------- certifi 2023.7.22 charset-normalizer 3.3.0 docutils 0.20.1 idna 3.4 importlib-metadata 6.8.0 jaraco.classes 3.3.0 keyring 24.2.0 markdown-it-py 3.0.0 mdurl 0.1.2 mergefast 0.1.0 /Users/adam/sandbox/mergefast/mergefast
We can also test the packages the same way as in the setuptools based article, using
pip install mergefast-0.1.1.tar.gz and using an Earthfile to test installation in a clean container.
That’s a great practice. But another testing method available to us it is using
https://test.pypi.org/ to test the publish process end to end.
Using Test PyPi
Before publishing your package directly to PyPI, it’s a good idea to test the entire packaging and distribution process on TestPyPI first.
TestPyPI is a separate instance of the Python Package Index (PyPI) designed specifically for testing and experimentation. It allows us to practice the process of packaging and publishing our code without affecting the main PyPI repository.
It’s basically a staging release location we can use to test things out.
To use TestPyPI, we go through the same registration and key creation process as on PyPI.
- Create an Account
- Setup 2-Factor Auth
- Create an API Key
Once you have your API key, poetry can take your API token in
Or it can be passed as a parameter to
poetry publish, if you indicate
--repository testpypi :
poetry publish --repository testpypi -u __token__ -p your_generated_token
Either way, the key is to tell
poetry publish to use
--repository testpypi and your package will publish to TestPyPi.
> poetry publish --build --repository testpypi -n There are 2 files ready for publishing. Build anyway? (yes/no) [no] y Building mergefast (0.1.1) - Building sdist - Built mergefast-0.1.1.tar.gz - Building wheel - Built mergefast-0.1.1-py3-none-any.whl Publishing mergefast (0.1.1) to PyPI - Uploading mergefast-0.1.1-py3-none-any.whl 100% - Uploading mergefast-0.1.1.tar.gz 100%
Once that package is up on test PyPI, you can test it end to end with
pip install --index-url https://test.pypi.org/simple/ mergefast. Or as I’m fond of doing wrapping the whole thing up in a Earthfile target, so I can test it end to end.
poetry-test-publish:FROM +build RUN poetry config repositories.testpypi https://test.pypi.org/legacy/ RUN poetry publish --build --repository testpypi -n test-pypi-install:FROM python:3.11-buster RUN pip install --index-url https://test.pypi.org/simple/ mergefast COPY tests . RUN python test.py
Then I can always test the latest test published package on a clean container like this:
> earthly +test-pypi-install +test-pypi-install | --> COPY +build/dist dist +test-pypi-install | --> expandargs ls ./dist/*.tar.gz +test-pypi-install | --> RUN pip install --index-url https://test.pypi.org/simple/ mergefast ... +test-pypi-install| --> COPY tests . +test-pypi-install | --> RUN python test.py +test-pypi-install | timsort took 6.349711754999589 seconds +test-pypi-install | mergefast took 27.499190239999734 seconds
The Final Poetry Publish
And now that we have tested our package end to end, we can publish it onto PyPi.org with a simple
poetry publish --build
> poetry publish --build There are 2 files ready for publishing. Build anyway? (yes/no) [no] y Building mergefast (0.1.1) - Building sdist - Built mergefast-0.1.1.tar.gz - Building wheel - Built mergefast-0.1.1-py3-none-any.whl Publishing mergefast (0.1.1) to PyPI - Uploading mergefast-0.1.1-py3-none-any.whl 100% - Uploading mergefast-0.1.1.tar.gz 100%
And there you go, the package is published and you have a repeatable process for doing so. For Python only packages,
poetry publish is a simple way to go. No need for setuptools and
setup.py at all.
For a simple package like this, this whole testing workflow might be overkill. But if your package has users and you want to make sure you don’t break their workflow I think it makes sense to sanity test your packages.
Next Up: C Extensions
Next up, let’s tackle the c version of this code and publish a python c extension on to PyPi. Publishing a native extension is a bit trickier, but we now have the skills to easily tackle this problem. The testing setup we’ve established here, with Earthly and test.pypy.org and our knowledge of poetry and setup tools will all come together in part three
There is not a lot of ways for packaging to wrong with a single file package, but in the next article, testing end to end is will really pay off.