Announcing Earthly v0.6
We are pleased to announce the release of Earthly v0.6 - our 80th release to date! This version promotes a number of features from Experimental, or Beta status, to GA. What this means practically is that they are now enabled by default, their API is now finalized, and they have been tested thoroughly.
At Earthly we’re on a mission to build the standard foundational build layer for automation to be run upon.
The #1 reason people use Earthly today is to be able to reproduce CI builds locally. Earthly helps create a consistency layer, providing a level of guarantee that the build will run the same on your machine as on your colleagues machine, and as in CI. We call this consistency repeatability (note that this is distinct from reproducibility as full determinism is not guaranteed).
This helps avoid cases where tweaks to the CI script take long interaction cycles to test (
git commit -m "try again..." anyone?), but it also prevents CI vendor lock-in, as Earthly also helps with maintaining consistency between different CI vendors too.
Another reason people use Earthly today is because they want builds to be standardized across multiple projects. They want to put together a sort of template that allows any engineer from any team to be able to immediately execute the builds of a project they are unfamiliar with.
earthly +test always runs the unit tests. Running
earthly +lint always runs the linter. Running
earthly +docker always builds the image used in production. And so on…
This helps bring down cross-team collaboration barriers and allows for a familiar interface wherever you are in the company’s codebase.
What’s more, Earthly is able to create references across directories and across repositories. Many users leverage Earthly as a bridge between builds of different programming languages. If you need to work with artifacts or container images from varying ecosystems, like JS (NPM, Yarn etc), Java (Maven, Gradle etc), Python (Pip etc), Ruby (Bundler etc) or any other language, you can use Earthly to link these otherwise disconnected worlds.
Beyond that, there may be company-specific ways of doing things. Like tagging the docker images needs to respect a certain format. This is another key use-case that many users accomplish through Earthly.
How Earthly V0.6 Helps
Supporting all of the use-cases of a CI, while being based on containers yourself can have pros and cons. The typical container-based CI, for example, does not allow you to run containers within their container, or if it does, they are limited in functionality (e.g. mounting volumes not being available), or don’t support Docker Compose or Kubernetes well. This makes integration testing a nightmare.
With Earthly v0.6 and the newly GA’d feature
WITH DOCKER, Earthly can execute any number of containers in parallel, via an isolated Docker daemon that it starts up within. Because the daemon is local, the user can freely use any local volume mount. What’s more, as the daemon is ephemeral, it is not influenced by previous runs in any way, like a previously leftover image, volume or network configuration. Through this feature, Earthly brings the best of both worlds: a containerized build that can also execute containers within it without any compromises.
WITH DOCKER also has builtin support for Docker Compose and it has been tested with Kubernetes kind as well. Since it was released as Beta,
WITH DOCKER has been an integration testing powerhouse throughout the Earthly community. Today
WITH DOCKER becomes GA.
FROM earthly/dind:alpine WITH DOCKER \ --load kvserver:latest=+kvserver-docker \ --load kv-py-client:latest=+kvclient-docker \ --load kv-rb-client:latest=+kv-ruby-client-dockerRUN \ docker run -d kvserver:latest && \ docker run kv-py-client:latest python3 client.py fish=salmon && \ docker run kv-py-client:latest python3 client.py fish | grep salmon && \ docker run kv-rb-client:latest ruby client.rb fish | grep salmon END
To get started with integration testing head over to Earthly’s integration testing guide.
User-Defined Commands (UDCs)
In order for builds to be standardized across multiple projects, Earthly needed to provide a way for users to specify common, arbitrary ways of performing certain key tasks within a build. This might entail compiling in a specific way, or downloading dependencies from an internal artifactory, or tagging production images using a specific convention, or installing a key dependency (internal CLI tool?) in a standard way.
Whatever the use-case, Earthly’s new user-defined commands (UDCs) got you covered.
MY_COPY: COMMAND ARG src ARG dest=./ ARG recursive=false RUN cp $(if $recursive = "true"; then printf -- -r; fi) "$src" "$dest" build: FROM alpine:3.13 WORKDIR /udc-example RUN echo "hello" >./foo DO +MY_COPY --src=./foo --dest=./bar RUN cat ./bar # prints "hello"
UDCs work especially well when they are combined with the remote referencing syntax. For example, calling
will install docker-in-docker on top of the current build environment, regardless of whether it’s based on an alpine or a debian base image.
To read more about UDCs, head over to the UDC guide page in the Earthly documentation.
Many modern CI runners are often ephemeral runners - the environment is wiped clean on every run. Caching in this situation can be challenging and as a result builds can get slow.
Through BuildKit’s remote cache capabilities, Earthly v0.6 now provides shared caching via regular container registries, which helps speed up builds in this situation.
The cache can be uploaded to a registry either embedded as metadata into already uploaded images (inline cache), or as part of a separate image entry that is not meant to be used for anything else (explicit cache). These caches can be used between CI runs, but can also be shared with individual engineers to speed up builds in their local environments.
To get started with shared caching, head over to the shared cache guide in the Earthly docs.
Consistency of builds is important in cases when different engineers, or CIs are running on different CPU architectures. In Earthly v0.6 we’ve also GA’d cross-platform building, where Earthly will use QEMU to execute parts of the build that are designed for non-native platforms.
In addition, Earthly has been thoroughly tested on the new Apple Silicon (M1) and that platform is now also supported. (Please test your environment before buying an M1, though. Emulation isn’t always perfect).
To get started with multi-platform builds see the Earthly documentation.
And Much More
Earthly v0.6 packs many other spicy features - such as the ability to execute commands directly on the host via
LOCALLY, the builtin backwards compatibility with older scripts via
VERSION, support for flow control constructs such as
FOR, the ability to import Dockerfiles via
FROM DOCKERFILE, dramatically increased speed of image exports, and much more!
To read the full release notes check out the releases page on GitHub.
With v0.6 in the bag, the next question is “what’s next”. While our roadmap is publicly available on GitHub and we have previously written about our platform values, we are now also working on some exciting unique cloud-based features, which we will launch in 2022. If you’re interested in being part of the closed Beta, please email email@example.com.
Much of the work released today could not have been accomplished without the instrumental feedback from the Earthly community and the many code contributions from developers all around the planet! Thank you for making the world of builds a better place!
If you’re new to Earthly, head to our Get Earthly page and try out the tutorial. We have recently surpassed 5500 stars on GitHub, we’ve been named a top growing GitHub project on numerous occasions, and we’re hearing from community members how tremendously useful Earthly is to their work. And if you haven’t already, come say hi on Slack!