Earthly for Monorepos - Make Builds Super Simple

What is a Monorepo?

A centralized source of truth for dev teams. Multiple projects stored in one repository for easier dependency management and improved collaboration.

Why Are Monorepos Gaining Popularity?

Having every project’s code in one place simplifies project management.
Monorepos enable easy reuse of developer infrastructure – like build and release scripts – and more straightforward dependency management across projects.
Monorepos foster better team collaboration, with all project assets in one place.

Challenges with Monorepo Builds

Monorepo builds are frequently inefficient because small changes in individual projects trigger rebuilds of the whole repo.
Large monorepos can lead to increased build and test times.
They require more compute resources, potentially increasing costs.
Managing builds in these complex environments can be challenging.
Standard CI/CD systems often require workarounds or hacks to work effectively with monorepos.

Overcoming Monorepo Build Challenges

Incremental Builds: Only rebuilding the parts of the repository that have changed.
Parallel Processing: Breaking up builds into independent tasks and running those tasks in parallel, sometimes across multiple cores or machines.
Caching: Storing and reusing build components, such as generated code and binaries, if their determinants haven’t changed.

Why Earthly Excels in Monorepos?

Earthly simplifies and speeds up monorepo builds by only building what's needed, parallelization, caching, and a modern import system for better code reuse.

Build Graphs

Monorepos often involve intricate and interconnected build structures. Changes in one part of the repo can impact several others.
Earthly’s build graphs allow for managing these complex structures effectively.
Reuses unchanged components within the build graph, avoiding unnecessary recompilation.

Well-Known Dependencies

In monorepos, tracking dependencies can be challenging due to the intertwined nature of projects.
Earthly understands and tracks build dependencies and changes.
Determines the minimal rebuild scope, saving time by not rebuilding unchanged components.

Automatic Parallelization

Parallel processing for builds in a monorepo can significantly reduce build times across multiple projects
Earthly infers parallelism in build processes based on clear interdependencies.
Plan for future versions to run in compute clusters for increased speed and scalability.

Holistic Layer Caching

Expands Docker's layer caching to all build artifacts, not just container images.
Reuses build components like generated code and binaries if unchanged, speeding up the build process.
Particularly beneficial in monorepos, where multiple projects share common components.

Instantly Available Cache

A feature of Earthly Satellites.
Provides cache sharing between CI runs without download/upload delays.
Satellites offer high cache hit rates and facilitate cache reuse among team members outside of CI.

Modern Import System

Earthly can be used to reference and build targets from other directories in a monorepo or even other repositories.
Helps foster effective reuse of common build logic, ensure uniformity, and minimize redundancy across projects, especially in a monorepo.

Why Earthly?

With Earthly, all builds are self-contained, consistent, portable, and language agnostic.

Before:

Bash and YAML spaghetti

Clunky and Brittle

Build guru required

With Earthly:

Consistent

Robust

Easy to Understand

July 2023

Monorepo vs Polyrepo

The decision of whether to use a monorepo or a polyrepo structure for your source code can be a very emotional (maybe even religious!) battle. On its surface, it’s not much different than “tabs vs spaces”...