Earthly for Monorepos
Simpler, faster monorepo builds by only building what's needed, parallelization, caching, and a modern import system for better code reuse.
Why Are Monorepos Gaining Popularity?
- Having every project’s code in one place simplifies project management.
- Monorepos enable easy reuse of developer infrastructure – like build and release scripts – and more straightforward dependency management across projects.
- Monorepos foster better team collaboration, with all project assets in one place.
Challenges with Monorepo Builds
- Monorepo builds are frequently inefficient because small changes in individual projects trigger rebuilds of the whole repo.
- Large monorepos can lead to increased build and test times.
- They require more compute resources, potentially increasing costs.
- Managing builds in these complex environments can be challenging.
- Standard CI/CD systems often require workarounds or hacks to work effectively with monorepos.
Overcoming Monorepo Build Challenges
- Incremental Builds: Only rebuilding the parts of the repository that have changed.
- Parallel Processing: Breaking up builds into independent tasks and running those tasks in parallel, sometimes across multiple cores or machines.
- Caching: Storing and reusing build components, such as generated code and binaries, if their determinants haven’t changed.
- Monorepos often involve intricate and interconnected build structures. Changes in one part of the repo can impact several others.
- Earthly’s build graphs allow for managing these complex structures effectively.
- Reuses unchanged components within the build graph, avoiding unnecessary recompilation.
- In monorepos, tracking dependencies can be challenging due to the intertwined nature of projects.
- Earthly understands and tracks build dependencies and changes.
- Determines the minimal rebuild scope, saving time by not rebuilding unchanged components.
- Parallel processing for builds in a monorepo can significantly reduce build times across multiple projects
- Earthly infers parallelism in build processes based on clear interdependencies.
- Plan for future versions to run in compute clusters for increased speed and scalability.
Holistic Layer Caching
- Expands Docker's layer caching to all build artifacts, not just container images.
- Reuses build components like generated code and binaries if unchanged, speeding up the build process.
- Particularly beneficial in monorepos, where multiple projects share common components.
Instantly Available Cache
- A feature of Earthly Satellites.
- Provides cache sharing between CI runs without download/upload delays.
- Satellites offer high cache hit rates and facilitate cache reuse among team members outside of CI.
Modern Import System
- Earthly can be used to reference and build targets from other directories in a monorepo or even other repositories.
- Helps foster effective reuse of common build logic, ensure uniformity, and minimize redundancy across projects, especially in a monorepo.
Monorepo vs Polyrepo
The decision of whether to use a monorepo or a polyrepo structure for your source code can be a very emotional (maybe even religious!) battle. On its surface, it’s not much different than “tabs vs spaces”...
Building a Monorepo in Golang
A repository in Go traditionally contains a single Go Module, which lends naturally to a polyrepo setup – but what if you try to build multiple Golang projects in a single monorepo? It sounds simple enough...