We’ve been working to make Earthly easier for new users. I recently tackled the self-sign-up feature. We wanted to make signing up simple, to draw in more developers and grow our community.
The Streamlined Onboarding for Earthly gets us closer to our goal: making CI/CD simple so developers can focus on their work. But was it working? I needed to find out, but getting to the bottom of that involved mastering Segment and Snowflake, learning what DBT was, and 20 other things I’ve probably forgotten by now.
After we launched self-sign-up, we used what data we had to watch how new users used our platform. Did they like what Earthly offered? We tracked their path from the first click to sign up all the way to buy-in.
But we needed to dig deeper into our customer journey funnel. Track the journey from sign-up to paying customer. Our goal was to make onboarding smoother, and get people to having a satisfy experience with Earthly faster.
The problem is tracking and analyzing people to see where they get stuck in our on-boarding process. Earthly has a bit of a steep onramp at first. How can we make it smoother? How can we measure the drop-off? Traditional tools fell short, and custom solutions seemed too heavy and too costly. So, we turned to specialized tools like FunnelStory to really understand people’s onboarding journey’s.
Diving into funnel analysis, I hit some snags. It wasn’t just about new tools or dashboards. It was about understanding users and tackling tricky data.
I faced a big challenge: messy data. Sometimes, what I needed was just not there, or our numbers didn’t add up. It was a wake-up call for me. I had to take a second look at the assumptions behind the data and sometimes add new instrumentation. And getting the data into the right format was another challenge. SQL queries and analytics tools – each with its own quirks.
As I dug into how customers interact and whether they stick around, creating a funnel visualization was another challenge. Standard tools were too vague or too much custom work. Not scalable, not efficient. That’s when we decided to try Funnel Story.
It showed us, plain and simple, the path people took from signing up to being active users and maybe even paying us. It was a fairly easy setup, and once I had that in shape, it meshed well with our data.
I turned to Hex and other data tools to build out custom usage models beyond the funnel. With Hex, I crafted custom data models and made visualizations that showed us custom data specific to our customers’ usage.
We mixed Funnel Story’s funnel tracking with custom stuff made with Hex, and together, it was a powerful toolkit. It lets us see what users do right away and where they bail.
Using data helps you make smart choices, better your product, and keep users happy. It’s not just about collecting numbers. It’s about understanding them and using them to keep getting better and to innovate.
Diving into analytics, I hit some tough spots. After dropping my daughter off at pre-school, I often work out of a cafe in Brooklyn. If you saw me there, laptop open, confusion on my face, wrestling with SQL variants and Snowflake gotchas, trying to make sense of huge piles of data, I might have looked stressed. Trying to assess accuracy, getting feedback from others on the numbers, but doing much of this solo. I had to double-check my work, with no formal review or second pair of eyes. We are a start-up, and I needed to take the lead on this and power through any impediments.
If I were giving myself advice for making an effort like this again, here’s what I’d want to know.
In short, uUse Funnel Story for funnel analysis, then add tools like Hex for performing custom usage analysis to complete the picture. These tools will show you where to tweak your product and how to on-boarding. You’ll see exactly how accounts move through your product and where you can improve.
Using data helps you make smart choices that matter for your product and the people using it. Our data analysis taught us a lot. We tracked how accounts behaved from sign-up to using our platform’s features. We saw what they liked and didn’t, and where we were losing them. This helped us figure out what to build next and how to keep them coming back.
And that’s where we’re focusing now. And it’s paying off – we’re not just getting users, we’re getting paying customers.
Earthly Cloud: Consistent, Fast Builds, Any CI
Consistent, repeatable builds across all environments. Advanced caching for faster builds. Easy integration with any CI. 6,000 build minutes per month included.
ARM processor architecture is a family of instruction set architectures (ISAs) for central processing units (CPUs). An ISA helps applications talk to the hardware by specifying the processor’s capabilities and outlining how user instructions are executed based on those capabilities.
ARM processors adopt a simplified set of instructions using the reduced instruction set computer (RISC) model, which contributes to their efficiency, compactness, and lightness. This architectural choice has fueled the popularity of ARM-based devices, like the affordable and adaptable Raspberry Pi.
In addition to IoT and server devices, ARM architecture is gaining popularity in the personal computing industry due to its efficiency and cost-effectiveness. Apple’s M1 and M2 chips are examples of this shift towards ARM architecture.
The rapid growth of ARM-based devices, combined with the growing popularity of containerization, means that application developers need to publish container images for multiple platforms while ensuring compatibility across ARM and x86-x64 architectures as well as Linux and Windows environments. This multi-platform approach helps ensure that applications can run smoothly on a variety of devices, ranging from high-powered servers and desktops to low-end mobiles and ubiquitous IoT devices.
In this article, you’ll learn how to automate the creation and deployment of Docker images specifically designed for ARM architecture using GitHub Actions.
With the rising popularity of ARM devices, more and more applications need to be able to run on them. However, since ARM devices aren’t typically powerful enough to run heavy development jobs, the development world is mainly X86-based. This means developers usually create Docker images on non-ARM devices, and these images can’t run on ARM devices.
You can, in theory, use an emulator such as QEMU on ARM devices to run Docker images built on non-ARM devices, but that emulation is painfully slow and can reduce productivity. That’s why, it’s recommended that you build an ARM-based Docker image so that you can run it directly on ARM devices.
For old Docker versions, you needed to have an ARM device or use QEMU to build ARM-based images. However, with the advent of buildx
, it’s possible to easily build ARM-based images on non-ARM devices.
This tutorial uses a simple Python application that prints out basic information about the system it’s running on. All the source code for this tutorial is available in this GitHub repository.
Before you can continue, you’ll need:
buildx
, a Docker CLI plugin that extends the build capabilities of Docker. It enables Docker to build images for multiple platforms. You can also use it to build Docker images in parallel, which can significantly reduce build time. The latest Docker Engine requires you to install buildx
separately, as provided in the installation instructions.Once you’ve completed these prerequisites, you’re ready to create your demo application and set up GitHub Actions to automate the Docker image generation process.
In the first section of this tutorial, you’ll build and run a Docker image on an ARM device natively. For this reason, this section is executed on an ARM device.
First, create a new directory, add a file named main.py
within the directory, and copy and paste the following code into the file:
import platform
print("This program is running on " + platform.machine())
print("Platform system: " + platform.system())
print("Platform version: " + platform.version())
print("Platform node: " + platform.node())
print("Platform architecture: " + str(platform.architecture()))
This script uses the platform
module to access the system information and prints it to the console. It prints the machine type, operating system, version, node, and architecture. The following is an example output of the script running on a Raspberry Pi with an ARM processor:
This program is running on armv7l
Platform system: Linux
Platform version: #1559 SMP Wed Jun 1 13:24:16 BST 2022
Platform node: raspberrypi
Platform architecture: ('32bit', 'ELF')
You can also verify this output by running uname -m
. This command prints the architecture of the system it’s running on:
uname -m
$ armv7l
Now it’s time to write the Dockerfile. Create a new file named Dockerfile
in the same directory as the Python script, and copy and paste the following code into it:
# Use an official Python runtime as a parent image
FROM python:3.9-slim-buster
# Set the working directory in the container to /app
WORKDIR /app
# Add the Python script into the container at /app
ADD main.py /app
# Run the command to execute your Python script
CMD ["python", "./main.py"]
This Dockerfile uses the official Python image as the base image. Then, it sets the working directory to /app
and adds the Python script to the container. Finally, it executes the Python script using the CMD
instruction.
To build the Docker image from the Dockerfile on your local machine, run the following command in the same directory as the Dockerfile (replace <your-registry-username>
) with the username of your container registry account:
docker build -t <your-registry-username>/python-app .
This command builds the Docker image using the Dockerfile and tags it with the name python-app
. Once the build is complete, you can run the Docker image using the following command:
docker run <your-registry-username>/python-app .
This command creates a container from the Docker image and executes it to print the host platform information to the console. The output generated from the command should be similar to the previous one.
It’s not always possible to use an ARM machine to build ARM-based Docker images. If your development or CI machines are predominantly non-ARM, it makes sense to build the Docker image on non-ARM devices. However, as you’ll see, images built on a non-ARM device will not run on an ARM device by default.
To demonstrate, on your non-ARM device, either copy and paste the same code or clone this GitHub repo. Then, run the following command to build the Docker image and push it to the registry:
docker build --push -t <your-registry-username>/python-app .
Note: Make sure you are logged in to the registry by running the
docker login
command beforehand.
Go back to your ARM device and run the newly built image:
docker pull <your-registry-username>/python-app
docker run --rm <your-registry-username>/python-app
You’ll face an error message like this:
WARNING: The requested imahe's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
exec /usr/local/bin/python: exec format error
This simply means that since the image was created on a non-ARM device, it won’t run on an ARM device.
To fix this, you’ll need to use buildx
to build for the ARM platform:
docker buildx build --platform linux/arm/v7 --push -t \
<your-registry-username>/python-app .
Note: You might need to use something else instead of
linux/arm/v7
, depending on your ARM device. You can figure out what value to use by looking at the “host platform” value in the error message in the previous step.
Go back to your ARM device and run the image again:
docker pull <your-registry-username>/python-app
docker run --rm <your-registry-username>/python-app
This time, the image will execute without an error.
At this point, you’ve created a Docker image for your Python application and tested it on your system. Now it’s time to automate the build and image push process using GitHub Actions. In the following section, you’ll set up GitHub Actions to build and push the Docker image to a Docker registry every time you push updates to your code.
To set up GitHub Actions, you have to create a GitHub repository for your application and push the code to it. Once you’ve pushed your code to the repository, create a new file named main.yml
in the .github/workflows
directory. This file contains the workflow definition for your GitHub Action.
Copy and paste the following code into the file:
name: Build ARM Docker Image
on:
push:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: Log in to GitHub Container Registry
uses: docker/login-action@v1
with:
registry: ghcr.io
username: $
password: $
- name: Build and push
uses: docker/build-push-action@v2
with:
context: .
push: true
tags: ghcr.io/$:latest
platforms: linux/arm/v7
Here’s the breakdown of the workflow definition:
name
field defines the name of the workflow (Build ARM Docker Image
).on
field defines the event that triggers the workflow (the push
event on the main
branch).jobs
field defines the jobs that are executed as part of the workflow. In this case, there is only one job named build
.runs-on
field defines the operating system on which the job is executed (ubuntu-latest
).steps
field defines the steps that are executed as part of the job. In this case, there are four steps:
buildx
to build Docker images for the ARM platform.context
field to specify the directory containing the Dockerfile. Then, it uses the push
field to specify that the image should be pushed to the registry. It also uses the tags
field to specify the name of the image. Finally, it uses the platforms
field to specify the platforms for which the image should be built. In this case, it is linux/arm/v7
, which means that the image will be built for the ARM architecture.To trigger the workflow, you need to commit and push the workflow definition to the repository. Once you’ve committed the changes, the workflow is triggered automatically. You can view the status of the workflow by going to the Actions tab in your repository. The following screenshot shows the status of one of the workflow runs:
)
Once the workflow is complete, you can view the Docker image in the GitHub Container Registry by clicking the generated package displayed on the Code tab in your repository:
Click the package to view the details of the Docker image, including the tags, platforms, and the command you can use to pull the image:
)
You’ll notice that you’re using Docker’s platform emulation using buildx
to build the ARM-based image. This is because GitHub runners are X86-based. However, ARM-based runners are in private beta, and once they’re available to the public, you can build ARM-based Docker images natively. Meanwhile, you can use a self-hosted ARM runner if you want to natively build ARM images.
Now that you’ve created the Docker image for your Python application, you can run it on your ARM machine. Simply pull the image from the GitHub Container Registry and run it:
docker run ghcr.io/<user-name>/<repository-name>:latest
This command creates a container from the Docker image and executes it. The output should be similar to the one shown in the previous section.
In this article, you learned how to automate the creation and deployment of Docker images for ARM architecture using GitHub Actions. You learned how to set up a Python application, create a Dockerfile, and configure GitHub Actions to automate the build and image push process. This automation ensures that your users have access to the latest version of your application, regardless of their platform.
Earthly Cloud: Consistent, Fast Builds, Any CI
Consistent, repeatable builds across all environments. Advanced caching for faster builds. Easy integration with any CI. 6,000 build minutes per month included.
Based on my previous post, you’d correctly assume that I would want an expressive language aimed at experts. And I do. But there’s a big problem when things scale up in a flexible language. Too many coding styles, too many ways to program. You end up needing style-guides to nail down the right way to do things.
Which subset of C++ or Kotlin are you using? Are you using project.toml
or requirements.txt
? Your language now has gradual typing with Type Annotations. Do you want to adopt those or not? Are you going to use multi-threading, Tokio, or Async-std for concurrency?
The more expressive the language, the harder this is. This is where Go shines. It’s not just about gofmt
, but also its standard library and the consistent way of doing things. In Kotlin, you’re left wondering: exceptions or Result for errors? But with Go, you know the drill. Look for err
. Sure, it’s wordy, but it’s predictable.
Expressive languages are great, but they can be messy. You can have a language that’s rich and complex without a million ways to do the same thing. That’s what I want to show you. How do we keep the power but ditch the clutter? How do we avoid having 500 subdialects of the language? But before we dive into solutions, let’s talk Scala.
To highlight the problem, take Scala. A language I absolutely love, by the way. But it’s got this one big issue. There’s no idiomatic Scala. It’s way too flexible.
I can write a single file, calculator class and start with a Java style:
// Returns, braces and semi-colons
getResult(): Double = {
def return result;
}
multiply(number: Double): Calculator = {
def if (number == 0) {
println("Multiplication skipped: number is 0");
else {
} abs(number);
result = result *
}return this;
}
Same class, same file, I can switch to a pseudo-Python style:
// significant whitespace, no returns, no semi-colons
def add(number: Double): Calculator =
+= abs(number)
result
this
def subtract(number: Double): Calculator =
-= abs(number)
result this
And then, when I call the whole thing, I can use no braces and no dots style. The Ruby DSL style:
Calculator add -5 subtract -3 multiply -2 val calc = new
Hopefully, nobody’s stuck with code like this. You pick your dialect of Scala and you stick to it. But as code grows, it’s like Montreal. Every part of the city is different. On a long enough timescale, every quirk possible in your programming language will show up in your code.
Eventually, someone copies and pastes code in a different style. Maybe they like it better. Or in a new service, they do things their way. Or a junior mimics a library’s style from the docs. And style divergence starts1.
( Every Scala thread on hn has a comment from someone who inherited a Scala codebase that they are struggling to make sense of, in part because its in a foreign style. )
C++20 had a lot of good ideas, but lots of code predates that standard. And so drift occurs. You either don’t adopt the new way or you end up with a code base with more than one style. If you do the latter, you end up with the Montreal Problem. If you are doing work in the old-Montreal code section. It’s like a different dialect. You now need to know multiple dialects of the language and when and where to apply each one.
So, how do you evolve a language without splitting it apart? This gets trickier with a whole community involved. Big open-source projects often have their own style. The natural tendency towards divergence means it’s hard just to jump into existing codebases we aren’t familiar with because they’ve got their own style. And with that, the community fractures.
Style Guides, especially if they can be machine-enforced, can help a lot at the level of a large code base. I think it’s great when languages can experiment with things. Maybe we don’t know if it totally makes sense to use types in Python everywhere yet or how much you should use generics in Go or whatever.
But for a specific project, we can set rules. Say, ‘this Python needs types’ or ‘Use generics in Go when you can.’ We set standards for testing libraries and build tools. And we attempt to enforce these rules with tooling where we can. But I think we can do even better at a language community level.
Codebase-specific style guides aren’t enough.
When Scala 2.0 launched in 2006, internal DSLs were all the rage, and with Ruby on Rails leading the charge, Scala embraced a more fluid writing style. But times change and that style is no longer idiomatic Scala.
But how would you know that if you’re not in the right circles? Big ORM frameworks still use that style in their guides. The tricks for writing modern idiomatic Scala are trapped in the minds of the community leaders. That’s not great.
We need a Style Czar. Someone in the language community who can say that this is idiomatic Scala 2.1
def ABSOrSeven(maybeNumber: Option[Int]): Int = {
if (maybeNumber.isDefined) Math.abs(maybeNumber.get)
else 7
}
But in Scala 3.1, this is preferred:
def ABSOrSeven(maybeNumber: Option[Int]): Int = {
map(Math.abs).getOrElse(7)
maybeNumber. }
What I’m suggesting is that every release of a language should come with a style-guide. No one has to follow it; companies and projects may diverge, and the standard might be highly contested; but it should exist and be written down somewhere.
Python folks love their Pythonic code, the zen of Python, and PEP 8. And that is what I’m thinking about, but evolving over time, and with a larger scope. I think the language creators need to not just create the language but be shepherds for emerging standards in how programming is done in the language. They need to talk to us, tell us: do this, not that. And it should be a conversation, with debates, with tools to help us follow the rules.
This standard will keep changing, right? It will evolve as the language does. Maybe type annotations, when they first roll out in Python, are considered experimental. But once everyone is comfortable with them and thinks they are a good idea, Python should take a stance and say type annotations are required in pythonic code. Or say that they are not. But please have an opinion. As the language grows and the community starts to diverge in various ways, the scope of the style document should expand as well.
Here’s an example: Python has to pick a lane with package managers and virtual environments. I think poetry and project.toml
should be the way forward, and others have a requirements.txt-for-life
tattoos. But any solution would be better than what we have now.
We need someone in charge to step up. “Okay, everyone, we’re standardizing on Hatch for Python packaging. If you’ve got issues with Hatch, speak up. We’ll look into them. But just so you know, we’re aiming to make Hatch the go-to for Python 3.16.”
This goes for testing frameworks, standard libraries, and even how we handle concurrency. Language communities love to experiment and explore. But after the exploring, we need to come together. And that’s where the language creators step in. They’re the ones who can really make it happen.
Ok, so how does this tie to being expressive? Well, if your language is on the ‘expert readability’ train, you probably have a bunch of features and aren’t afraid to add new ones. The problem is you slowly end up in a world where each codebase is written in its own subset of the language. So, deprecate things. Sure, keep old stuff for compatibility, but let’s nudge everyone towards a common standard.
The fact you are evolving the language implies you have an opinion about what great code looks like. Tell us! Write it down. Talk it out with the community. Using macros is not idiomatic in C++20. Using if’s to check the types of a returned object is not idiomatic Kotlin 1.17. Don’t use explicit returns in Scala. And so on.
This way, there’s always a target of what great code looks like. Even if that target moves, every sane code base is just at a specific point along that journey to the latest and grandest code style.
In other words, you can end up in a world where all idiomatic Java 20 code is as uniform as Go, but to get there, you have to take a stance on when it’s appropriate to use the streams API and when not. I mean, maybe you never quite get there: one person’s clear stream processing one-liner is another man’s spaghetti code, but I really think we could do better at converging on what we want the language to look like.
This brings up a bunch of questions. How much of this can be tool-enforced? When should a popular library be canonized? How much style guidance is too much and stifles innovation? I don’t really know. Just start with something, like a version of Python PEP 8, and evolve and expand it over time.
If something is a community norm, write it down. If the community is fighting over whether to eat toast butter side up or butter side down, flip a coin, make a call, and move on. The community will be better for it.
Show us the way, Style Czar!
This is just an easy to show example. If I were the style Czar, then yes Don't use explicit returns or semicolons
would be a rule. But so would Don't pattern match if you can use fold or map
and Don't use fold if you can use getOrElse
and Don't do manual recursion
and Don't use actors unless you have a really good reason
and Don't write custom operators. Just don't. Really.
and so on and on. Lot’s of features are only valuable in specific circumstances.↩︎
As a software developer, you’re probably familiar with the slow nature of Docker builds. These local builds can consume anywhere from an hour to several hours of your day. The slow pace of these builds not only delays project schedules but also hinders your ability to iterate quickly, forcing you to find a faster, more streamlined build process.
Enter Docker Build Cloud, a solution designed to transform how you build Docker images. In this article, you’ll learn all about Docker Build Cloud and how it works. By the end of the article, you’ll be well-equipped to save valuable time in your development cycle and enhance your overall productivity and efficiency.
Docker Build Cloud is a groundbreaking service aimed at speeding up Docker builds by up to 39 times compared to conventional local builds.
Build Cloud works similarly to local BuildKit instances, but with a key distinction in how it executes: when you initiate a build with Build Cloud, the build data is securely transmitted to a remote builder using end-to-end encryption. After the remote builder finishes the build tasks, it sends the output back to your chosen destination, be it your local Docker image store or an online image registry.
This approach focuses on utilizing on-demand cloud resources. When a build job is received, Docker Build Cloud dynamically allocates cloud-based BuildKit instances to handle the build tasks.
A key feature of Docker Build Cloud’s architecture is team-wide caching. This innovative caching solution allows all team members to share cached build layers across projects. That means that when one team member builds an image, the resulting layers are cached on the cloud, and subsequent builds by any team member can reuse these cached layers if the build context hasn’t changed. This dramatically reduces build times. Additionally, since Build Cloud natively supports multiplatform builds, this advantage extends to any image type, regardless of the underlying platform.
Another advantage of Docker Build Cloud is that it supports building images for different platforms, such as AMD64 and ARM64, and you don’t need multiple native builders or slow emulators. Moreover, builds run on managed infrastructure, ensuring that each build operates in isolation on a dedicated Amazon EC2 instance with a dedicated EBS volume for the build cache. This setup guarantees that there are no shared processes or data between cloud builders, maintaining strict end-to-end encryption and security.
In essence, Docker Build Cloud is not just a tool but a transformational shift in how Docker image builds are approached. It elevates development productivity by streamlining the build process and leveraging cloud resources to vastly reduce build times. Docker Build Cloud also fosters collaboration, significantly reducing duplicate efforts and ensuring that everyone is working in the latest build environment so that builds are both fast and consistent.
Now that you understand the benefits of Docker Build Cloud, it’s time to dive into its implementation. This section shows you how to set up Docker Build Cloud and build images. It also introduces some tips for optimizing your Docker Build Cloud setup so you get the most out of its capabilities.
To start using Docker Build Cloud, you need to link a payment method to your Docker account, even if you plan to only use the free tier.
If you visit https://build.docker.com/, you’ll see a screen asking which profile you want to use for Build Cloud:
After selecting a profile, you’ll be shown the available plans. For this guide, the free Starter plan is sufficient, but you should choose the plan that best fits your needs.
After choosing a plan, you’ll see a pop-up notification telling you that you need to add a valid credit card for account verification:
Once you add your card, you’ll be directed to the Docker Build Cloud dashboard:
From the main dashboard, you can check the remaining build minutes in your plan, upgrade your Docker Build Cloud plan, and create cloud builders. To create a builder, simply click the Create a Cloud Builder button. A pop-up window will appear, asking you to give the builder a name:
Name your builder (here, it’s named mastodon
) and click Create.
After creation, you’ll be directed to the Cloud Builders screen, where you can view all your available builders:
Select the builder you just created, and instructions for installing a cloud build driver on your local machine and integrating it with CI/CD processes will appear:
Begin by executing the first two steps. Follow the on-screen commands, which are already populated with your Docker organization and builder name for ease of use. After completing these steps, you can use Docker Build Cloud from your CLI.
Alternatively, because Docker Desktop includes Build Cloud as a preinstalled feature, when you log in to Docker Desktop with your user or organization credentials, you can directly access Build Cloud from the Builders tab.
Then, since you’ve created a cloud builder, you can find it under the Available builders section. You’ll see instructions for connecting the builder via the CLI, but you can skip this step since you’ve already completed it. Simply click the Connect to builder button to start using Build Cloud through Docker Desktop:
Once connected, you can use the menu to use the builder, stop it, or disconnect from it:
The final step in the setup process is to integrate Build Cloud with your existing CI/CD pipelines and tools. Choosing your builder on https://build.docker.com/ only provides detailed instructions for GitHub Actions and CircleCI, but additional integration guidance is available for GitLab, Buildkite, and Jenkins.
Now that you have everything you need to start using Build Cloud from the CLI or from Docker Desktop, it’s time to start leveraging its power by building some images.
This section compares Build Cloud with the traditional docker build
command that you’re probably very familiar with. This demonstration uses a basic Flask application:
from flask import Flask
= Flask(__name__)
app
@app.route('/')
def hello():
return 'Hello, World!'
This app is written in Python and creates a basic web application that responds with “Hello, World!” when accessed. To create the Docker image for this app, you can use the following Dockerfile:
FROM python:3.13.0a3-bookworm
WORKDIR /app
RUN pip install flask==2.3
COPY . /app
ENV FLASK_APP=app.py
CMD ["flask", "run", "--host=0.0.0.0", "--port=5000"]
Selecting the base image python:3.13.0a3-bookworm
over a lighter alternative, such as one based on Alpine, is a deliberate decision to increase the workload for the Docker build process. Once you save both files, you can start building Docker images.
Building a local image establishes a baseline that allows you to compare your traditional workflow with Docker Build Cloud. For that reason, run the following command to build the image without using Build Cloud:
docker buildx build -t <YOUR_DOCKER_USERNAME>/sample-flask-app:local .
This command builds a Docker image from the Dockerfile and tags it as local
to indicate that it’s a local version of the sample-flask-app
image under the <YOUR_DOCKER_USERNAME>
repository.
Now, switch to Docker Build Cloud to build the same image using the following code:
docker buildx build --builder cloud-<YOUR_DOCKER_USERNAME>-<YOUR_BUILDER_NAME> --tag <YOUR_DOCKER_USERNAME>/sample-flask:cloud .
This command initiates the build and is essentially the same as the previous one. The main difference is where and how the build is processed, which is determined with the --builder
flag.
The only reason you don’t specify the --builder
flag when building the image locally is because you’re using the default builder. If you want to set Docker Build Cloud as the default builder (and save you from typing --builder
every time), you can use the following command:
docker buildx use cloud-<ORG>-<BUILDER_NAME> --global
Keep in mind that if you do so, each time you build an image, your build will be processed on the cloud. This might impact your billing.
This screenshot shows your results, in which the container image took 5.3 seconds to build using the default (local) BuildKit instance and 1.6 seconds using Docker Build Cloud. In other words, the image was built 3.3 times faster.
Before drawing conclusions, remember that your results may vary depending on your hardware and/or internet connection. Regardless, for such a simple image, the results are promising, especially considering that no optimization has been implemented yet.
As the complexity of Docker images increases, they require more computing resources to build. That’s why adopting best practices and effective strategies to enhance Docker build performance is so important.
Consider using .dockerignore
files to exclude unnecessary files from your build context, choosing slim base images to reduce final image size, leveraging multistage builds to minimize redundancy and speed up builds, and fetching files directly in your build from remote locations rather than including them in your build context.
If the results of Build Cloud do not meet your expectations, consider upgrading your plan, as doing so gives you access to instances with more CPU power, RAM, cache storage, and parallel builds.
That said, none of these optimizations address the main limitation of Docker Build Cloud, which is that it cannot be leveraged to build artifacts beyond Dockerfiles.
Earthly is a CI/CD framework that expands Docker’s capabilities and makes remote builds super simple. It offers Earthly satellites, which provide an innovative approach to build optimization.
Earthly satellites are remote runner instances that facilitate remote caching, allow for a simplified syntax in build scripts, enable parallel execution of build stages, and manage build artifacts more effectively than Docker alone. A noteworthy application of this approach is seen in the case of ExpressVPN, which dramatically cut CI build times by utilizing Earthly’s caching features and remote runners. This example highlights the potential for significant efficiency gains in CI processes through thoughtful optimization and the adoption of advanced building tools like Earthly.
In this article, you learned how Docker Build Cloud—with its cloud-based infrastructure, native multiplatform support, and team-wide caching—can help you achieve significantly lower build times. However, you also learned that Docker Build Cloud cannot be used to build artifacts beyond Dockerfiles. That’s where Earthly satellites can help.
Earthly satellites can help you unlock the full potential of your CI/CD pipeline with remote runner instances designed for efficiency and scalability. Start optimizing your build processes today and experience groundbreaking speed and reliability in your deployments.
Earthly Cloud: Consistent, Fast Builds, Any CI
Consistent, repeatable builds across all environments. Advanced caching for faster builds. Easy integration with any CI. 6,000 build minutes per month included.
Containers are the Gutenberg press of the IT world: an innovation that profoundly improves the way applications get deployed and managed during runtime.
The primary purpose of a container is to provide applications with a well-defined, replicable, and isolated environment. If an application expects to run on an Alpine Linux system, a container can provide the app with the appropriate Alpine Linux libraries, commands, and system services. This container can run on a Debian or Arch Linux host, and the application inside can still use an Alpine Linux system. Moreover, the application sees restricted versions of the process ID space, the file system, the network, and other OS resources.
These restrictions help isolate containerized apps from each other, which is an important aspect not only from a security perspective but also when it comes to avoiding resource conflicts. (Imagine two processes trying to reserve the same network port.)
To isolate apps, Docker uses Linux namespaces. In short, Linux namespaces divide global system resources into distinct compartments. For instance, a process that is created with a dedicated network namespace can access any network port without coming into conflict with processes in other network namespaces.
In this article, you’ll learn more about Linux namespaces and how Docker uses them to achieve process isolation.
Processes need access to resources and contexts provided by the operating system, including the file system, network ports, user context, and shared memory. These resources and contexts are globally available to all processes, which causes two problems:
Linux namespaces solve both problems. A namespace restricts access to resources and virtualizes these resources. For example, when a process runs inside a network namespace, it can always access port 80, as if that port were exclusively reserved for that process. However, this port is only virtual. The Linux kernel maps this virtual port to a different port at the OS level. In the same manner, a process can get a restricted view of the file system, running processes, and more.
The idea of namespaces is not new. Before Linux, the operating system Plan 9 from Bell Labs had a concept of namespaces that controlled access to the file system. Since all system resources in Plan 9 are mapped into the file system, Plan 9 namespaces effectively control access to any resource, not just files and folders.
Another early approach to namespaces is the Jails concept from FreeBSD and other BSD derivatives. Jails virtualize access to the file system, the set of users, and the network, effectively partitioning a BSD system into several independent virtual systems.
Linux utilizes various types of namespaces for controlling access to specific resource types, including the following:
Closely related to namespaces is the chroot
command (and system call) that moves the root directory for the current process to a chosen directory within the file system.
Running a process in a separate namespace is easy to achieve. The unshare
command runs a process and “unshares,” or separates, one or more namespaces from the parent.
For example, the following command runs a shell in a separate UTS namespace:
sudo unshare --uts bash
If you want to test this, the following Bash commands read the host name and spawn a shell with a separate UTS namespace, rename the host inside the shell, and read the host name inside and outside the shell:
alice@earthly $ hostname
earthly
alice@earthly $ sudo unshare --uts bash
hostname
$ earthly
hostname martian
$ hostname
$ martian
exit
$ alice@earthly $ hostname
earthly
While the host name inside the shell is successfully changed, the parent shell does not see the change.
In the same way, you can isolate a process from the host’s PID space, network devices, file mounts, users and groups, or interprocess communication.
To manipulate namespaces programmatically, an application can use three syscalls: clone
, unshare
, and setns
.
When it comes to isolating containers from each other, Docker doesn’t reinvent the wheel. Because it’s a Linux-native technology, Docker uses Linux namespaces to achieve isolation and resource access control.
Using the PID namespace, Docker hides the global process list from processes inside a container. The first process spawned inside a container gets assigned a PID of 1. If it spawns further processes, they get assigned subsequent PIDs.
Containerized applications do not need to access all the mounted file systems on the host to do the tasks they were built for. Separate mnt
namespaces prevent containers from accessing mounts that belong to another container or the host.
Together with the chroot
syscall, the mnt
namespace can change the visible file system for a process. The chroot
syscall changes the root directory that is visible to a process to a given directory of the host system. The chroot
ed process can only see that directory and its subdirectories.
If two processes communicate with each other through Linux interprocess communication, they have to share the same ipc
namespace. In general, containerized applications do not talk to applications in other containers (or, if they do, they typically use a REST API or a message queue system). That means that for isolation purposes, each container receives its own ipc
namespace.
Sometimes it’s unavoidable that applications inside containers must run as root
. To avoid security breaches, Docker can remap the root user inside a container to an unprivileged user ID on the host, effectively preventing privilege escalation attacks.
With a network namespace, a container gets a unique, virtual network stack. This includes network interfaces, IP addresses, ports, and routing tables. When each container has its own IP address, virtual networks between containers can be established that isolate network traffic between containers from traffic to and from the host or the host’s networks.
Linux namespaces address three requirements around containerization: security, resource management, and container management.
By limiting access to system resources and contexts, namespaces help improve container security. Intruders that break into a running container are unable to see other containers or host processes. The root user inside the container maps to a user without privileges on the host. That means intruding into a system through a Docker container is much harder than intruding into a system where all processes run inside the host’s global namespaces.
Namespaces also provide the means for managing the resources for a process. If a process does not initiate network connections outside the host, the container can be set up to allow only network communication with the host. Additionally, a reduced list of mounted file systems ensures that a process does not interfere with other processes’ file resources.
Finally, managing containers is simpler if they do not interfere with each other. Imagine that you start two containers that both run a web server listening on port 80. With separate network namespaces in place, each container’s port can be remapped to a different available port on the host system.
How does Docker use Linux namespaces in practice? Let’s take a look at three typical requirements for containers and how Docker meets these requirements using namespaces.
Containers provide isolated environments for processes, but they still need to access global resources on the host. To avoid conflicts, Docker differentiates containers by remapping the IDs of resources, such as network ports, process IDs (PIDs), user IDs (UIDs), or group IDs (GIDs).
For instance, a containerized process can have different PIDs inside the container and on the host. If it is the first process spawned inside a container, it sees itself as the process with ID 1. Unless this process spawns subprocesses, it does not see any other processes on the system.
To examine this behavior, you can run a container and inspect the output of ps
inside and outside the container.
As a prerequisite, you need Docker installed on your system. The host used in the following examples is a Linux system. If you use a different OS, Docker runs inside a Linux VM. In this case,
ssh
into the VM to follow the steps of the examples. It’s also assumed that Docker is set up for use withoutroot
.
The following command creates a new container based on an Alpine Linux image and runs the sh
shell interactively:
docker run --rm -it alpine sh -C
Please note: Alpine does not come with
bash
. Alpine Linux is used here because the image is small compared to other Linux distros.
The -C
flag only serves to make finding the sh
command in the host’s ps
output easier.
You should now see Alpine’s sh
prompt.
Type ps
to see the PID of the shell:
/ # ps
PID USER TIME COMMAND
1 root 0:00 sh -C
7 root 0:00 ps
/ #
You can see that sh
has PID 1. Child processes get assigned subsequent PIDs. (Run ps
again to see its PID increase.)
Now open a new shell on the host and find the sh
process using ps
and grep
:
ps ax | grep "sh -C"
$ 3150 pts/1 Sl+ 0:00 docker run --rm -it alpine sh -C
3186 pts/0 Ss+ 0:00 sh -C
3208 pts/2 S+ 0:00 grep --color=auto sh -C
On the host, the PID of the same sh
process is entirely different—3186 in this example.
If you start a second Docker container in the same way, the shell inside the container will also have a PID of 1. But as each container has its own PID namespace, the PIDs inside the two containers will not conflict with each other or with the PIDs on the host.
Similarly, Docker can remap other resources, such as network ports or user IDs, to further isolate containers from each other and from the host.
Container security is a complex topic, and many layers of security technology are involved in making containers secure, but everything starts at the Linux namespaces layer.
Linux namespaces are at the core of container security. All other security layers and techniques are stacked on top of namespaces.
In the Linux kernel, these layers include:
The Docker platform provides additional security techniques, such as Docker trusted content and Docker secrets, but these don’t apply to the level of single-container instances.
Although namespaces are the lowest layer of these security measures, they are indispensable for container security in two ways: they help prevent security breaches and resource hijacking, and they help achieve data privacy in multitenant applications.
Imagine criminals gaining access to a containerized process. In this scenario, thanks to a separate mnt
namespace and chroot
, they can only see a small part of the host’s file system. A user namespace and UID/GID remapping cause the root user inside the container to be an unprivileged user outside the container. If the intruders manage to break out of the container, they’ll have no root privileges outside.
Due to a separate PID namespace, the intruders can only see the processes that run inside the container. They have no way of determining which processes run on the host or in other containers. If the intruders try to scan or block all network ports, a separate network namespace confines their malicious activity to container-local ports, keeping the host network out of reach.
Because namespaces compartmentalize essential resources, they’re ideally suited for multitenant scenarios where isolating data is crucial. The same concepts and ideas for preventing security breaches and resource hijacking apply here, too. By isolating processes, file systems, mount points, users, and networks of different tenants, users can confidently run their applications without worrying about data leaking into other tenants’ containers.
After talking so much about achieving isolation, it’s time to try running two containers in the same namespace.
In this scenario, let’s assume that two containers can run in the same PID namespace so that they can see each other’s processes. Let’s also assume a given process has the same PID in both containers.
You’ll try this in two different ways: through the Docker CLI and with Kubernetes pods.
Docker’s run
command has a --pid
flag that assigns a container to the PID namespace of another container.
To test this, open two shells. In one shell, type the following to start a container named waldorf
:
docker run --rm -it --name waldorf alpine sh $
In the other shell, start a second container named statler
with the --pid
flag:
docker run --rm -it --name statler --pid=container:waldorf alpine sh $
The --pid=container:waldorf
flag tells Docker to use waldorf
’s PID namespace for the statler
container as well.
Now run ps
in both shells. You’ll see two sh
processes in each output instead of one, and the PIDs of the shell processes are identical in both containers.
Shell one looks like this:
/ # ps
PID USER TIME COMMAND
1 root 0:00 sh
7 root 0:00 sh
13 root 0:00 ps
And shell two looks like this:
/ # ps
PID USER TIME COMMAND
1 root 0:00 sh
7 root 0:00 sh
14 root 0:00 ps
Be aware that the two containers can influence each other. For instance, they could send signals to the other container’s processes, including a kill signal. That means you need to use shared namespaces with caution and only if there is no other more secure way of making two containers work together.
Containers that run in the same Kubernetes pod can share PIDs as well. It takes nothing more than a single line in the pod configuration.
To run the following steps locally, you need a running Kubernetes node, such as the standalone Kubernetes setup provided by Docker Desktop.
As an example, the configuration file below starts two containers running a sleep 1000
command (you’re not using interactive shells here):
apiVersion: v1
kind: Pod
metadata:
name: shared-pid-pod
spec:
shareProcessNamespace: true
containers:
- name: waldorf
image: alpine
command: ["sleep"]
args: ["1000"]
- name: statler
image: alpine
command: ["sleep"]
args: ["1000"]
Note the following line:
shareProcessNamespace: true
This line shares the PID namespace between the containers running in this pod.
Start the pod by calling the following:
kubectl apply -f shared-pid-pod.yaml $
Then, check the status of the pod until the status is Running
:
kubectl get pods
$ NAME READY STATUS RESTARTS AGE
shared-pid-pod 2/2 Running 0 9s
Now start an interactive shell in one of the containers and run ps
to see the running processes:
kubectl exec -it shared-pid-pod -c waldorf -- sh
$ / # ps
PID USER TIME COMMAND
1 65535 0:00 /pause
7 root 0:00 sleep 1000
13 root 0:00 sleep 1000
19 root 0:00 sh
25 root 0:00 ps
/ #
There are two sleep 1000
processes visible, one from each container, and they both share the same PID namespace.
Sharing namespaces between processes lowers the level of isolation between the involved containers and should therefore be used sparingly and only if absolutely necessary. That being said, which of the above approaches—Docker CLI or Kubernetes pods—would be more suitable for production use?
It might be tempting to say that it doesn’t really matter because the benefits and risks of sharing namespaces stay the same. However, there are a few arguments that lean towards using Kubernetes pods rather than the Docker CLI for production use:
shareProcessNamespace: true
in the pod spec. This is less error-prone and much easier to maintain than having to manually specify the right container names and IDs when using docker run
.In general, starting containers through CLI commands is more of an ad hoc approach that does not scale well. Productive environments should be equipped with a software orchestration solution like Kubernetes.
Linux namespaces are at the core of container isolation in Docker. Namespaces compartmentalize global kernel resources, such as network interfaces, processes, file mounts, and users and groups. With namespaces and chroot
, a process inside a container may find itself running as the only process on a Linux system, equipped with root
privileges, while in fact it’s only isolated from other processes through separate namespaces and a virtual file system root.
By leveraging namespaces, Docker provides a robust platform for deploying diverse applications on a shared system with a high degree of control and separation.
Earthly Cloud: Consistent, Fast Builds, Any CI
Consistent, repeatable builds across all environments. Advanced caching for faster builds. Easy integration with any CI. 6,000 build minutes per month included.
Service Organization Control 2 (SOC 2) is important for assessing and verifying the controls around the security, availability, processing integrity, confidentiality, and privacy of systems that process user data. This is particularly significant for SaaS providers like Earthly.
There are two types of SOC 2 compliance:
Our SOC 2 Type 2 compliance doesn’t impact the functionality of Earthly Cloud (or Earthly Satellites). Both will continue to work exactly as they always have and as our users expect.
From a non-functional standpoint, SOC 2 Type 2 compliance helps assure our users that they can rely on Earthly Cloud (and Earthly Satellites) for continuous business operations and that downtime and potential disruptions are minimized. It also helps assure users that risks associated with data breaches, unauthorized access, and data loss are mitigated. Also, many industries and companies have stringent regulatory requirements that require SOC 2 Type 2 compliance from SaaS vendors. For those users, it means that Earthly meets their security needs and is a valid option for them to consider and choose.
To request a copy of our SOC 2 Type 2 report, contact security@earthly.dev .
Earthly Cloud: Consistent, Fast Builds, Any CI
Consistent, repeatable builds across all environments. Advanced caching for faster builds. Easy integration with any CI. 6,000 build minutes per month included.
Developer platforms centralize the internal tools and processes that developers use to build and deliver software. These platforms improve DevOps outcomes by providing automated mechanisms for developers to achieve their tasks.
More teams and organizations are launching their own internal developer platforms (IDPs) to reduce friction in their software delivery. Establishing an IDP takes time but offers substantial benefits, including improved productivity, easier collaboration, and less risky development. This article will explore these advantages and how they apply when building or buying an IDP solution.
DevOps enhances software delivery by tightening feedback loops, encouraging cross-discipline communication, and automating key processes around testing, quality, and deployment. However, DevOps doesn’t solve every challenge, nor does it tell you how to implement its concepts. Teams often struggle to understand where to start automating their work, or they run into difficulties when apps, team members, environments, and tools change because it’s hard to enforce which technologies devs are using.
Developer platforms are a holistic solution for achieving DevOps ideals. Consolidating your toolchain around a shared internal platform makes it easier to integrate different processes and standardize them across developers. An IDP gives devs automated access to the resources they need, such as the ability to create a new test environment on demand or request access to inspect what’s running in production.
You can do DevOps without an IDP, but it’ll probably be much less efficient. IDPs remove the responsibility for managing infrastructure processes from individual devs, letting them concentrate on delivering new software. Engineers don’t need to worry about how complex processes like deployments happen as they’re implemented within the IDP. The platform’s capabilities are usually maintained by a dedicated platform engineering team that exists to support DevOps requirements.
Developer platforms provide many benefits that collectively enable DevOps teams to quickly deliver impactful work while experiencing fewer problems. This section covers some of the ways in which adopting an IDP will improve your team’s engineering output.
Developer platforms can have a transformational effect on software development velocity. They shorten the software development lifecycle (SDLC), tighten feedback loops, and remove work from developers by automating previously time-consuming processes that depend on multiple tools.
IDPs achieve this by providing functions that support developer autonomy and efficiency. Instead of having to wait for separate teams to apply actions, developers are empowered to achieve their aims themselves using self-service options available within the platform.
For example, creating a new staging environment is often a complex procedure with many steps involved. Traditionally, developers would have contacted an infrastructure team to request that new resources be provisioned, then asked ops for help in deploying an environment, and finally consulted a senior developer to learn how to seed the deployment with some test data. This sequence is long-winded, dependent on multiple stakeholders, and a potential source of inconsistencies between environments.
Making the process available in an IDP solves these challenges. Developers would be able to use the platform to automatically launch a new environment on demand, such as by clicking a button, running a CLI command, or even using an extension added to an IDE or chat app. Moreover, as everyone deploys using the same automated action, the IDP standardizes the workflow and prevents configuration discrepancies from occurring. This helps you scale as more apps and developers are added to your organization.
The throughput increase afforded by an IDP was observed by Mercado Libre after it implemented its own FURY platform. Introducing FURY unlocked exponential growth in the developer count, while the number of microservices seamlessly grew to over 24,000. The organization found that the “decrease in cognitive load and the gain in efficiency … have undoubtedly made every investment in a platform worthwhile.”
Accelerating the SDLC results in developer productivity improvements too. Increasing the use of automation removes friction from processes, helping developers stay focused on meaningful work instead of being distracted by infrastructure management tasks.
In turn, this contributes to improved developer satisfaction. Devs are happiest when they have the tools they need to do their work without having to wait for others. A good platform supports developers so they can be more efficient, making them feel valued within the organization.
IDP automation and self-service capabilities are key to this effect as they promote autonomy. Devs can trigger tasks when they need them, even if they don’t understand exactly how they work or which tools are being used. This also helps to democratize development by allowing each engineer to safely apply changes to any area of a project, including those that might fall outside their existing skills.
The actual productivity increase created by an IDP will naturally vary by organization. Some of the effects are quantifiable by looking at metrics such as the number of issues closed, pull requests approved, and deployments created; a successful IDP implementation should deliver increases to these values as developer time is freed up.
Other aspects are purely qualitative enhancements to your developer experience (DX): developers should feel less pressured, more empowered, and better equipped to obtain visibility into operations across the SDLC. These benefits don’t always show up directly in your metrics but will have a positive impact on morale, motivation, and team retention, which further helps sustain development velocity.
One of the best examples of the productivity benefits of IDPs comes from Spotify, the original creator of the popular Backstage developer portal system. After creating and adopting Backstage internally across 280 teams and more than 2,000 backend services, Spotify observed a 55 percent decrease in new developer onboarding time, highlighting how platforms support developers to deliver more value faster.
Beyond productivity improvements at the individual level, developer platforms also help foster better collaboration within and between DevOps teams. Developer platforms make it easy to align the efforts of different teams and working groups, ensuring standard results even where complex multidiscipline processes are used.
Additionally, centralizing tools, processes, and documentation into a unified platform means that everyone’s working with the same resources. All developers can access the assets they require and understand how neighboring teams are tackling problems, thus preventing information from becoming siloed.
IDPs also make it easy to integrate changes into workflows and then roll them out consistently across the whole organization. This reduces the number of operating procedures, security policies, and compliance rules that need to be directly communicated to each team, limiting the places where knowledge can fall between the cracks.
In some cases, IDPs don’t even need to contain communication-specific functionality, as centralized docs and specs can be sufficient to achieve process improvements. During the development of its internal platform, Zalando spent time writing standards for how different teams should work together using common practices, including authoring API guidelines and identity management expectations. Once a behavior is defined as part of your platform, everyone has a common spec to work against. You can then implement automated tooling later on to detect and prevent spec noncompliance.
Of course, IDPs can’t do everything; indeed, this surfaces one of their potential weaknesses. For an IDP to be successful, all teams need to use it to achieve their aims, even where they may have previously reached for or built their own solutions. Obtaining acceptance from different teams is therefore a priority for any IDP implementation. This prevents fragmentation from occurring if some groups start creating their own tools that exist separately from the platform.
Delivering software using an IDP reduces the costs involved in the SDLC. Instead of maintaining multiple environments and resources for developers to use, you can focus on supporting a single platform with optimized utilization and provisioning. The removal of manual process steps frees up developers to stay focused on feature development, reducing overall sprint durations and minimizing the cost to your organization.
Additionally, improving developer productivity and increasing throughput allow extra value to be delivered to customers, making you more competitive and reducing time to market. Cutting the hours spent manually connecting tools or waiting for processes to complete ensures all developers are occupied with meaningful work.
Standardizing development activities around automated platform-centric tasks also reduces the risk in the SDLC. Replacing manual workflows with automated ones provides stability and helps eliminate many common errors, such as inadvertently skipping a step or supplying an incorrect input to a command.
Moreover, IDPs make it easier to govern the entire SDLC using standard policies and frameworks. If you’re subject to specific security, compliance, or regulatory requirements, you can implement guardrails in your platform that ensure continual enforcement across all teams and projects. This defends against accidental compliance breaches when developers use unapproved tools or accidentally push insecure code.
Developer platforms also make you more resilient to other types of risks, such as the productivity threats posed by staff absences. Centralizing processes into self-service platform actions means work can continue even when senior developers or adjacent teams are unavailable, making it less likely that you’ll miss critical deadlines.
Teams that have adopted internal platforms can deploy up to four times faster than those without, according to analysis by Humanitec. Additionally, their change failure rate falls simultaneously—down to 4 percent from 15 percent for teams without a platform. When failures do occur, the mean time to recovery is just 1.3 hours with an IDP instead of six hours without, making it much less likely that SLAs will be breached. Ultimately, IDPs protect you from costly incidents and make it easier to deploy more frequently, giving you a competitive edge.
There’s no one-size-fits-all developer platform solution. Some teams assemble their own platforms from scratch, which incurs a high initial cost but enables a high degree of customization. Others favor prebuilt commercial solutions that enable immediate adoption, but these may be less adaptable to future change. Between these two approaches, open-source platforms such as Backstage provide a robust foundation for your own tools, giving you some of the benefits of both DIY and off-the-shelf approaches.
Here are some aspects to consider when choosing whether to build or buy an IDP.
The following are a few advantages of building your own IDP:
The following are a few disadvantages of building your own developer platform:
If you’re interested in buying a prebuilt IDP, the following are a few of the advantages:
While the benefits of buying a prebuilt IDP are tempting, you also need to consider the following disadvantages:
The decision to build or buy should be based on the organizational context in which your IDP will be received. Building your own platform guarantees you can implement your exact workflow requirements and specific customization needs, but you need to be prepared for the complexity involved. If platform adoption is urgent, then selecting a prebuilt solution is a pragmatic way to reduce integration time and obtain immediate DevOps improvements.
Nonetheless, your planning should also account for your long-term strategy and development vision, as either type of IDP can affect your ability to execute in the future. A self-built platform is infinitely flexible but requires ongoing access to skilled internal teams; purchased options allow you to consistently focus on development work with minimal platform engineering investment but pose a constant threat of vendor lock-in.
Selecting a popular open-source platform like Backstage–seen by many as the original tool for building internal developer portals–is a third approach to consider. Not only can open-source mitigate or even eliminate cost and lock-in concerns, it also leaves you free to build your own extensions and bespoke components atop the platform.
You can evaluate prospective solutions across the following four priorities:
Priority | Build or Buy? |
---|---|
Organizational maturity and technical expertise | Build: Mature software organizations with access to skilled platform engineers will be able to exactly model their processes with minimal risk of being encumbered by the platform. |
Specific requirements and customization needs | Build: Building your own platform allows you to accommodate your precise requirements, including unique aspects that are unlikely to be supported in prebuilt solutions. |
Time to market and urgency of implementation | Buy: Prebuilt platforms are invariably faster to integrate and require fewer resources to get started. |
Long-term strategy and vision for the platform | Build: Consider building if the platform will be essential to your future strategy and you’re prepared to spend time maintaining it. Buy: Consider buying if you want the platform to be transparent, lack the resources to manually maintain it, or are unsure about your future commitment to DevOps and want to follow established best practices that are defined for you. Alternatively, use an open-source platform to get off the ground quickly, while retaining the option to fork the project and contribute custom functionality in the future. |
In this article, you learned how developer platforms can address the challenges of modern DevOps by offering a centralized platform that automates key processes in the SDLC. By giving devs self-service access to the tools and procedures they need for their work, IDPs can accelerate throughput, enhance developer productivity and satisfaction, and prevent you from becoming dependent on risky manual tasks.
Whether you choose to build or buy a solution, launching an IDP requires an upfront investment to integrate it with your processes and train your developers. You’ll also need to budget for the platform’s ongoing maintenance to support the changing requirements of your DevOps teams. However, the long-term efficiency improvements enabled by successful adoption can easily offset these costs, making an IDP one of the most valuable assets for high-performing software organizations.
Earthly Cloud: Consistent, Fast Builds, Any CI
Consistent, repeatable builds across all environments. Advanced caching for faster builds. Easy integration with any CI. 6,000 build minutes per month included.
In this roundup, you’ll learn about five popular platforms—Backstage, Qovery, Clutch, OpsLevel, and Appvia—by analyzing their core features, integration capabilities, user experience, and scalability, as well as the quality of their support and documentation. By the end of the article, you’ll have a better idea of which platform may be right for you.
Backstage, originally built by Spotify, is an open source Kubernetes-based developer platform that’s scalable, flexible, and easy to use.
Let’s take a look at a few of the reasons Backstage is a popular developer platform.
One of the main reasons for Backstage’s popularity is its modular architecture. All base functionalities (Software Catalog, Kubernetes, Software Templates, Backstage Search, and TechDocs) are part of the Backstage Core, while additional features can be added later (more on that shortly).
The Software Catalog acts as the inventory for all your services, enabling efficient organization and governance. Backstage’s Kubernetes feature is a monitoring dashboard that allows developers to assess the status of their services effortlessly, regardless of the deployment location, be it local or across multiple remote production clusters.
Software Templates automate the scaffolding of new projects, ensuring consistency and best practices, while Backstage Search offers a centralized search across all your documentation and resources, enhancing discoverability. Last but not least, TechDocs integrates documentation directly into the developer portal, allowing for seamless access and maintenance of technical content, streamlining workflows, and boosting productivity within the development lifecycle.
Backstage boasts notable flexibility through an extensive ecosystem of open source integrations and plugins. It supports the seamless incorporation of existing tools and services, enabling customization to suit unique workflow requirements. Developers can leverage an array of community-contributed plugins or create proprietary ones to extend Backstage’s functionality, ranging from CI/CD, monitoring, and cloud services to security scanning and incident management. Additionally, Spotify recently launched the Spotify Marketplace for Backstage, which offers enterprise-level and trusted third-party plugins. These paid plugins further improve the aspects of Backstage related to visibility, collaboration, and security.
This adaptability ensures that Backstage can evolve with your tech stack and maintain its role as a comprehensive yet friendly developer interface that provides an incredible developer experience.
Backstage is user-friendly and offers a developer-centric experience that simplifies navigation through a well-organized interface you can explore in the demo portal. Its intuitive design allows you to speed up onboarding, while the standardized setup across tools reduces complexity.
The platform’s consistent developer environment and the ability to access various services from a single portal enhance productivity. With features like Software Templates and TechDocs, Backstage empowers developers to focus on innovation rather than getting bogged down by processes and systems.
Backstage is engineered for scalability and caters to both small startups and large enterprises. As discussed, its modular architecture allows you to start small and expand as your needs grow without compromising performance. Additionally, since Backstage runs on Kubernetes, the platform manages an increasing number of services, plugins, and users with ease, maintaining a smooth experience. This, combined with Backstage’s ability to integrate with a vast range of tools and services, ensures that as your team and tech stack expand, Backstage can scale with you, facilitating continuous growth and development.
Backstage’s robust documentation is a cornerstone of its appeal, guiding users through setup, customization, and development with comprehensive material. Moreover, the Backstage Community Hub makes it easy to stay tuned to the latest developments and news regarding the platform.
However, for organizations seeking commercial support, Backstage might not be the best fit since neither Spotify nor the Cloud Native Computing Foundation (CNCF), which is currently in charge of the project, offer a managed service. Generally, the lack of direct commercial support could be seen as a disadvantage, placing the onus on in-house teams to deploy, maintain, and troubleshoot the platform. That said, the number of Backstage adopters continues to grow, including companies as large as American Airlines, Netflix, and Splunk, among others.
Clutch, born from Lyft’s engineering challenges, is a resilient open source platform for infrastructure tooling. Its customizable workflow engine uniquely supports diverse operational tasks, setting it apart with flexibility in managing infrastructure.
Clutch’s origin story and adaptability make it a noteworthy solution for dynamic infrastructure needs.
Clutch is known for its highly secure environment, offering fine-grained authentication and authorization control for resources and comprehensive security auditing for transparency. However, Clutch’s real strength lies in its modularity, or—as the Clutch team prefers to call it—the workflows and components approach.
The frontend is made up of different workflow packages, while the backend uses components named according to their task: services, modules, resolvers, and middleware. As you’ll learn soon, this provides great flexibility at the cost of some convenience. Nevertheless, workflows and components allow for seamless integration without the need for messy hacks, supporting both public and private extensions.
Additionally, thanks to its modularity, Clutch simplifies infrastructure management by serving as a single access point to various tech stacks, making complex processes simple.
Clutch is designed for easy customization and adaptability, thanks to its open and modular architecture. Teams can craft custom workflow packages with React and backend components using Go. For custom feature development, you create API definitions in Google’s proto3 format. Keep in mind, though, that while Clutch’s components are straightforward to tweak and enhance, it lacks a marketplace for plug-and-play integrations. This means that if the current features don’t meet your needs, you’ll have to actively develop your own solutions.
As is often the case with platforms that favor flexibility and extensibility, Clutch configuration is done via command line arguments. That is, configuring Clutch is done through Protobuf definitions. Similarly, at build time, your team should determine which workflows you want to install and register using Clutch’s command line scaffolding tool. This means that your comfort using command line tools (in comparison to graphical interfaces) will influence how user-friendly you find Clutch.
At the moment, Clutch does not distribute prebuilt binaries, so you have two options: build Clutch as a Docker container or run it locally using Go and Node.js. That means that if your goal is to ensure scalability, you can use Clutch’s supplied Dockerfile to build a container with all its core components, or you can build the container from scratch using only the components you want.
Keep in mind that this platform differs significantly from others in this roundup, as it doesn’t run natively on Kubernetes. Whether this is a benefit or drawback depends on your specific needs and how you plan to deploy your IDP.
Clutch, like Backstage, is a free, open source platform with a vibrant community. However, unlike Backstage, Clutch lacks commercial support. If you experience issues, you’ll have to rely on community channels such as GitHub or Slack, which may not meet the support needs of many organizations.
Qovery is a powerful platform designed to help developers and platform engineers accelerate deployment and streamline cloud infrastructure management. Originating as a solution to common DevOps challenges, it distinguishes itself with its strong governance and security capabilities, seamless integration with major cloud providers, and cost optimization for Amazon Web Services (AWS) and Kubernetes environments.
Qovery streamlines cloud infrastructure management by providing developers with self-service control over their infrastructure through automated environment provisioning. This powerful feature enables developers to create ready-to-run production and staging environments, even ephemeral environments, for quick tests. This ultimately accelerates application deployment.
Regarding governance and security, Qovery was designed from the ground up to comply with the General Data Protection Regulation (GDPR) as well as System and Organization Controls 2 (SOC2) security best practices. The proof of this lies in its powerful backup and restore feature, its encryption for data in transit as well as data storage and secrets, and its multifactor authentication and fine-grained access controls. On top of all that, Qovery recently released a public beta that provides access to detailed audit logs that facilitate debugging complex issues.
Qovery also offers cost optimization features, such as auto-start and stop environments and automatic deployment rules, which help reduce cloud expenses. This combination of automation, integration, governance, and cost efficiency positions Qovery as a robust solution for developers and platform engineers aiming for accelerated and controlled cloud-native development.
Qovery shines when it comes to flexibility, supporting over 100 integrations and plugins that cater to various DevOps tools and services. Additionally, Qovery provides interfaces for all tastes, including a CLI, a REST API, and a user-friendly web UI. Thanks to these diverse interfaces, Qovery manages to connect seamlessly with popular version control systems, container registries, and monitoring solutions to create a cohesive development ecosystem.
The platform’s flexibility also allows for the addition of custom functionalities via webhooks and the API, fostering a highly adaptable and efficient development process.
Qovery was crafted with a focus on simplicity and developer experience. Its intuitive interface and seamless integration with familiar developer tools (GitHub, GitLab, and Bitbucket, as well as Helm repositories) enable you to deploy applications with a minimal learning curve.
In addition, the platform makes it easy to deal with complex processes, like managing CI/CD with GitLab CI, CircleCI, GitHub Actions, and Jenkins, freeing up developers to concentrate on coding rather than infrastructure management. The result is a streamlined workflow that minimizes setup time and accelerates development cycles.
Another aspect in which Qovery excels is scalability. This is possible because the platform runs on top of a Kubernetes cluster, providing developers with inherent advantages to Kubernetes, such as horizontal and vertical autoscaling. Additionally, the Qovery Control Plane and Qovery Engine facilitate smooth environment provisioning, advanced app deployment rules, and resource adjustment in response to application demands, ensuring optimal performance and availability.
Overall, thanks to its autoscaling features, Qovery effortlessly handles traffic surges and load variations to maintain system efficiency. This scalability is crucial for modern applications that need to adapt to changing workloads swiftly, making Qovery an ideal choice for growth-oriented development.
A notable advantage of Qovery over Backstage is that it offers both a free tier and paid plans. In addition to its robust community support, Qovery also provides dedicated support for Team and Enterprise plans, including priority issue handling, to ensure minimal downtime and rapid resolution.
In addition to the official documentation, Qovery provides developers with detailed blog posts, guides, case studies, and tutorials that can help users through every feature and process, while also providing best practices and troubleshooting tips.
OpsLevel is a modern internal development platform that facilitates developer-centric operations and streamlines the complexity of modern software delivery by placing a huge focus on service ownership, maturity, and standardization.
OpsLevel seeks to address the pain points of engineering teams and developers alike by providing a variety of features in three key areas: a software catalog, service maturity, and self-service.
The software catalog enhances visibility with an up-to-date repository where developers can track services, systems, domains, infrastructure, and service dependencies, thus centralizing crucial information. A novelty that differentiates OpsLevel from other IDPs is that it can automatically populate service descriptions using generative AI, making it easier for team members to understand what each service is for.
Service maturity, standardization, and quality of code are achieved through automated service checks and maturity reports where you can define rubrics and scorecards that help you understand your services’ health and status, as well as campaigns that help visualize progress in the adoption of new standards or initiatives like upgrading libraries or addressing tech debt. Additionally, service quality and standardization are also encouraged through predefined action templates and integrated tech and API documentation, ensuring consistency across the board.
The platform’s self-service capability empowers developers with custom actions that enable them to execute workflows autonomously, thus accelerating task completion and fostering a culture of ownership and rapid innovation.
Flexibility is at the forefront of OpsLevel, with extensive integrations and plugins that enable teams to seamlessly connect with a vast array of tools and services in their ecosystem, such as Slack, New Relic, Kubernetes resources, Jenkins, and Grafana.
Additionally, OpsLevel provides out-of-the-box integrations for single sign-on (SSO) authentication with Okta, Auth0, Google, and other providers. It also offers integrations for automatic user provisioning using Okta, GitHub teams, and other providers supporting SCIM.
By accommodating custom plugins and supporting a wide range of third-party applications and services, the platform ensures that teams can tailor their workflows to their specific needs. This adaptability allows for a more cohesive and efficient development process that aligns with various tech stacks and operational strategies.
OpsLevel prioritizes a seamless developer experience, streamlining operations with a fully customizable internal developer portal from which your team can manage, view, or control services, groups, systems, domains, and more. Additionally, developers can interact with the OpsLevel API via the CLI or using the config as code paradigm by editing opslevel.yml
and pushing changes to the corresponding repository. Among other advantages, this versatility facilitates the quick onboarding of new team members.
All in all, OpsLevel’s ease of use and excellent developer experience not only boost efficiency but also keep developers focused on high-impact work, elevating overall satisfaction and output.
Like other IDPs in this roundup, OpsLevel’s scalability is anchored in its deployment on Kubernetes, whether as a software-as-a-service or self-hosted solution. Kubernetes ensures OpsLevel can efficiently handle growth in services, teams, and workloads. This flexibility allows organizations to scale their operations seamlessly, adapting to increased demands without compromising on performance or reliability and maintaining a consistent, responsive experience across the platform as their engineering ecosystem evolves.
Unlike Qovery, OpsLevel does not have a free tier, only a fourteen-day free trial, after which you can select a custom plan tailored to your organization’s needs. Support is provided via email, a dedicated Slack channel, and in-app chat.
One-on-one support is complemented by a variety of resources, including detailed docs, blog posts, guides, podcasts, and tech talks that cover the full spectrum of OpsLevel features, from setup and configuration to advanced usage. This extensive knowledge base is designed to facilitate self-guided learning and troubleshooting, allowing teams to leverage the platform’s full potential and streamline their operations with confidence and minimal external support.
Appvia Wayfinder is an IDP that originally addressed the UK Home Office’s complex tech challenges. It provides self-service cloud infrastructure for developers and platform teams with robust security, valuable cost management features, and a developer-centric approach.
Appvia’s ecosystem emphasizes a Kubernetes-centric approach to infrastructure management. This means that it aligns closely with Kubernetes methodologies and leverages its capabilities for container orchestration. In this regard, the Appvia ecosystem is akin to Rancher’s in that it provides Kubernetes management at an enterprise level, simplifying Kubernetes as a service.
Appvia’s key offerings include Wayfinder for centralized Kubernetes management and Cloud Landing Zones for establishing secure, compliant, and scalable cloud foundations.
You can think of Wayfinder as a developer self-service platform that simplifies deploying and managing Kubernetes infrastructure and applications. Wayfinder features are provided through Kubernetes custom resource definitions (CRDs). This can be positive or negative, depending on whether or not you want to build a developer platform following a Kubernetes-centric path. In any case, keep in mind that when installing Wayfinder, it only comes with a handful of CRDs that provide functionalities for cloud access, networking, cost optimization, DNS management, app management, security, and RBAC policies. To add extra functionality, you must also use CRDs (more on that later).
On the other hand, Appvia’s Cloud Landing Zones provide a secure, scalable foundation for cloud operations, incorporating governance, cost controls, and workload isolation to facilitate compliant and efficient cloud adoption and management. You can think of them as opinionated frameworks that provide out-of-the-box access management policies and governance controls, central audits, and compliance. Appvia provides on-request cloud landing zones for major cloud providers like Amazon Web Services (AWS) and Microsoft Azure. You can learn more about Cloud Landing Zones and their architecture in this blog post.
As mentioned, Wayfinder revolves around a Kubernetes-first approach; for this reason, its flexibility is not as impressive as the other IDPs in this roundup. However, you can expand functionality by creating your own CRD or following the Kubernetes operator pattern. For example, you could install the Jenkins Operator for continuous integration or simply browse Kubetools to find a tool suitable for your use case. All in all, don’t expect an Appvia marketplace for Wayfinder, which can be a drawback for some.
Once you install Wayfinder on AWS, Azure, or GCP, your team can interact with it either through its kubectl-like CLI, its API, or the Wayfinder portal. This is an advantage since Wayfinder fits with different workflows. Regardless of whether you prefer the UI or the command console, you can manage Wayfinder using abstraction layers and objects similar to those native to Kubernetes.
For instance, Wayfinder uses an abstraction layer called workspaces, which groups users and cloud infrastructure so they can be managed independently of other workspaces. This way, you can create users in a given workspace and assign them groups, roles, and access policies according to the needs of your organization.
Users with sufficient permissions can create clusters within their workspace and deploy applications that run on it. Users can also define individually deployable parts of applications, called components. Similar to Kubernetes namespaces, Appvia uses environments to isolate groups of resources within a cluster.
Overall, Wayfinder’s approach, based mostly on Kubernetes-like concepts rather than services that run on Kubernetes as other IDPs do, is something to keep in mind, given that it can impact the developer experience. That means developers familiar with Kubernetes will feel at home with Wayfinder, while developers with no prior experience may prefer more user-friendly platforms like Backstage, Qovery, or OpsLevel.
There’s not much to add about Wayfinder’s scalability. Since it runs on Kubernetes, it shares the scalability of other Kubernetes-based IDPs in the roundup. That said, its scalability in terms of functionality and adaptability to your organization’s needs is debatable. Wayfinder’s backbone is robust thanks to Kubernetes, but the lack of ready-to-use plugins and integrations could be a deal-breaker.
Appvia has numerous resources to help developers. For instance, Wayfinder’s official docs cover how to install Wayfinder, how to install the CLI, how to access the GUI, how to configure SSO and set cloud access, and how to manage DNS. Additionally, the documentation has an API reference and a CRD reference, which can be useful for developing your own solutions. Other resources available include Appvia’s blog and YouTube channel, as well as e-books and the Cloud Unplugged podcast.
Regarding commercial support, Wayfinder and Appvia Cloud Landing Zones are treated as independent products. Wayfinder has a free trial, after which you can choose between the Standard plan (support from 9 a.m. to 5 p.m. weekdays) and the Premium plan (support 24/7). Likewise, Cloud Landing Zones have a Standard plan (9 a.m. to 5 p.m., Monday to Friday support) and a Premium plan (one-hour response service-level agreement).
In this article, you learned that Backstage offers versatile yet generalized capabilities that are ideal for organizations looking for a user-friendly and widely known platform. However, maintaining the platform could be a challenge given the lack of commercial support.
Both Qovery and Appvia Wayfinder prioritize governance and security. However, Qovery edges out with superior flexibility and developer experience. OpsLevel, for its part, distinguishes itself by advocating for service ownership, maturity, and standardization alongside an impressive developer experience.
Which developer platform is the best? The answer revolves around what the specific needs of your organization are.
Earthly Cloud: Consistent, Fast Builds, Any CI
Consistent, repeatable builds across all environments. Advanced caching for faster builds. Easy integration with any CI. 6,000 build minutes per month included.
Medium to large companies typically use several different components and services for operations, including apps developed in-house, third-party services such as Google Cloud Provider (GCP) or Amazon Web Services (AWS), and third-party APIs. Managing all these services can become a complex task, which is why Backstage, an open source project, helps companies create developer portals that consolidate all their services, configurations, and secrets into one place. With Backstage, the portal that you create gives you a place to document each of your services and provides an overview of the services in use, their locations, and interdependencies.
In this article, you’ll learn how to create a developer portal with Backstage.
Picture this: you’re a developer at a growing company with a popular app. One day, you get a complaint from a customer that they’re not receiving emails from the app. You put your debugging glasses on and realize that the app uses a third-party service for sending emails. However, you have no idea what service is being used. You ask around, and it turns out that the developer who set up this integration has left the company. After an entire week of brainstorming, you finally succeed in replacing the component responsible for emails with a shiny new component. But now you realize that some other crucial functionality of the app is broken because you had no idea it depended on the old mailing system.
This is a familiar story in many organizations. As you add more and more software, services, and APIs to your tech stack, keeping track of them becomes increasingly challenging.
Backstage was created because Spotify ran into this same issue. Spotify needed a place to collect and document all its services in one place, and Backstage allowed it to do just that and eventually grow and become the company it is today.
In 2020, Spotify donated Backstage to the Cloud Native Computing Foundation (CNCF), where it received the love and support of the open source community and was developed into a community-driven effort aimed at simplifying development.
At its core, Backstage is a developer portal. A developer portal works as the heart of development and allows developers to quickly find what they need.
The following are a few of the reasons Backstage is a great developer portal:
Additionally, as previously stated, Backstage is open source, which means you can modify it to your heart’s content and host it in your own architecture.
If some of these features interest you, read on! In the next section, you’ll get a quick hands-on tutorial on how to set up Backstage and use its Software Catalog.
Before you begin this tutorial, you’ll need the latest version of Node.js installed as well as Yarn Classic installed and set up.
Note that you can upgrade to the latest version of Yarn later on, but to create the Backstage instance, you’ll need Yarn Classic.
You’ll also need to install and set up PostgreSQL on your computer.
To begin, you need to create a Backstage instance by running the following command:
npx @backstage/create-app@latest
When you’re prompted, enter the name of the directory where you want to set up the instance (eg backstage
).
Once the setup is complete, your output will look like this:
// Some output omitted
Moving to final location:
moving backstage ✔
init git repository ◜
Installing dependencies:
init git repository ✔
determining yarn version ✔
executing yarn install ✔
executing yarn tsc ✔
🥇 Successfully created backstage
All set! Now you might want to:
Run the app: cd backstage && yarn dev
Set up the software catalog: https://backstage.io/docs/features/software-catalog/configuration
Add authentication: https://backstage.io/docs/auth/
Navigate to the backstage
directory and install the dependencies by running npm install
.
Then, start the Backstage server with the following command:
yarn dev
This launches a Backstage instance at http://localhost:3000
and opens a browser window where you’ll be greeted with the default Backstage instance:
Before you proceed with the rest of the article, you’ll need to configure Backstage to use PostgreSQL as the database. By default, Backstage works with an in-memory database, which means any changes you make will be lost when you restart the server. To prevent that, a database such as PostgreSQL is needed where Backstage can store the data.
To configure PostgreSQL, install the pg
library by running the following command:
yarn add -cwd packages/backend pg
Open the app-config.yaml
file where the configuration for Backstage is stored. You’ll find a database
key that looks like this:
database:
client: better-sqlite3
connection: ':memory:'
By default, this sets Backstage up to use an in-memory database. You could edit this file to set up the PostgreSQL connection, but that’s not a secure approach because the configuration contains sensitive information, such as the database URL and password. The app-config.yaml
file is checked into version control, which means anyone who has access to your company’s version control system can read it.
It’s better to use app-config.local.yaml
for sensitive configurations. This file is not checked into version control, and any configuration in this file overrides the same from app-config.yaml
. So, open app-config.local.yaml
and add the following:
# Backstage override configuration for your local development environment
backend:
database:
client: pg
connection:
host: 127.0.0.1
port: 5432
user: USER
password: PASSWORD
Replace USER
with the PostgreSQL user and PASSWORD
with the password.
Restart the Backstage server and look for a line like this:
Performing database migration
This means the database connection is successful.
The default Backstage instance doesn’t perform any authentication, and anyone who can access the URL can access the Backstage instance. It’s a good idea to set up authentication for security purposes. With just a few lines of code, you can set up a GitHub login page using OAuth.
First, go to https://github.com/settings/applications/new and create a new OAuth app. Enter http://localhost:3000
in the Homepage URL field and http://localhost:7007/api/auth/github/handler/frame
as the Authorization callback URL. Give the app a name and click Register application:
On the next page, you’ll be shown a client ID that you’ll need to copy. Click the Generate a new client secret button and copy the generated client secret:
Open the app-config.local.yaml
file and paste the following YAML code into it:
auth:
# See https://backstage.io/docs/auth/ to learn about auth providers
environment: development
providers:
github:
development:
clientId: YOUR_CLIENT_ID
clientSecret: YOUR_CLIENT_SECRET
Replace YOUR_CLIENT_ID
and YOUR_CLIENT_SECRET
with the client ID and client secret you acquired.
Open packages/app/src/App.tsx
and add the following imports:
import { githubAuthApiRef } from '@backstage/core-plugin-api';
import { SignInPage } from '@backstage/core-components';
Search for const app = createApp({
, and below apis
, add the following:
components: {
SignInPage: props => (
<SignInPage
{...props}
auto
provider={ {
id: 'github-auth-provider',
title: 'GitHub',
message: 'Sign in using GitHub',
apiRef: githubAuthApiRef,
} }
/>
),
},
Restart the server, and you’ll be prompted with a login screen:
Once you log in with GitHub, navigate to Settings and verify that your name and email have been taken from GitHub:
Note: By default, Backstage comes with a guest sign-in resolver. With this resolver, all users share a single “guest” identity. You can read more about configuring user identities in the official docs.
To better understand some of the benefits of Backstage’s Software Catalog system, let’s imagine a scenario where your company has a product with a Node.js backend and a Python frontend client. You want to add them to Backstage’s Software Catalog.
The apps have already been created and hosted on GitHub to make your life easier. Here’s the Node.js backend and the Python client. You’ll be modifying the repos, so you must fork them to your GitHub account and clone them to your local computer. If you prefer to simply see the end result, both repos have a branch named final
that contains the final code.
Note: It isn’t necessary to run the apps to register them in Backstage, but if you want to run them, you can find instructions in the README
files in the repos.
To register components in Backstage, each component must have a catalog-info.yaml
file. This file contains metadata about the project and acts as the single source of truth for the components. Here, you’ll add the catalog-info.yaml
files to both repos.
To start, create a file named catalog-info.yaml
in the root of the Node.js app with the following code:
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: Blog-App
description: This is a dummy Node.js app
tags:
- node
spec:
type: service
lifecycle: experimental
owner: user:guest
In this code, the kind
key sets the kind of entity. In this case, Component
denotes that it’s a piece of software that you’re registering to Backstage. The metadata
key records information related to the component, such as name, description, and annotations. Finally, in the spec
key, you set the type, lifecycle, and owner of the component.
Commit this file and push the changes to your GitHub repo.
In the Backstage dashboard, click Create, and then click REGISTER EXISTING COMPONENT:
In the URL field, enter the URL to your repo’s catalog-info.yaml
file. It should be something like https://github.com/<YOUR_GITHUB_USERNAME>/<YOUR_GITHUB_REPO_NAME>/blob/<BRANCH_NAME>/catalog-info.yaml
. Replace <YOUR_GITHUB_USERNAME>
with your GitHub username, <YOUR_GITHUB_REPO_NAME>
with the name of the repo in your account, and <BRANCH_NAME>
with the name of the branch you’re working on:
When you click ANALYZE, Backstage fetches the catalog-info.yaml
file and extracts the entities from it:
Click IMPORT, and after the component is registered, click View Component. You’ll be taken to the entity page:
You can see that the name and description have been fetched from the catalog-info.yaml
file. On the left side, you’ll find a VIEW SOURCE button that takes you straight to the GitHub repo of the app. On the right-hand side, you’ll find a relations graph that shows the relations of this entity with other entities.
Right now, you can see that the user:guest
entity has an ownerOf/ownedBy
relationship with this component.
APIs are at the center of modern software development. Almost every piece of software either exposes an API for other software to communicate with or consumes an API to communicate with other software. The Blog-App
component also exposes an API. With Backstage, you can also catalog the API in the Software Catalog.
Open the catalog-info.yaml
file and add the following in the spec
key:
providesApis:
- blog-api
This tells Backstage that the Blog-App
component provides an API named blog-api
.
Now, let’s define the API. Add the code below at the end of the catalog-info.yaml
file. As before, replace the GitHub-related parts with information specific to your repo:
---
apiVersion: backstage.io/v1alpha1
kind: API
metadata:
name: blog-api
description: The Blog API
spec:
type: openapi
lifecycle: experimental
owner: user:guest
definition:
$text: https://github.com/<YOUR_GITHUB_USERNAME>/<YOUR_GITHUB_REPO_NAME>/blob/<BRANCH_NAME>/api/swagger.yaml
Notice that kind
is set to API
because this is an API entity, and the definition
key refers to the api/swagger.yaml
file in the repo. Backstage can generate a Swagger UI using the Swagger file.
The full catalog-info.yaml
file looks like this:
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: Blog-App
description: This is a dummy Node.js app
annotations:
backstage.io/managed-by-location: https://github.com/<YOUR_GITHUB_USERNAME>/<YOUR_GITHUB_REPO_NAME>/blob/<BRANCH_NAME>/catalog-info.yaml
tags:
- node
spec:
type: service
lifecycle: experimental
owner: user:guest
providesApis:
- blog-api
---
apiVersion: backstage.io/v1alpha1
kind: API
metadata:
name: blog-api
description: The Blog API
spec:
type: openapi
lifecycle: experimental
owner: user:guest
definition:
$text: https://github.com/<YOUR_GITHUB_USERNAME>/<YOUR_GITHUB_REPO_NAME>/blob/<BRANCH_NAME>/api/swagger.yaml
Commit and push the changes.
Once Backstage registers a component, it periodically refreshes the catalog-info.yaml
file by re-fetching it. That means once you push your changes, you should see them reflected automatically after a few minutes.
Once the component is refreshed, you’ll see a new entity has been added to the relations graph. The api:blog-api
entity has an apiProvidedBy/providesApi
relationship with the Blog-App
component:
If you go to the API tab, you’ll see blog-api
listed under Provided APIs:
Click blog-api
, and you’ll see that a Swagger UI has been generated from the Swagger file:
In a real-world project, components often depend on other components. With Backstage’s robust Software Catalog, you can record the dependencies between components.
In this section, you’ll focus on the Python client, which depends on the Blog-App
component and consumes the blog-api
API.
Add a catalog-info.yaml
file in the repo for the Python client:
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: Python-Client
description: This is a dummy Python app
tags:
- python
- web
spec:
type: website
lifecycle: experimental
owner: user:guest
consumesApis:
- api:blog-api
dependsOn:
- component:Blog-App
This is very similar to the metadata for Blog-App
. The consumesApis
key tells Backstage that this component consumes the blog-api
API, and the dependsOn
key tells Backstage that this component is dependent on the Blog-App
component.
As before, commit and push this code and register the component. You’ll notice that the relations graph shows how the Python-Client
component is related to the blog-api
API and the Blog-App
component:
If you go to the DEPENDENCIES tab, you’ll see that the Blog-App
component shows up as a dependency:
Click Blog-App
to be taken to its overview page, where you’ll see the relations graph has been updated to include the new Python-Client
component:
Congratulations! Now you not only have a catalog of your components, but you also have a clear understanding of how they’re related to each other.
Developer portals are a must for any company that uses a lot of services and components in its ecosystem. With a developer portal, you can consolidate all your components in one place, which increases developer productivity and provides a bird’s-eye view of the entire ecosystem.
Backstage helps you build a powerful developer portal with features like a Software Catalog, TechDocs, and Templates. In this article, you learned all about these features as well as how to set up a Backstage instance and register components in the Software Catalog. You also learned how to add APIs and dependencies among components.
Earthly Cloud: Consistent, Fast Builds, Any CI
Consistent, repeatable builds across all environments. Advanced caching for faster builds. Easy integration with any CI. 6,000 build minutes per month included.
Docker is a popular containerization solution for packaging, distributing, and running applications in lightweight environments. However, with growing container density and workload variety comes increased pressure to control container performance. Thankfully, Linux offers powerful tools, including namespaces and control groups (cgroups), that enable fine-grained resource allocation and guarantee the optimal performance of each container.
In this article, you’ll learn more about namespaces and cgroups and how to use them to control Docker performance.
Linux namespaces provide a mechanism for isolating system resources, enabling processes within a namespace to have their own view of the system, such as process IDs, network interfaces, and file systems. Docker uses namespaces to create isolated containers, each with its own set of resources. This ensures application separation and security.
The following are a few different types of namespaces:
cgroup
namespaceNamespaces in Linux provide a way to isolate and virtualize system resources, thus enhancing security by preventing processes in one namespace from directly interacting with processes in another namespace.
Namespaces increase security by providing a level of isolation that prevents unintended interactions between processes. This isolation is particularly valuable in containerization and virtualization scenarios, where multiple applications or services share the same host system but must be kept separate for security reasons.
cgroups
cgroups are a Linux kernel feature that enable the management and partitioning of system resources by controlling the resources for a collection of processes. Administrators can use cgroups to allocate resources, set limits, and prioritize processes. Docker utilizes cgroups to control and limit the resources available to containers.
Different types of available cgroups include CPU cgroup, memory cgroup, block I/O cgroup, and device cgroup.
While cgroups are not explicitly designed for security, they play a crucial role in controlling and monitoring the resource usage of processes.
Although namespaces and cgroups may appear similar in definition, they are fundamentally different and serve different purposes. Namespaces perform isolation by creating separate environments for processes that prevent one process from accessing or affecting other processes and/or the system. In contrast, cgroups distribute and limit resources like CPU, memory, and I/O among groups of processes. Often, namespaces and cgroups are used together for process isolation and resource management.
Now that you know more about namespaces and cgroups, it’s time to learn how to use them to control Docker performance.
All the code for this article is available in this GitHub repo.
To decrease a container’s attack surface, Docker offers the --user
option. The default user inside the containers is the root user. With the --user
flag, you can specify a non-root user and group to run the first process in containers. This lets you limit the potential impact of any security vulnerabilities.
Using the --user
option may hinder application functionality since some applications require elevated privileges to operate efficiently. When this occurs, your container configuration or app configuration may need to be altered accordingly to provide those permissions and grant the required privileges.
This is where a user namespace can help. By default, Docker runs containers with identical user and group IDs as their host system, which means that if an attacker gains entry through any one container in the system, they could potentially escalate their privileges. User namespaces can help mitigate such attacks by remapping the user IDs (UIDs) and group IDs (GIDs) used in the inside container with the outside container.
To better understand this concept, let’s use the user namespace to isolate the containers for security purposes.
namespaces
With DockerIn this scenario, you’ll learn about some of the advantages of namespaces in Docker. Run the following command to create a file in the host machine and make it readable to only the root user:
echo "This is a super sensitive file" | sudo tee /secret
sudo chmod 600 /secret
Let’s pretend that the file /secret
is a very sensitive file that should never be accessed by anyone other than the root. But, what happens when you mount the host filesystem in a Docker container? Let’s find out. Run the following command to start a busybox container and mount the host filesystem to it:
sudo docker run -it --rm -v /:/host busybox /bin/sh
Then, try to access the sensitive file:
cat /host/secret
$ This is a super sensitive file
As you can see, the root user in the Docker container has UID 0. When this user (with UID 0) tries to access the file in the host filesystem owned by the root user (who also has UID 0) the system happily obliges. But, this is a huge security risk as the Docker container can now freely read or modify any file on the host machine.
To prevent this, you need to make use of namespaces.
First, make sure that your Linux kernel supports user namespaces. To do so, find the configuration file for your kernel and use grep to search for CONFIG_USER_NS
. The following command is an example of the Ubuntu kernel:
grep -E '^CONFIG_USER_NS=' /boot/config-$(uname -r)
The following output confirms that your kernel supports user namespaces:
CONFIG_USER_NS=y
Next, open or create the Docker daemon configuration file (usually located at /etc/docker/daemon.json
) and add the following configuration that sets up user namespaces with default remapping:
{
"userns-remap": "default"
}
Restart the Docker daemon to apply the changes:
sudo systemctl restart docker
Run the container again:
sudo docker run -it --rm -v /:/host busybox /bin/sh
Try to read the file:
cat /host/secret $
This time, you’ll be faced with a Permission denied
error.
To better understand what is happening behind the scenes, execute the following command in interactive mode to run a Docker container:
docker run -it nginx sleep 300
Your output will look like this:
latest: Pulling from library/nginx
af107e978371: Pull complete
336ba1f05c3e: Pull complete
8c37d2ff6efa: Pull complete
51d6357098de: Pull complete
782f1ecce57d: Pull complete
5e99d351b073: Pull complete
7b73345df136: Pull complete
Digest: sha256:2bdc49f2f8ae8d8dc50ed00f2ee56d00385c6f8bc8a8b320d0a294d9e3b49026
Status: Downloaded newer image for nginx:latest
After running the container, check to see if the container is up and running:
docker ps
In this scenario, you’re running an Nginx container and executing the sleep command inside it.
Run the following command to list all currently running processes and filter the results to only show lines containing the word “sleep”:
ps -aux | grep sleep
The ps
command is used to provide information about processes.
Your output will look like this:
osboxes 7216 0.0 0.4 1329172 24576 pts/0 Sl+ 13:39 0:00 docker run -it nginx sleep 300
231072 7279 0.0 0.0 2484 1280 pts/0 Ss+ 13:39 0:00 sleep 300
osboxes 7329 0.0 0.0 17732 2560 pts/1 S+ 13:40 0:00 grep --color=auto sleep
Observe that one sleep 300
process shows up in the list of processes. It is owned by the user with the UID 231072 (this UID may be different for you). Where is this user coming from? Use the following command to look at the file /etc/subuid
:
cat /etc/subuid
Your output should look like this:
osboxes:100000:65536
ansible:165536:65536
dockremap:231072:65536
What this tells us is that Docker creates a default user named dockremap
with host UID 231072. This user is mapped to the UID 0 inside Docker containers. Any process started by the root user in the container is owned by the UID 231072 on the host, thus protecting it from privilege escalation.
Next, use the docker info
command to verify that user namespace support is enabled correctly:
sudo docker info
Your output should look like this:
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.10.4
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.17.3
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 1
Server Version: 23.0.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
runc version: v1.1.7-0-g860f061
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
userns
cgroupns
Kernel Version: 6.2.0-26-generic
Operating System: Ubuntu 22.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 5
Total Memory: 5.744GiB
Name: osboxes
ID: 80a2b682-9225-423a-bd2e-a0a3c61e8cf0
Docker Root Dir: /var/lib/docker/231072.231072
Debug Mode: false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
The numbers at the end of the Docker Root Dir
line indicate that the daemon runs inside a user namespace. The numbers should match the subordinate user ID of the dockremap
user as defined in the /etc/subuid
file.
Now that you know how you can use namespaces to increase security, let’s use cgroups to configure resource limitations. In this example, you’ll run a Docker container with CPU limits.
Let’s take a look at what cgroups are set up when you run a container. But, before that, you need to know if your system is using cgroup v1 or v2. The easiest way to find that out is to look for the file /sys/fs/cgroup/cgroup.controllers
. If it exists, you’re using cgroup v2, otherwise, you’re using cgroup v1.
Run the nginx
container and note the container ID from the output:
docker run -d nginx $
To find the cgroups for this container, you’ll need to look into the following locations, based on the cgroup version and the cgroup driver:
/sys/fs/cgroup/memory/docker/<container_id>/
on cgroup v1, cgroupfs
driver (default)/sys/fs/cgroup/memory/system.slice/docker-<container_id>.scope/
on cgroup v1, systemd driver
/sys/fs/cgroup/docker/<container_id>/
on cgroup v2, cgroupfs
driver/sys/fs/cgroup/system.slice/docker-<container_id>.scope/
on cgroup v2, systemd
driver (default)Here, you can find out different metrics for the container. For example, you can read the max CPU allocated to this container by reading the cpu.max
file in this directory:
max 100000
This implies that this container is allowed to consume the maximum available CPU on the host. Let’s limit it to half of one CPU. First, kill the container and recreate it with the --cpus
option:
docker run --cpus 0.5 -d nginx
The cpu.max
file should show the following:
50000 100000
This shows that the container is only allowed 0.5 CPUs. You can adjust the --cpus
value according to your desired CPU utilization.
If you want to run a Docker container with memory limits using cgroups, you can use the --memory
option with the docker run
command. The following example uses the official Nginx image from Docker Hub:
docker run -d --name new-container --memory=256M ubuntu sleep infinity
In this command, -d
makes the container run in the background, and --name new-container
assigns a name to the container. --memory 256M
limits the container to use a maximum of 256 mebibytes of memory, and ubuntu
is the image that’s being used. sleep infinity
tells the Docker command to run.
Now, run docker ps
. You should see that a container named “new-container” is up and running:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d67613c74bd6 ubuntu "sleep infinity" 26 seconds ago Up 25 seconds new-container
To verify that the memory limits are applied, run the following docker stats
command:
docker stats d67613c74bd6
Your output will look like this:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
d67613c74bd6 new-container 0.00% 388KiB / 256MiB 0.15% 3.42kB / 0B 0B / 0B 1
You can see the memory limit for this container is now set to 256 MiB.
To get real-time statistics for the running container, including CPU and memory usage, run the following command:
docker stats new-container --no-stream --format " {{ json . }}" \
| python3 -m json.tool
Your output will look like this:
{
"BlockIO": "0B / 0B",
"CPUPerc": "0.00%",
"Container": "new-container",
"ID": "d67613c74bd6",
"MemPerc": "0.15%",
"MemUsage": "388KiB / 256MiB",
"Name": "new-container",
"NetIO": "3.6kB / 0B",
"PIDs": "1"
}
In this output, you can see that Docker has applied a memory limit of 256 MiB in the "MemUsage"
field.
Mastering Linux namespaces and cgroups is essential for optimizing Docker performance. With the help of these features, administrators can fine-tune resource allocation, enhance security, and ensure the smooth operation of containerized applications.
As the landscape of containerization continues to evolve, a solid grasp of Linux namespaces and cgroups empowers users to harness the full potential of Docker and deliver high-performance, scalable applications. In this article, you learned how namespaces and cgroups are used by Docker, and how you can utilize namespaces for container isolation and cgroups for limiting and monitoring resource usage.
Earthly Cloud: Consistent, Fast Builds, Any CI
Consistent, repeatable builds across all environments. Advanced caching for faster builds. Easy integration with any CI. 6,000 build minutes per month included.