How We Slashed Our 22-Minute GitLab CI Pipeline to Just 8 Minutes

The 22-Minute Coffee Break

That's what our monorepo's CI pipeline had become: a mandatory, 22-minute break for every developer pushing a change. While seemingly nice, it was a massive drag on productivity. Feedback loops were painfully long, context switching was rampant, and the cost of idle engineers was adding up. The culprit? We were rebuilding and re-downloading the same dependencies and artifacts over and over again.

Our pipeline was inefficient, and it was time to fix it.

Our Multi-Layered Caching Strategy

We tackled the problem by implementing a multi-layered caching strategy that targeted the two biggest time sinks: Docker image builds and code compilation.

Here’s a visual of how caching transforms the workflow:

graph TD
    subgraph "Before Caching (22 mins)"

        A[Start] --> B[Install Dependencies];
        B --> C[Build Base Image];
        C --> D[Compile Code];
        D --> E[Run Tests];
        E --> F[End];
    end

    subgraph "After Caching (8 mins)"

        G[Start] --> H{Cache Hit?};
        H -- Yes --> I[Pull from Cache];
        H -- No --> J[Run Full Build and Push to Cache];
        I --> K[Run Tests];
        J --> K;
        K --> L[End];
    end

    style C fill:#ffcccc,stroke:#333,stroke-width:2px
    style D fill:#ffcccc,stroke:#333,stroke-width:2px
    style I fill:#ccffcc,stroke:#333,stroke-width:2px

1. Layer 1: Docker Remote Caching with BuildKit: Instead of rebuilding every Docker layer on every run, we configured GitLab CI to use a remote cache stored in our container registry. The --cache-from flag tells Docker to pull layers from a previously cached image, dramatically speeding up the docker build step.

2. Layer 2: Shared Compiler Cache with sccache: For our services written in compiled languages like Rust and C++, we introduced Mozilla's sccache. It's a ccache-like tool that shares a compilation cache across multiple CI runners, meaning if one runner has compiled a specific piece of code, others don't have to.

3. Layer 3: Intelligent Cache Keys: A cache is useless if it's always stale or, worse, provides the wrong dependencies. We keyed our caches to the hash of our dependency files (go.sum, package-lock.json, etc.). The cache is only invalidated and rebuilt when a dependency actually changes.

The Implementation

Here is a simplified snippet from our .gitlab-ci.yml showing how to enable the remote Docker cache. The key is using BUILDKIT_INLINE_CACHE=1 and the --cache-from flag.

# .gitlab-ci.yml (snippet)
variables:
  # Enable the BuildKit engine
  DOCKER_BUILDKIT: 1
  # Enable caching metadata to be stored with the image
  BUILDKIT_INLINE_CACHE: 1

build:
  stage: build
  image: docker:27
  services: ["docker:27-dind"]
  script:
    # 1. Try to pull a cached image to use its layers
    # 2. Build the new image, tagging it as the new cache
    # 3. Push the new image and the updated cache
    - docker build \
      --cache-from "$CI_REGISTRY_IMAGE:cache" \
      --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" \
      --tag "$CI_REGISTRY_IMAGE:cache" \
      .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
    - docker push "$CI_REGISTRY_IMAGE:cache"

The Results: Faster Feedback, Happier Devs

The impact was immediate and profound:

• 62% Faster Pipelines: Median build time dropped from 22 minutes to just 8 minutes.
• Faster Iteration: Developers could get feedback on their pull requests in minutes, allowing for rapid iteration and testing.
• Reduced Risk: We had to mitigate the risk of cache poisoning with smart keys, but the performance gains far outweighed the effort.

The Bottom Line: A Massive ROI

By translating the saved time into engineer-hours, the business impact becomes crystal clear:

~14 minutes saved per pipeline × 120 runs/week ≈ 28 engineer-hours saved every single week.

This wasn't just a technical win; it was a significant boost to the entire engineering organization's velocity.