Heroku Pipelines is a feature that allows you to manage multiple environments through a workflow where changes are verified and then promoted through the pipeline.

A pipeline is a group of Heroku apps that share the same codebase. Each app in a pipeline represents one of the following stages in a continuous delivery workflow:

Development
Review
Staging
Production

What this means in practice is that you deploy your app to the first environment in the pipeline and then use the Promote command to push that change through the pipeline. I first used this feature a few years ago on a simple project where I had a staging and production environment. I discovered the pipeline feature and thought that it’s a neat way to mimic my existing workflow - although I would usually just push the latest change to staging, verify the change works in a deployed environment, and then push the latest change to production. Using pipelines seems like a more streamlined and professional approach to the same workflow.

Custom Font Problems

Initially using the pipeline feature was great, and I didn’t have any problems with it. However, a few weeks later I noticed that some of the fonts in production were not being rendered correctly. I was using Font Awesome for various custom icons and in production these were being rendered as squares, indicating some sort of problem with CORS, or asset precompilation. I assumed that I had misconfigured something in Amazon CloudFront or in the CORS settings, but what made the problem extra confusing was that I couldn’t reproduce it in the staging environment.

I spent a frantic few hours trying to understand the root of the problem and then found the smoking gun - the fingerprinted stylesheet files in production were referencing fonts in the staging environment. This helped me to narrow my search and I found a clear explanation on a different Heroku article - Please Do Not Use Asset Sync. To be clear, the problem was not that I was using asset sync, but rather that I was using a different CloudFront Distribution for each environment.

From the article:

If you are using pipelines to promote a staging slug to production and are serving assets via a CDN, then your asset pre-compilation will only be run once, on the staging app.

Using pipelines encourages the practice of always deploying to staging first to catch bugs. It also makes promotions to production very fast. The down side is that when an app is “promoted” to production the files on disk are moved to the new app and no new rake assets:precompile is run. Because of this, assets compiled with a “staging” CDN url inside of them will be promoted to production:

body { background-image: url(“http://staging-cdn.example.com/assets/smile-.png") }

For many applications sharing a CDN between staging and production is surprising, however causes no problems. You can manually configure both to have the same “asset_host”. This works because even if the copy of the assets has been modified on the staging app, all assets are fingerprinted, and the “production” app will point browsers at the correct fingerprints. The staging app will serve up to 3 old copies of assets, and the CDN should keep assets cached indefinitely.

This was clearly the reason why I was seeing staging assets being referenced incorrectly in production. I found two related Github issues - one on the Suspenders Gem and another on Heroku Pipelines itself confirming the behavior I was seeing. As far as I could tell I had three options for resolving this:

1. Stop using Heroku Pipelines
2. Use the same Cloudfront Distribution for both environments
3. Keep using a CloudFront Distribution for each environment, but configure CORS as allow-origin '*' (this is the most hacky solution, so it’s not really an option IMO)

I do take issue with this part of the Heroku Article:

For many applications sharing a CDN between staging and production is surprising, however causes no problems. You can manually configure both to have the same “asset_host”.

That really should say “you have to manually configure both to have the same asset_host otherwise your assets will be broken in production”. To their credit, this is clearly called out in the Pipelines documentation (Design Considerations) but I clearly missed it on first read.

Manually Configuring a Build Pipeline

I still feel strongly that having a separate CloudFront Distribution per environment is best practice - if I’m making a change to the CDN configuration I definitely want to be able to test that in a production-like environment first. To solve my original problem I simply deployed the latest git commit to production directly, without using the pipeline. I could simply keep doing this, but having some sort of automated or scripted process to promote the staging changeset to production was still appealing to me.

I ended up writing a simple bash script that would do the following

1. When deploying to stage, push the latest change to the Heroku staging environment
2. When deploying to production, fetch the latest change from staging and push that to the Heroku production environment

I relied on git tags to keep track of the latest deployed changes - so at the end of step 1 I created a tag with the timestamp, eg. staging/202401151156. In step 2 I would then iterate over the tags from the staging environment, find the latest tag, and push that to production. I also added a simple check to ensure that the latest SHA in the Heroku repository matches the SHA from the tag.

#!/bin/bash --login

set -e

GREEN=$(tput setaf 2)
RED=$(tput setaf 1)
RESET_COLORS=$(tput sgr0)

echo_green () {
  echo -e "${GREEN}$1${RESET_COLORS}"
}
echo_red () {
  echo -e "${RED}$1${RESET_COLORS}"
}

PIPELINE=("staging" "production")
ENVIRONMENT_NAME=$1

verify_heroku_commit_matches_tag() {
  heroku_environment=$1
  tag_name=$2

  echo_green "\nChecking sha integrity in heroku config"
  heroku_git_sha=$(git rev-parse ${heroku_environment}/main)
  tagged_git_sha=$(git rev-list -n 1 $tag_name)

  if [ $heroku_git_sha != $tagged_git_sha ]; then
    echo_red "Integrity checked failed!"
    echo_red "Heroku sha: $heroku_git_sha"
    echo_red "Tagged sha: $tagged_git_sha"
    echo_red "Aborting"
    exit 1
  fi
  echo_green "Integrity check passed"
}

create_deployment_tag() {
  heroku_environment=$1

  deployment_tag_name="${heroku_environment}/$(date +"%Y%m%d%H%M")"
  echo_green "\nCreating tag $deployment_tag_name\n"
  git tag -a $deployment_tag_name -m "Deployed to $heroku_environment on $(date +"%Y-%m-%d %H:%M %Z")"
  git push origin $deployment_tag_name
}

pipeline_index=-1
for i in "${!PIPELINE[@]}"; do
   if [[ "${PIPELINE[$i]}" = "${ENVIRONMENT_NAME}" ]]; then
       pipeline_index=$i
   fi
done

if [ $pipeline_index = -1 ]; then
  echo_red "Invalid environment specified!"
  echo_red "Usage: ./scripts/deploy.sh environment"
  echo_red "Valid environment values are ${PIPELINE[*]}"
  exit 1
fi

echo_green "\nStarting deployment to $ENVIRONMENT_NAME\n"

echo_green "Updating git to latest"
git checkout main
git fetch origin main
git fetch --tags
git reset --hard origin/main
echo_green "Done\n"

echo_green "Checking heroku git remote"
git remote show $ENVIRONMENT_NAME

if [ $pipeline_index = 0 ]; then
  latest_git_sha=$(git rev-parse HEAD)
  echo_green "\n$ENVIRONMENT_NAME is the first phase in the deployment pipeline - deploying the latest main commit $latest_git_sha\n"
  git push ${ENVIRONMENT_NAME} ${latest_git_sha}:main

  create_deployment_tag "$ENVIRONMENT_NAME"
  verify_heroku_commit_matches_tag "$ENVIRONMENT_NAME" "$deployment_tag_name"
else
  previous_environment=${PIPELINE[$pipeline_index-1]}
  echo_green "Deploying the latest tag from $previous_environment\n"

  latest_tag=$(git for-each-ref --sort=taggerdate --format '%(tag)' | grep "^${previous_environment}" | tail -n 1)
  if [ -z $latest_tag ]; then
    echo_red "\nCould not find the latest git tag for $previous_environment"
    echo_red "Aborting"
    exit 1
  fi

  echo_green "Latest tag from $previous_environment is $latest_tag"
  tagged_git_sha=$(git rev-list -n 1 $latest_tag)
  verify_heroku_commit_matches_tag "$previous_environment" "$tagged_git_sha"

  echo_green "Deploying commit $tagged_git_sha"
  git push ${ENVIRONMENT_NAME} ${tagged_git_sha}:main

  create_deployment_tag "$ENVIRONMENT_NAME"
  verify_heroku_commit_matches_tag "$ENVIRONMENT_NAME" "$deployment_tag_name"
fi

echo_green "\nAll done!"

exit 0

The environments in the pipeline are hardcoded - PIPELINE=("staging" "production"). This script assumes that you have a git remote matching the environment name, so I have these remotes:

production https://git.heroku.com/my-app-production.git (fetch)
production https://git.heroku.com/my-app-production.git (push)
staging    https://git.heroku.com/my-app-staging.git (fetch)
staging    https://git.heroku.com/my-app-staging.git (push)

This script allowed me to keep the concept of a pipeline without the downside of a single CDN. If I had to re-implement this I would probably try to make it all work as Github Actions.

Final Thoughts on Heroku Pipelines

I think Heroku is optimizing for the simplest solution, and the idea of taking the entire slug from one environment and promoting it to the next does make sense from that perspective. However, asset_host is a Rails configuration per environment - which to me means that the Heroku assumption of a single CDN across all environments at the very least clashes with the Rails conventions.

Ultimately I don’t think I’m willing to give up having a CDN per environment, especially when the Heroku Pipelines doesn’t add an a huge benefit over a simple scripted solution.

Share on: Twitter, Facebook, or use the Permalink