Fully automated dependency updates with source operations

This article is a follow up to the initial article that discussed using source ops for updates. The original article is still there for historic reasons.

Goal

Much of what we write these days is helped by dozens of dependencies. That is awesome, but keeping your site up to date is therefore not just about your own code. You also have to regularly update your dependencies to make sure security and bug fixes are applied to your dependencies.

Running composer update on your development machine is a good start, but we can automate this tedious task away with source-operations.

Assumptions

You will need:

• An API token added to your project as an environment variable. (learn how to do this here)

High level overview

What we want our automation to do is the following:

• Create an update branch from your production branch.
• Run composer update, npm update, etc… on the update branch.
• Run some tests on the update branch to verify that the update didn’t break anything
• Merge the update into prod

Steps

1. Add a updater.bash file in your project.

The updater.bash file is provided as is, feel free to read over the code (it’s quite readable, I promise!). This file will be included in your own project source, so tailor it to your needs if you want.

#!/bin/bash

function trigger_source_op() {
    ENV="$1"
    SOURCEOP="$2"

    if [ "$PLATFORMSH_CLI_TOKEN" == "" ]; then
        echo "env:PLATFORMSH_CLI_TOKEN is not set, please create it."
        exit 1
    fi

    ensureCliIsInstalled

    echo "Looking for production branch... "
    PRODUCTION_ENV=$(platform e:list --type=production --columns=ID --no-header --format=csv)
    echo "Production branch = $PRODUCTION_ENV"

    createBranchIfNotExists "$ENV" "$PRODUCTION_ENV"
    runSourceOperation "$SOURCEOP" "$ENV"


    waitUntilEnvIsReady "$ENV"
    test_urls "$ENV"
    mergeAndDelete "$ENV"

}

function ensureCliIsInstalled() {
    if which platform; then
        echo "Cli is already installed"
    else
        echo "Cli not installed, installing..."
        curl -sS https://platform.sh/cli/installer | php        
    fi
}

function createBranchIfNotExists() {
    BRANCH_NAME="$1"
    BRANCH_FROM="$2"

    echo "Creating branch '$BRANCH_NAME'"

    CURRENT_BRANCH=$(platform e:list --type=development --columns=ID --no-header --format=csv | grep "$BRANCH_NAME")

    if [ "$CURRENT_BRANCH" == "$BRANCH_NAME" ]; then
        echo "Branch already exists, reactivating"
        activateBranch "$BRANCH_NAME"
        if platform sync -e "$BRANCH_NAME" --yes --wait code; then
            echo "Branch synced"
        else
            echo "Failed to sync"
            exit
        fi
    else
        platform branch --force --no-clone-parent --wait "$BRANCH_NAME" "$BRANCH_FROM"
        echo "Branch created"
    fi
}

function activateBranch() {
    ENV_NAME="$1"
    echo "Activating branch '$ENV_NAME'..."
    platform environment:activate "$ENV_NAME" --wait --yes
    echo "Environment activated"
}

function runSourceOperation() {
    SOURCEOP_NAME="$1"
    ENV_NAME="$2"
    echo "Running source operation '$SOURCEOP_NAME' on '$ENV_NAME'..."
    if platform source-operation:run "$SOURCEOP_NAME" --environment "$ENV_NAME" --wait ; then
        echo "Source op finished"
    else
        echo "Source op failed to run"
        exit
    fi
}

function waitUntilEnvIsReady() {
    ENV_NAME="$1"
    echo "Waiting for '$ENV_NAME' to be ready..."
    
    until [ "$is_dirty" == "false" ] && [ "$activity_count" == "0" ]; do
        sleep 10
        is_dirty=$(platform e:info is_dirty -e "$ENV_NAME")
        activity_count=$(platform activity:list -e "$ENV_NAME" --incomplete --format=csv | wc -l)
    done
}

function test_urls() {
    ENV_NAME="$1"
    for url in $(platform url --pipe --environment "$ENV_NAME"); do
        echo -n "Testing $url";
        STATUS_RETURNED=$(curl -ILSs "$url" | grep "HTTP" | tail -n 1 | cut -d' ' -f2)

        if [ "$STATUS_RETURNED" != "200" ]; then
            echo " [FAILED] $STATUS_RETURNED"
            exit
        else
            echo " [OK] $STATUS_RETURNED"
        fi
    done
    echo "All tests passed!"
}

function mergeAndDelete() {
    ENV_NAME="$1"
    echo "Merging '$ENV_NAME'..."
    platform merge "$ENV_NAME" --no-wait --yes
    echo "Removing '$ENV_NAME'..."
    platform e:delete "$ENV_NAME" --no-wait --yes
}


function update_source() {
    declare -A cmds
    cmds["composer.lock"]="composer update --prefer-dist --no-interaction"
    cmds["Pipfile.lock"]="pipenv lock"
    cmds["Gemfile.lock"]="bundle update"
    cmds["package-lock.json"]="npm update"
    cmds["go.sum"]="go update -u all"
    cmds["yarn.lock"]="yarn upgrade"

    WAS_UPDATED=false

    echo "Updating source of $PLATFORM_BRANCH"

    # find each directory that has a .platform.app.yaml file
    for yaml in $(find . -name '.platform.app.yaml' -type f); do 
        DIRECTORY=$(dirname "$yaml")
        # then, check each directory for the existance of package files (composer.json)
        for PACKAGE_FILE in ${!cmds[@]}; do
        if test -f "$DIRECTORY/$PACKAGE_FILE"; then
            # and when we find one, execute the package update command (composer update, npm update, ...)
            echo "$PACKAGE_FILE exists. Running ${cmds[$PACKAGE_FILE]}"
            ${cmds[$PACKAGE_FILE]}
            WAS_UPDATED=true
        fi
        done
    done

    # if we did an update, commit the changes
    if $WAS_UPDATED; then
        date > last_updated_on
        git add .
        git commit -m "auto update"
    fi    
}


function help() {
    echo "Usage: "
    echo " bash updater trigger_source_op ENV SOURCEOP (makes sure a branch named ENV is created, and then triggers the source operation named SOURCEOP)"
    echo " bash updater update_source (runs composer update and git add/commit)"
    exit 1
}

ACTION="$1"

case $ACTION in

  trigger_source_op)
    ENV="$2"
    SOURCEOP="$3"
    if [ "$ENV" == "" ] || [ "$SOURCEOP" == "" ]; then
        help
    fi
    trigger_source_op "$2" "$3"
    ;;

  update_source)
    update_source
    ;;

  install_cli)
    ensureCliIsInstalled
    ;;

  *)
    help
    ;;
esac

2. Define the source operation

Open up your .platform.app.yaml file and add a few lines to it to define the source operation.

source:
  operations:
    update_dependencies: # you can name the source operation whatever you want, but remember the name, because you'll need it when calling it (in the cron)
      command: |
        bash updater.bash update_source

3. Define the cron operation

The cron will run on the production environment and will preferably run in the middle of the night.

timezone: "Europe/Paris" # Change this to your timezone https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
crons:
  update:
    spec: '0 3 * * *' # update every night at 3am
    cmd: |
      if [ "$PLATFORM_ENVIRONMENT_TYPE" == "production" ]; then
          bash updater.bash trigger_source_op update_dependencies psh_auto_updater_branch
      fi 

A deeper dive

What does the source op do?

update_source will check each directory that contains a .platform.app.yaml and by default it will run:

• composer update --prefer-dist --no-interaction if a composer.lock file is found
• npm update if a package-lock.json file is found
• bundle update if a Gemfile.lock file is found
• go update -u all if a go.sum file is found
• pipenv lock if a Pipfile.lock file is found
• yarn upgrade if a yarn.lock file is found

If any files were updated, it commits the changes to the branch by running:

        date > last_updated_on
        git add .
        git commit -m "auto update"

What does the cron do?

trigger_source_op takes 2 parameters.

1. The branch name to use to run the update
2. The source op to run

After some initial checks, it will:

• Create the branch name if it doesn’t exist
• Trigger the source op
• Check if each platform url still returns a HTTP 200 (or is redirected to one)
• Merge into production.

How to run the source op manually

platform source-operation:run update_dependencies -e updater_branch

The above assumes you already have a branch called updater_branch. If not, you can branch it using
platform branch --force --no-clone-parent --wait psh_auto_updater_branch <your_production_branch>

Note: you can run the source operation on your production branch directly, but I highly recommend against doing this unless you don’t value site uptime.

Conclusion

Updating dependencies with source operations was already possible. But having a handy bash script available to do the heavy lifting for us makes our .platform.app.yaml file cleaner and much easier to reason about.

We can expand on this idea further by adding notifications to tell us when the automation detects that the dependency update fails the tests. But that, my friends, is a story for another day…