Working with Upsun and DDEV - Tokens and Environment variables

The information in this post is accurate as of the published date July 17, 2024 08:49. Please make sure to check any linked documentation within the post for changes/updates.

DDEV (local container-based development environment) comes with Upsun (remote container-based hosting environment) integrations!

This means that you can use the `ddev pull upsun` & `ddev push upsun` commands to sync your instances - when it is correctly set up.

DDEV docs (DDEV: Upsun Integration) and
Upsun docs (Upsun : Use DDEV for local development)
describe how to get this set up. But it may help to explain a little deeper.

For DDEV to be able to read from and pull/push data from your Upsun environment, it needs to be given permission to act on your behalf.

The DDEV local containers (docker) are sandboxed by default - they cannot see or use your personal credentials by default.

You may have already noticed that you cannot simply `ssh` out from a ddev environment to your other target destinations - as a ddev container does not have access to your private keys (unless you deliberately share them, or set up ssh key-forwarding. That's another topic :-), or see `ddev auth ssh`.

Also, it's difficult to use the Upsun TFA (web login) methods from inside the docker container, so we don't do that. Instead we create a token for your account using the Upsun console, which becomes a key to your account that can be used to interact with the Upsun API.

Share your key with yourself, don't copy & paste it.

The documentation for both ddev and Upsun currently recommends adding this token directly to your `~/.ddev/global_config.yaml` file, either manually or by using the `ddev config` command.

This works, but it's also possible to:

Set your key as a local environment variable on your workstation for yourself.
Then share that key with ddev, instead of writing it into the config file.

This has an advantage I'll describe in this HOWTO.

To do this, we:

Create and fetch the API token from the console.
Add it to your `~/.profile` (or `~/.zprofile` or whatever is best for your shell)
export UPSUN_CLI_TOKEN='b1gl0ngenC0ded_y0uR70Ken-B1a'
Tell ddev to expose and share this value with the ddev web environments. Docs for ddev Custom Environment Variables.
- I use variable substitution instead of static literals.

# ~/.ddev/global_config.yaml

web_environment:
  - UPSUN_CLI_TOKEN=${UPSUN_CLI_TOKEN}

To verify this has worked:

Restart your local shell session, and verify that the environment variable is now available.
echo $UPSUN_CLI_TOKEN
Restart ddev completely,
ddev restart
Then verify that it's available inside the ddev container environment
ddev exec 'echo $UPSUN_CLI_TOKEN'
At this point, you should be able to also invoke upsun CLI commands from within the container.
ddev exec ‘upsun project:info’
- should respond with your project info.
- Although upsun CLI will work by auto-detecting the project from context, it is currently still required to set the UPSUN_PROJECT & UPSUN_ENVIRONMENT environment variable (mentioned in the docs) to do syncs.

Use this new authentication permission to run the ddev sync commands

ddev pull upsun

You're about to delete the current database and files and replace with the results of a fresh pull.
Would you like to continue? [Y/n] (yes): 
  Authenticating...
  Obtaining databases...
  ...
  Importing files...
  Pull succeeded.

ddev launch

Happiness!

Set up other very useful environment variables

You may not encounter this the first or second time, but at some point in development, you may start to get problems from nowhere! Unable to fetch from remote repository!

Because the build scripts that are running inside the ddev environment are automated, and because they may run frequently, then this can look a bit like `bot activity.

Rate limiting

The remote repositories that may be getting called from these build scripts - like github.com (for your build dependencies), or packagist.org (for composer) sometimes put rate-limiting in place. This is well documented by them - you just have to use your personal developer account to generate an personal access token, and use that when requesting content from them.

But we need to do this for our sandboxed ddev container. When ddev invokes invokes `composer install` or a build process that may do a git pull from github, we need to sign that request as coming from our account.

Private repository access

The same caveat applies to private repositories. If you are trying to pull content from a private repo (github or otherwise) you may still need to be able to provide your personal credentials to the ddev container so that it can request that resource on your behalf. With composer, this is done by setting a `COMPOSER_AUTH` variable (a json string). While this could be done per-project, it's also convenient to set it locally for your own user account, and share it with your projects.

Note that private repository access will also have to be granted to your Upsun project environments to allow it to do builds! Upsun: Private git repository, Upsun: Composer Authentication documents this. We have to do something similar within ddev

How to add and share a private token

Follow the providers instructions for adding a developer token to your local development environment.
- Fetch it from their console.
- Add it to your ~.profile.
Publish that new environment variable through to ddev instances, so that your key is used when ddev runs builds.

So my `~/.ddev/global_config.yaml`

ends up containing a whole bunch of shared credentials:

web_environment:
  - PLATFORMSH_CLI_TOKEN=${PLATFORMSH_CLI_TOKEN}
  - UPSUN_CLI_TOKEN=${UPSUN_CLI_TOKEN}
  - GITHUB_TOKEN=${GITHUB_TOKEN}
  - GITHUB_OAUTH=${GITHUB_OAUTH}
  - COMPOSER_AUTH=${COMPOSER_AUTH}

Summary

Working solely within a container-based development environment such as `ddev` has some differences from “native” workstation-based development. You don't have access to all your tools unless you install them again yourself, and you don't have access to all of the little security settings either.

With all these environment variable sharing rules in place, then many of your actions, and the actions of your build scripts, will be able to inherit the credentials needed to access the remote APIs and Repositories that are needed!

Many of the same limitations of a ddev container apply to the sandboxed Upsun deployment environments, so getting your dependencies right locally with ddev will be good practice for a seamless Upsun experience! You should plan to add your required authentication variables, API keys and tokens to your Upsun project in the same way as you did for ddev.