Articles in this section

A workflow for ongoing developing and testing with DDEV and Upsun

Matt Sinagra
Updated

With the DDEV integration support for Upsun, we can prepare a workflow for a sustainable development cycle. This works best if we take advantage of a few of the power tools for this. I won't be able to fully document them all, but I'll provide some pointers about what's available and where to find the full docs.

This tutorial assumes you are already past the "getting started" phase, and have a working Upsun project. We will now develop a workflow for the team members to configure their local (container based) development environments and contribute to the project over time.

Requirements

I will be working locally on OSX, but as we are using a container-based solution, most of the operations I describe should be fully portable through Windows and Linux desktops as well.

  • DDEV installed

      ddev --version
  ddev version v1.23.2
    • Requires a docker hosting system. I use Orbstack this month as recommended, but original Docker is fine.

  • Upsun CLI installed and working.

    upsun --version
Upsun CLI 5.0.16

  • A working account registered with Upsun with permission to work on a project

    upsun auth:info
+-----------------------+--------------------------------------+
| Property              | Value                                |
+-----------------------+--------------------------------------+
| id                    | u5e7he-f0rc3-D00D-bf36-987654321000  |
| first_name            | Thomas                               |
| last_name             | Anderson                             |
| username              | novo                                 |
| email                 | thomas.anderson@matrixwebhosting.com |
| phone_number_verified | false                                |
+-----------------------+--------------------------------------+

  • You should have access to the existing, working Upsun project, and the project ID. I will refer to it using the variable from now on.

     UPSUN_PROJECT=iii000bbbccc
  • git should be installed and working. All deployments happen through git.

I assume a comfortable IDE (I use PHPStorm) and a bit of familiarity with a working Shell environment. I use zsh because OSX, but WSL on windows seemed to work for me also.

Scenario

I will work through some processes to make syncing data from your production site down into development, testing updates locally, and safely releasing them.

I will start from the point at which you have a working Upsun site deployed already, and need to take a copy of that into a local DDEV project into your development environment, make a change, and safely release that.

This is not a getting started guide - that's already written up in other tutorials. This is a "making progress" guide. Useful for what you need to share with your team after doing the complicated work of getting it set up.

Setup

Pull the code

You will need to start with a copy of the codebase. In the place where you want to put your workspace, run:

upsun get $UPSUN_PROJECT

Selected default environment: main (type: production)
Directory (default: my-upsun-project): my-upsun-project
Downloading project My Upsun Project (i4nqxhyy3fnfa)
Cloning into '/Users/dman/www/my-upsun-project'...
remote: counting objects: 9058, done.        
Receiving objects: 100% (9058/9058), 29.95 MiB | 4.46 MiB/s, done.
Resolving deltas: 100% (1696/1696), done.

The project My Upsun Project (i4nqxhyy3fnfa) was successfully downloaded to: my-upsun-project

cd my-upsun-project

Get DDEV working

... before moving forwards, as DDEV provides a safe virtual environment to work within without disturbing your local environment.

If your team has already set up and shared ddev configs for this project, then you can maybe run

ddev start 

Otherwise, follow the ddev config setup instructions

You should soon be able to connect to the virtual environment, but don't expect to see the web service running well just yet.

ddev ssh
 

Attach your IDE

You've got a copy of the code locally now. I like to attach the IDE to that as soon as possible. You've also got the URL for the local development environment that ddev provided, so now is a good time to warm up PHPStorm.

If using PHPStorm, it looks like this:

  • "File > New Project from Existing Files"
  • "Web server is installed locally, source files are located under its document root"
  • Choose your project root location /var/www/my-upsun-project
  • "Specify the local server" > "Add new local server"
    • Use the URL that ddev provided when it started, or found with dev launch
    • Choose “Don’t check HTTPS status” for now, our site isn’t ready to ping yet.
  • “Specify project web path” : leave blank

You’ve now got a new workspace

At this point, your vendor or libraries or node_modules directory is still empty, so things don't work yet.

Do not build them from your IDE or host environment yet

Build your codebase

Most templates and project frameworks are expected to run a build process. We haven't done that yet, so the dependencies are missing.. In some cases, maybe the bulk of your framework is still missing!

Build scripts are often very fussy about the versions of libraries available in the build environments. This is why we use containers in both ddev+Docker, and Upsun deployment environments. It's this aspect of development - the "configuration in code" that ensures that every version of every library and service is exactly the same in all environments, and helps to avoid discrepancies when testing.

We need to ssh in to the web container before running the build. Then the versions of the build tools used wil be specific to your project, and not to your client machine.

DDEV configuration

  • Set up ddev to share your Upsun token with the containers, as described in Upsun Docs.
    • (edit ~/.ddev/global_config.yaml)
  • You can bring forwards all of your preferred shell preferences by adding things like .bashrc to DDEV containers via ~/.ddev/homeadditions, see In-Container Shell Configuration
    • This can bring in your preferred .gitconfig and .gitignore , as well as everything else. See those docs for suggestions.

Workflow

A straightforwards development workflow would look something like this:

  • Get a working local copy of the latest state running.
  • Start a development branch and make changes locally.
    • During iteration, you may periodically refresh your local state by pulling the latest versions of code and content from production or staging.
    • Refreshing your local state from upstream will allow you to rehearse the release process - ensure that all the settings or changes that you need to have deployed will in fact be applied 
  • Push that branch into Upsun to produce a new preview environment for testing and sharing.
    • The feature branch in Upsun can also be easily updated with content from its parent as appropriate.
  • Eventually merge that branch into the Upsun staging environment and ensure integration testing works.
  • Merge from the tested staging into production to release.

 

Sync your local development environment with the main production state

  • Check out the most recent codebase for your project, into a DDEV container.

    upsun get $UPSUN_PROJECT
cd $UPSUN_PROJECT

    OR

    cd $UPSUN_PROJECT
git pull

  • It may help to ensure your ddev config is up to date, so you get the correct services.

    ddev restart

    Restarting project demo-upsun... 
Building project images...
.Project images built in 1s.
Network ddev-demo-upsun_default  Created
Container ddev-demo-upsun-db  Created
Container ddev-demo-upsun-web  Created
Container ddev-demo-upsun-web  Started
Container ddev-demo-upsun-db  Started
Starting Mutagen sync process...
Mutagen sync flush completed in 2s.
Waiting for containers to become ready: [web db]
Starting ddev-router if necessary...
Container ddev-router  Running
Waiting for additional project containers to become ready...
All project containers are now ready.
Restarted demo-upsun
Your project can be reached at http://demo-upsun.ddev.site http://127.0.0.1:33036

  • Run the build scripts that may be required to get the application running.

    ddev ssh
# upsun build

    By using 'upsun build' you are invoking the build process in the same way that the hosted environment does remotely. Often that may just be the same as running composer install, but if you use the CLI tool consistently, there will be less variations and surprises.

  • Note: "upsun build" May not always work!
    Instead you may do
    composer install
    or
    npm install
    etc as needed

  • Sync the latest content (database and files) down from the production site.

    ddev pull upsun

    You're about to delete the current database and files and replace with the results of a fresh pull.
Would you like to continue? [Y/n] (yes):
Authenticating...
Obtaining databases...
Creating gzipped SQL dump file: /var/www/html/.ddev/.downloads/db.sql.gz
Downloaded db dumps for databases 'db'
Importing databases [/Users/dman/www/client-test/.ddev/.downloads/db.sql.gz]

7.26MiB 0:00:01 [6.19MiB/s] [===========================================================================================================================================================>] 100%            
Obtaining files...
Importing files...
Importing files via custom files_import_command...
Pull succeeded.

if pull fails, check the setup instructions and ensure that the PLATFORM_PROJECT, PLATFORM_ENVIRONMENT & PLATFORMSH_CLI_TOKEN are present in the ddev environment (not just your development environment shell)

At this point you've got a fully-running copy of the real site. Be sure to check it out and test that your local copy now works.

Feature development and preview

  • Branch your local codebase to begin work on a fix or feature.

    git checkout -b feature-ZD12345

  • Develop locally

    ...

  • Push your changed code into a test environment on Upsun hosting and test it.

    git push --set-upstream upsun feature-ZD12345

    Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
Validating submodules
Validating configuration files
Processing activity: Dan Morrison pushed to feature-ZD12345
Found 8 commits
    No package to build: environment is inactive
To git.au.platform.sh:iii000bbbccc.git
* [new branch]      feature-ZD12345 -> feature-ZD12345
  branch 'feature-ZD12345' set up to track 'upsun/feature-ZD12345'.

    You are not expected to push database changes up directly. It is possible to do this during the first stages of a project, but not expected as a regular update action.

  • Creating a test environment for your feature branch will set it up with a copy of the data taken from the branch parent. So, optionally, set the parent for this environment BEFORE activating it, to ensure the initial data clone comes from the expected parent.

    upsun environment:info 'parent' 'staging'

  • Activate your new feature test environment

    upsun environment:activate

    Activating environment feature-ZD12345
Waiting for the activity: [xneswpecomrp2] Dan Morrison activated environment feature-ZD12345
  
Configuring resources
Using resources from environment 'staging'
Building application 'app' (runtime type: php:7.2, tree: 63c138f)
Reusing existing build for this tree ID
Provisioning certificates
Creating environment feature-ZD12345, as a clone of staging
Taking a temporary backup of the parent environment
Preparing to restore from a temporary backup
Starting environment based on the parent's temporary backup
Updating endpoints for app
Opening application app and its relationships
Executing deploy hook for application app
W: No database updates required                                           [success]
  
Opening environment
Deleting temporary backups (1)
Environment configuration
app (type: php:7.2, cpu: 0.5, memory: 224, disk: 512)
db (type: mariadb:10.2, cpu: 0.5, memory: 1408, disk: 512)
Environment routes
http://feature-zd12345-r4qb75i-iii000bbbccc.au.platformsh.site/ redirects to https://feature-zd12345-r4qb75i-iii000bbbccc.au.platformsh.site/
http://www.feature-zd12345-r4qb75i-iii000bbbccc.au.platformsh.site/ redirects to https://www.feature-zd12345-r4qb75i-iii000bbbccc.au.platformsh.site/
https://feature-zd12345-r4qb75i-iii000bbbccc.au.platformsh.site/ is served by application `app`
https://www.feature-zd12345-r4qb75i-iii000bbbccc.au.platformsh.site/ redirects to https://feature-zd12345-r4qb75i-iii000bbbccc.au.platformsh.site/
Blackfire build scheduled
[============================] 00:02:00 (complete)
The activity succeeded: [xneswpecomrp2] Dan Morrison activated environment feature-ZD12345

You can now preview the effect of deploying your code updates against an exact and up-to-date replica of the production site, in a production-like environment.

Iterate

It is common for time to pass between the beginning of a feature and the end of testing. It may take several iterations.

In this time, the main production site state may have changed, or other features and fixes may have been released. This is why integration testing is so important to the whole workflow.

The sync tools make it easy to get in the habit of regularly pulling the latest data state from your live project.

It is possible to fetch from the corresponding branch using the command:

ddev pull upsun

And that is useful to reset your copy to a previous stable state. But that will only reflect your current working branch.

Fetch latest data changes from the parent environment into your local development container

More helpful for resetting to test if your release will go smoothly, will be to fetch the latest data from the parent branch or production environment to overwrite the DB that you may have been manipulating during development. This way you can ensure that your changes will apply cleanly when released.

UPSUN_PARENT_ENVIRONMENT=$(upsun environment:info --project="${UPSUN_PROJECT}" --environment="${UPSUN_ENVIRONMENT}" parent)
upsun db:dump --yes --gzip --file=/var/www/html/.ddev/.downloads/db.sql.gz --project="${UPSUN_PROJECT}" --environment="${UPSUN_PARENT_ENVIRONMENT}"

Integration Testing

You are expected to have a staging branch environment of some sort, to use to test releases before pushing to production. It may sometimes be called test or even integration. Its purpose is to merge todays live data with your code that is waiting to be released, and confirm that all goes well.

You can do integration testing locally within DDEV, and then repeat on the Upsun staging environment.

  • Ensure that staging is using the same code as is in use on production.
  • Sync the current content and data from production into staging
    • staging is now a true clone of production.
  • Merge your feature branch into staging so that the latest code changes get applied
  • Execute and Note any extra steps (rebuilds, updates, cache-clears) that may be required to ensure the upgrade is complete.
  • Test and verify the behavior.

 

Alternative workflows

The Upsun hosting setup also supports you cloning dev or feature branches directly from the Upsun management UI or CLI. This also clones the parent environment data at that time. You would then check out the feature branch to your local dev and work forwards from there.

This original method is a little quicker, but it is incompatible with external GitHub or Gitlab integrations. As many development agencies are now using GitHub as the primary repository, then it becomes less appropriate to attempt branching from within the Upsun repository directly.

Making changes and keeping up to date.

Variations of the above steps can be performed by team members in various orders, depending on what makes the most sense for your team and your tasks. A major change that requires database schema changes, like some library version upgrades, should be tested well. A front-end-only change probably doesn't need to be so concerned about having the very latest copy of the production database every day.

Summary

  • There are a few extra setup steps you can do with DDEV to ensure it performs well in cooperation with your IDE, and forms a good relationship with your upstream project to keep well synchronized.
  • When using DDEV, it's easy to regularly use ddev pull upsun to fetch the latest copy of the data from the upstream environment that you are working on. The code should still be synced using git methods.
  • When working in a team, it's important to keep the staging environment as a close mirror of production, so it's a useful place to perform integration testing.

Related to

Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.