How to monitor long running scripts with pv

Avatar
Matthias Van Woensel

Goal

Sometimes you want to run a long running process without feedback. PV is a tool that allows you to do just that.

Example

  • Running a mysqldump
    mysqldump <yourparams> | pv --progress --size 100m > /tmp/dump.sql
  • Importing a mysql dump
    pv dump.sql | mysql <yourparams>
  • Copying a large file
    pv /originallocation/largefile.bin > /otherlocation/largefile.bin
  • Create a zip file
    pv largefile.txt | zip > largefile.zip
  • Create a tar archive
    tar -czf - ./foldertotar | (pv -p --timer --rate --bytes > backup.tgz)

Problem

Sounds cool, but pv isn’t available by default.

Solution

Luckily, there is an easy solution.

Simply add the pv package in your .platform.app.yaml file. This will automatically install pv. So you can use it.

dependencies:
  nodejs: # Specify one NPM package per line.
    pv: '~1.0.1'

Note: you can do this in any container type. PHP, nodejs, golang, …

This works because there is a nodejs package that does the heavy lifting for us.

For more information on dependencies, check our documentation

0

Comments

2 comments
Date Votes
  • Avatar
    Platform.sh DevRel

    Hi @matthias-van-woensel,

    I’d like to import a large db, something like pv latest--dump.sql | $(drush sql-connect)

    Seems to have installed correctly as pv -v displays v1.0.1 and (as @travis.raup pointed out) cat /dev/urandom | pv --size 100MB > /dev/null also seems to work.

    However, most things don’t - including the pv --help command:
    Screen Shot 2020-11-16 at 9.22.27 AM

    They just get stuck (as per screenshot above). In the case of db import commands wrapped with PV - they still do their job, they just don’t update the cli (i.e. the pv part doesn’t work)

    Opened a support ticket too, but figured you might know what to try? :slight_smile:

    Thanks!
    Julian

    0
  • Avatar
    Platform.sh DevRel

    Sharing the results of my support request here:

    William David Friday at 14:51

    Hi Julian,

    I set up pv in my own environment and replicated the behavior you saw. Taking a look at the code it looks like the version of pv that’s installed with node differs a bit from the version you might get if someone where to install it as a linux package. For instance, the “help” option just seems to not be built into this version.

    In terms of monitoring database imports, it seems to work just by cat ing the file through it rather than invoking it directly, e.g. something like:

    web@app.0:/tmp$ cat dump.sql | pv | mysql -h database.internal -u main -p main
Enter password: 
 100.00% | 0 ETA | 7.5MB Transferred | 1.2MB/s

    Try that out and see if it helps?

    Regards,
    -Bill

    Edit: confirmed that it works! Thank you, William David!

    0

Please sign in to leave a comment.

 

Didn't find what you were looking for?

New post