Articles in this section

How can I forward my logs to Splunk?

Splunk recommended forwarder, Filebeat, and the Splunk Universal Forwarder expect a file system with write access to ship logs to their indexers. Upsun by design is set-up to host applications on a read-only file system, so it is necessary to utilize mounts to provide these services with write access.

Installation, configuration, and shipping are explained briefly below for Splunk, but more detailed How-tos are also available for more in-depth instruction:

Forwarding to Splunk

1. Define mounts

Define a pair mount in .upsun.app.yaml that the Forwarder can write to when it ships your logs. It will use splunk to write modified log files before they ship and .splunk to write authorization files so that it can remember configurations and credentials after it connects to your indexer.

# .upsun.app.yaml

mounts:
    'splunk':
        source: local
        source_path: splunk
    '/.splunk':
        source: local
        source_path: splauths

Same as above, since mounts aren’t available during build we can install the Universal Forwarder in a sub directory called config/filebeat where we can store installation scripts and temporary builds.

2. Modify build and deploy hooks

# .upsun.app.yaml

hooks:
    build: |
        if [ ! -z $SPLUNK_CONFIG ]; then
            ./config/splunk/scripts/install.sh
        fi
        pipenv install --system --deploy
    deploy: |
        if [ ! "$(ls -A splunk)" ]; then
             ./config/splunk/scripts/config.sh
        fi
        ./splunk/splunkforwarder/bin/splunk restart

In the build hook, the Forwarder will be installed in the temporary directory config/splunk/build, so long as a project level variable that denotes a completed configuration has not been set.

# config/splunk/scripts/install.sh

#!/usr/bin/env bash

TEMP_SPLUNK_HOME=config/splunk/build

# Install Splunk Universal Forwarder
[ ! -d $TEMP_SPLUNK_HOME ] && mkdir -p $TEMP_SPLUNK_HOME
cd $TEMP_SPLUNK_HOME
wget -O splunkforwarder-7.2.5.1-962d9a8e1586-Linux-x86_64.tgz 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.2.5.1&product=universalforwarder&filename=splunkforwarder-7.2.5.1-962d9a8e1586-Linux-x86_64.tgz&wget=true'
tar xvzf splunkforwarder-7.2.5.1-962d9a8e1586-Linux-x86_64.tgz
rm splunkforwarder-7.2.5.1-962d9a8e1586-Linux-x86_64.tgz

In the deploy hook,

# config/splunk/scripts/config.sh

#!/usr/bin/env bash

cd $PLATFORM_APP_DIR

TEMP_SPLUNK_HOME=config/splunk/build/*
SPLUNK_HOME=$PLATFORM_APP_DIR/splunk/splunkforwarder

# Copy temp build to writable storage
cp -v -r $TEMP_SPLUNK_HOME splunk

# Migrate used-seed.conf to the forwarder
cp -v config/splunk/seeds/user.conf $SPLUNK_HOME/etc/system/local/user-seed.conf

# Start Splunk for the first time, accepting license
./splunk/splunkforwarder/bin/splunk start --accept-license

# Update outputs.conf with receiver address seed
cp -v $PLATFORM_APP_DIR/config/splunk/seeds/outputs.conf $PLATFORM_APP_DIR/splunk/splunkforwarder/etc/system/local/outputs.conf

# Update inputs.conf with monitor inputs seed
cp -v $PLATFORM_APP_DIR/config/splunk/seeds/inputs.conf $PLATFORM_APP_DIR/splunk/splunkforwarder/etc/system/local/inputs.conf

3. Create configuration files

Splunk has a dedicated CLI that allows you to configure inputs and outputs, but it recommends creating a set of configuration seed files instead.

  • user.conf: The Forwarder comes with a default admin user with the password changeme. You will need to create this file to update the password or else remote access will not be available. Once it has been moved, you can start the Forwarder for the first time and modify the remaining configurations.

    # config/splunk/seeds/user.conf
    [user_info]
    USERNAME = admin
    PASSWORD = testpass
    
  • outputs.conf: This file configures information about the Splunk Receiver/Indexer. Replace <receiver ip> with the IP address of the Indexer. If you have changed the default listening port for the Indexer from 9997, you will need to change that as well.

    # config/splunk/seeds/outputs.conf
    [tcpout] defaultGroup=default 
    
    [tcpout:default] server=<receiver ip>:9997 
    
    [tcpout-server:<receiver ip>:9997]
    
  • inputs.conf: Finally, configure the forward to monitor all files in /var/log, the location of Upsun log files.

    # config/splunk/seeds/inputs.conf
    [monitor://var/log/]
    disabled = false

Each of these seed files can be placed in config/splunk/seeds in your project directory, and each is moved to the Forwarder’s final installation location during the deploy hook.

4. Complete

These configuration settings will install and configure the Splunk Universal Forwarder to ship Upsun logs on every deployment with the last line of the deploy hook: ./splunk/splunkforwarder/bin/splunk restart.

If you keep the build hook as written, a new install will occur with each deployment unless you set a project level variable that designates that Splunk is already configured:

$ Upsun variable:create --level project --name SPLUNK_CONFIG --value 'true'
Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.