Splunk recommended forwarder, Filebeat, and the Splunk Universal Forwarder expect a file system with write access to ship logs to their indexers. Upsun by design is set-up to host applications on a read-only file system, so it is necessary to utilize mounts to provide these services with write access.
Installation, configuration, and shipping are explained briefly below for Splunk, but more detailed How-tos are also available for more in-depth instruction:
Forwarding to Splunk
1. Define mounts
Define a pair mount in .upsun.app.yaml that the Forwarder can write to when it ships your logs. It will use splunk to write modified log files before they ship and .splunk to write authorization files so that it can remember configurations and credentials after it connects to your indexer.
# .upsun.app.yaml
mounts:
'splunk':
source: local
source_path: splunk
'/.splunk':
source: local
source_path: splauths
Same as above, since mounts aren’t available during build we can install the Universal Forwarder in a sub directory called config/filebeat where we can store installation scripts and temporary builds.
2. Modify build and deploy hooks
# .upsun.app.yaml
hooks:
build: |
if [ ! -z $SPLUNK_CONFIG ]; then
./config/splunk/scripts/install.sh
fi
pipenv install --system --deploy
deploy: |
if [ ! "$(ls -A splunk)" ]; then
./config/splunk/scripts/config.sh
fi
./splunk/splunkforwarder/bin/splunk restart
In the build hook, the Forwarder will be installed in the temporary directory config/splunk/build, so long as a project level variable that denotes a completed configuration has not been set.
# config/splunk/scripts/install.sh #!/usr/bin/env bash TEMP_SPLUNK_HOME=config/splunk/build # Install Splunk Universal Forwarder [ ! -d $TEMP_SPLUNK_HOME ] && mkdir -p $TEMP_SPLUNK_HOME cd $TEMP_SPLUNK_HOME wget -O splunkforwarder-7.2.5.1-962d9a8e1586-Linux-x86_64.tgz 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.2.5.1&product=universalforwarder&filename=splunkforwarder-7.2.5.1-962d9a8e1586-Linux-x86_64.tgz&wget=true' tar xvzf splunkforwarder-7.2.5.1-962d9a8e1586-Linux-x86_64.tgz rm splunkforwarder-7.2.5.1-962d9a8e1586-Linux-x86_64.tgz
In the deploy hook,
# config/splunk/scripts/config.sh #!/usr/bin/env bash cd $PLATFORM_APP_DIR TEMP_SPLUNK_HOME=config/splunk/build/* SPLUNK_HOME=$PLATFORM_APP_DIR/splunk/splunkforwarder # Copy temp build to writable storage cp -v -r $TEMP_SPLUNK_HOME splunk # Migrate used-seed.conf to the forwarder cp -v config/splunk/seeds/user.conf $SPLUNK_HOME/etc/system/local/user-seed.conf # Start Splunk for the first time, accepting license ./splunk/splunkforwarder/bin/splunk start --accept-license # Update outputs.conf with receiver address seed cp -v $PLATFORM_APP_DIR/config/splunk/seeds/outputs.conf $PLATFORM_APP_DIR/splunk/splunkforwarder/etc/system/local/outputs.conf # Update inputs.conf with monitor inputs seed cp -v $PLATFORM_APP_DIR/config/splunk/seeds/inputs.conf $PLATFORM_APP_DIR/splunk/splunkforwarder/etc/system/local/inputs.conf
3. Create configuration files
Splunk has a dedicated CLI that allows you to configure inputs and outputs, but it recommends creating a set of configuration seed files instead.
-
user.conf: The Forwarder comes with a defaultadminuser with the passwordchangeme. You will need to create this file to update the password or else remote access will not be available. Once it has been moved, you can start the Forwarder for the first time and modify the remaining configurations.# config/splunk/seeds/user.conf [user_info] USERNAME = admin PASSWORD = testpass
-
outputs.conf: This file configures information about the Splunk Receiver/Indexer. Replace<receiver ip>with the IP address of the Indexer. If you have changed the default listening port for the Indexer from9997, you will need to change that as well.# config/splunk/seeds/outputs.conf [tcpout] defaultGroup=default [tcpout:default] server=<receiver ip>:9997 [tcpout-server:<receiver ip>:9997]
-
inputs.conf: Finally, configure the forward to monitor all files in/var/log, the location of Upsun log files.# config/splunk/seeds/inputs.conf [monitor://var/log/] disabled = false
Each of these seed files can be placed in config/splunk/seeds in your project directory, and each is moved to the Forwarder’s final installation location during the deploy hook.
4. Complete
These configuration settings will install and configure the Splunk Universal Forwarder to ship Upsun logs on every deployment with the last line of the deploy hook: ./splunk/splunkforwarder/bin/splunk restart.
If you keep the build hook as written, a new install will occur with each deployment unless you set a project level variable that designates that Splunk is already configured:
$ Upsun variable:create --level project --name SPLUNK_CONFIG --value 'true'
Comments
Please sign in to leave a comment.