Get data with the Journald input

The journald input is a modular input that collects logs that Linux journald system logging produces for:

• 64-bit Intel x86 (x86-64) chipsets
• Linux on ARM architecture machines.

The journald process runs on versions of the Linux operating system that use systemd as the system management service. The systemd-journald.service writes journal entries to the journald database, and they are read with the journalctl utility.

Your Splunk platform deployment accepts journald data from universal forwarders that capture the data and send it to other parts of your Splunk platform deployment. Both Splunk Cloud Platform and Splunk Enterprise accept journald data.

How the journald reader works

The journald input takes its configurations from defined stanzas in the inputs.conf file. Each defined stanza contains the settings that define what data your Splunk Universal Forwarder collects. The number of defined stanzas determine the number of journalctl processes that run. For example, if you define five stanzas, the journalctl utility runs five times. If you configure five identical defined stanzas, the journalctl utility reports identical data to the Splunk software five times.

Every setting you define in a journald stanza is a filter for your data. If you do not configure filters, journald ingests the full contents of the journal, starting with the oldest entry. To avoid this, you define filters to refine your data ingestion. Journald configuration supports both prescriptive and restrictive filters. If one or more FIELD=VALUE match arguments are passed, the output is filtered accordingly. For example, _SYSTEMD_UNIT=httpd.service refers to the components of a structured journal entry.

Version and field compatibility

The journald input collector uses the systemd utility journalctl. The older versions (versions 236 and lower) of journalctl do not support the journalctl-include-fields configuration parameter.

To determine your version of journalctl, enter the following line into your command line interface (CLI):

journalctl --version

Journalctl version	Supported fields
Versions 236 and higher	journalctl-include-fields, journalctl-exclude-fields, journalctl-filter, journalctl-grep, journalctl-user-unit
Versions 236 and lower.	journalctl-exclude-fields, journalctl-filter, journalctl-grep, journalctl-user-unit

Configure the journald input

Use the inputs.conf file to configure your input.

1. On your Splunk Universal Forwarder, navigate to splunkforwarder/etc/apps/journald_input/default/.
2. Copy the inputs.conf file.
3. Navigate to splunkforwarder/etc/apps/journald_input/local/.
4. Paste the copy of the inputs.conf file.
5. Open the inputs.conf file with a text editor.
6. Navigate to the journald stanza, and configure the stanza with the parameters that include or exclude the data fields that you want.

   [journald://my-stanza]
   # The following fields are included by default, and required: MESSAGE, _REALTIME_TIMESTAMP, _CURSOR
   journalctl-include-fields = PRIORITY,CMD,EXE
   journalctl-exclude-fields =
   journalctl-filter =_SYSTEMD_UNIT=my.service,_PID=232+_SYSTEMD_UNIT=sshd
   journalctl-grep =^WARN.*disk,.*errno=\d+\S+restarting
   journalctl-user-unit =unit1,unit2
   

   If you are running old instances of systemd, you must define fields in the stanza in the new installation and also clear that data from the older versions.

   For a full list of parameters, see the Reference section in this topic.

7. Verify that your field configurations begin with underscores. The journald input stanza configuration requires underscores
8. Save your changes.
9. Restart your Splunk Universal Forwarder.
10. (Optional) Use a deployment server to push the changes to your settings to other forwarders in your Splunk platform deployment. For more information, see Use forwarder management to manage apps topic in the Updating Splunk Enterprise Instances manual.

Reference

For a full list of available parameters, see the following table:

Parameter	Description
journalctl-include-fields = <string>	Filter which fields to send to the Splunk platform instance.
journalctl-exclude-fields = <string>	Restrict the fields that sent to the Splunk platform instance.
journalctl-filter = <string>	Filters fields using the matches concept in journalctl. For example, _SYSTEMD_UNIT=avahi-daemon.service _PID=28097 + _SYSTEMD_UNIT=dbus.service will show all messages from the Avahi service process with a PID of 28097, plus all messages from the D-Bus service.
journalctl-unit = <string>	Show messages for the specified systemd unit.
journalctl-identifier = <string>	Show messages for the specified syslog identifier SYSLOG_IDENTIFIER.
journalctl-priority = <string>	Filter output by message priorities or priority ranges.
journalctl-boot = <string>	Messages from a specific boot.
journalctl-facility = <string>	Filter output by syslog facility.
journalctl-grep = <string>	Filter output to entries where the MESSAGE= field matches the specified regular expression.
journalctl-user-unit = <string>	Show messages for the specified user session unit.
journalctl-dmesg = <string>	Show only kernel messages.
journalctl-quiet = <string>	Suppresses all informational messages.
journalctl-freetext = <string>	Reserved for future use.

Troubleshoot the journald input

Best practices for the journald input

• Check for errors in your splunkd.log file.
• Start with a simple configuration before you build something more complex.
• For more information on configurations, see the spec file for this product.

Verify data

To verify the data that your journald input is collecting, enter the following into your command line interface (CLI):

ps aux | grep journalctl

.

You need to define at least one stanza in your inputs.conf for data to be collected.

Data duplication

When configuring journald, Each stanza will run a journal reader. Duplicate stanza definitions will result in data duplication.

Data collection efficiency

The include-fields is more efficient than using exclude-fields, include-fields filtering happens at the journald level, instead of the Splunk Universal Forwarder level.

Fields are not available

Review your journalctl version; the include-fields and grep are not available on earlier versions.

For deployment server scenarios where some machines use a newer version of journalctl while some machines user the older version, the Splunk deployment server will push the same configurations to the new and older machines, and the older machines will fail to collect data.

Troubleshoot missing field options

The journald input uses an externally-developed utility to function. Because of this, older versions of journalctl might not have all configuration options available.  The following options might be missing in CentOS 7 filtering configurations:

• The journalct-include-fields option lets you declare which journald fields you want to retrieve. Without this option, you can only specify which fields you do not want to include.  If, for example, you want to retrieve only the MESSAGE field, you must list every field you do not want your Splunk platform instance to ingest. This method can generate multiple lengthy data fields.

• The grep option lets you apply a regular expression filter before data reaches the indexer.

journald logs unable to open cursor file for the first start on the Splunk Universal Forwarder

If, when starting the journald modular input for the first time on the universal forwarder, you receive the following journald log error:

09-04-2020 03:50:49.221 +0000 ERROR ExecProcessor - message from "/home/ec2-user/splunkforwarder/bin/splunkd journald-modinput '$@'" journald-modinput - unable to open cursor file /home/ec2-user/splunkforwarder/var/lib/splunk/modinputs/journald/s1.checkpoint for reading (No such file or directory) 
09-04-2020 03:50:49.221 +0000 INFO  ExecProcessor - message from "/home/ec2-user/splunkforwarder/bin/splunkd journald-modinput '$@'"  journald-modinput -  starting (mode = Data Collection) Config -  serverHost=ip-172-31-49-163.us-west-2.compute.internal serverUri=https://127.0.0.1:8089 checkpointDir=/home/ec2-user/splunkforwarder/var/lib/splunk/modinputs/journald stanza=journald://s1 PropertiesMap: {host -> '$decideOnStartup' index -> 'default' interval -> '30' journalctl-exclude-fields -> '__MONOTONIC_TIMESTAMP,__SOURCE_REALTIME_TIMESTAMP' journalctl-include-fields -> 'PRIORITY,_SYSTEMD_UNIT,_SYSTEMD_CGROUP,_TRANSPORT,_PID,_UID,_MACHINE_ID,_GID,_COMM,_EXE' journalctl-quiet -> 'true'}

The cursor file might not be available the first time you run journald. This is where the Splunk software keeps track of the data that has been ingested from the journal. This does not affect journald input functionality. The data injection is normal and generates this error log response.

Journald ID is missing checkpoint file error

If you run the journald input for the first time and receive the following error:

10-03-2020 18:29:35.241 -0600 ERROR ExecProcessor - message from "/opt/splunkforwarder/bin/splunkd journald-modinput '$@'" journald-modinput - unable to open cursor file /opt/splunkforwarder/var/lib/splunk/modinputs/journald/all.checkpoint for reading (No such file or directory)

When the journald reader starts, it first checks whether the Splunk software needs to resume reading after a previous stop. The first time this process runs, no checkpoint file exists, so the journalid reader will fail to find the checkpoint file, and will write the error to the log.

No additional actions need to be taken, the Splunk software restarts after the ID in this error.