Get data with the Journald input

The journald input is a modular input that collects logs that the Linux journald system logging process produces on 64-bit Intel x86 (x86-64) chipset architecture, ARMv6 architecture and ARMv8 architecture systems. The journald process runs on computers that run versions of the Linux operating system that use systemd as the system management service. The systemd-journald.service writes journal entries to the journald database, and they are read with the journalctl utility.

Your Splunk platform deployment accepts journald data from universal forwarders that capture the data and send it to other parts of your Splunk platform deployment.

How the journald reader works

The journald input takes its configurations from defined stanzas in the inputs.conf file. Each defined stanza contains the settings that define what data your Splunk Universal Forwarder collects. The number of defined stanzas determine the number of journalctl processes that run. For example, if you define five stanzas, the journalctl utility runs five times. If you configure five identical defined stanzas, the journalctl utility reports identical data to the Splunk software five times.

Every setting you define in a journald stanza is a filter for your data. If you enable and configure without parameters, it ingests the full contents of the journal, starting with the oldest entry. To refine your data ingestion, define filters. Journald configuration supports both prescriptive and restrictive declaration of data filters. If one or more FIELD=VALUE match arguments are passed, the output is filtered accordingly. For example, _SYSTEMD_UNIT=httpd.service refers to the components of a structured journal entry.

Version and field compatibility

The journald input collector uses the systemd utility journalctl. The older versions (versions 236 and lower) of journalctl do not support the journalctl-include-fields configuration parameter.

To see the version of journactl, enter the following line into your command line interface (CLI):

journalctl --version

Journalctl version	Supported fields
Versions 236 and higher	journalctl-include-fields, journalctl-exclude-fields, journalctl-filter, journalctl-grep, journalctl-user-unit
Versions 236 and lower.	journalctl-exclude-fields, journalctl-filter, journalctl-grep, journalctl-user-unit

Configure the journald input

Configure the journald input on a supported Splunk Enterprise Linux Universal Forwarder to ingest journald logs into your Splunk platform deployment. Use .conf files to make your input configurations.

1. On your Splunk Universal Forwarder, navigate to splunkforwarder/etc/apps/journald_input/default/.
2. Copy the inputs.conf file.
3. Navigate to splunkforwarder/etc/apps/journald_input/local/.
4. Paste the copy of the inputs.conf file.
5. Open the inputs.conf file with a text editor.
6. Navigate to the journald stanza, and configure the stanza with the parameters that include or exclude the data fields that you want. See the following example:

   [journald://my-stanza]
   # The following fields are included by default, and required: MESSAGE, _REALTIME_TIMESTAMP, _CURSOR
   journalctl-include-fields = PRIORITY,CMD,EXE
   journalctl-exclude-fields =
   journalctl-filter =_SYSTEMD_UNIT=my.service,_PID=232+_SYSTEMD_UNIT=sshd
   journalctl-grep =^WARN.*disk,.*errno=\d+\S+restarting
   journalctl-user-unit =unit1,unit2
   

   For a full list of parameters, see the Reference section in this topic.

7. Verify that your field configurations begin with underscores. The journald input stanza configuration requires underscores
8. Save your changes.
9. Restart your Splunk Universal Forwarder.
10. (Optional) Use a deployment server to push the changes to your settings to other forwarders in your Splunk platform deployment. For more information, see Use forwarder management to manage apps topic in the Updating Splunk Enterprise Instances manual.

Reference

For a full list of available parameters, see the following table:

Parameter	Description
journalctl-include-fields = <string>	Filter which fields to send to the Splunk platform instance.
journalctl-exclude-fields = <string>	Filter which fields to not send to the Splunk platform instance.
journalctl-filter = <string>	Filters using the matches concept in journalctl. For example, _SYSTEMD_UNIT=avahi-daemon.service _PID=28097 + _SYSTEMD_UNIT=dbus.service will show all messages from the Avahi service process with a PID of 28097, plus all messages from the D-Bus service.
journalctl-unit = <string>	Show messages for the specified systemd unit.
journalctl-identifier = <string>	Show messages for the specified syslog identifier SYSLOG_IDENTIFIER.
journalctl-priority = <string>	Filter output by message priorities or priority ranges.
journalctl-boot = <string>	Messages from a specific boot.
journalctl-facility = <string>	Filter output by syslog facility.
journalctl-grep = <string>	Filter output to entries where the MESSAGE= field matches the specified regular expression.
journalctl-user-unit = <string>	Show messages for the specified user session unit.
journalctl-dmesg = <string>	Show only kernel messages.
journalctl-quiet = <string>	Suppresses all informational messages.
journalctl-freetext = <string>	Reserved for future use.

Troubleshoot the journald input

Best practices for the journald input

• Check for errors in your splunkd.log file.
• Start with a simple configuration before you build something more complex.
• For more information on configurations, see the spec file for this product.

Verify data

To verify what your journald input is collecting, enter the following into your command line interface (CLI):

ps aux | grep journalctl

.

You need to define at least one stanza in your inputs.conf for data to be collected.

Data duplication

When configuring journald, Each stanza will run a journal reader. Duplicate stanza definitions will result in data duplication.

Data collection efficiency

The include-fields is more efficient than using exclude-fields, include-fields filtering happens at the journald level, instead of the Splunk Universal Forwarder level.

Fields not available

Review your journalctl version. The include-fields and grep are not available on earlier versions. Deployment server scenarios where you have some machines using newer journalctl while some machines user the older version. The Splunk deployment server will push the same configurations to the new and older machines, and the older machines will fail to collect data.

Troubleshoot missing field options

The journald input uses an externally-developed utility to function. Because of this, older versions of journalctl might not have all configuration options available.  The following options might be missing in CentOS 7 filtering configurations:

The journalct-include-fields option lets you declare which journald fields you want to retrieve.  Without this option, you can only specify which fields you do not want to include.  If, for example, you want to retrieve only the MESSAGE field, you must list every field you do not want your Splunk platform instance to ingest. This option might generate multiple lengthy data fields.
The grep option lets you apply a regular expression filter before data reaches the indexer.

journald logs unable to open cursor file for the first start on the Splunk universal forwarder

If, when starting the journald modular input for the first time on the universal forwarder, you receive the following journald log error:

09-04-2020 03:50:49.221 +0000 ERROR ExecProcessor - message from "/home/ec2-user/splunkforwarder/bin/splunkd journald-modinput '$@'" journald-modinput - unable to open cursor file /home/ec2-user/splunkforwarder/var/lib/splunk/modinputs/journald/s1.checkpoint for reading (No such file or directory) 
09-04-2020 03:50:49.221 +0000 INFO  ExecProcessor - message from "/home/ec2-user/splunkforwarder/bin/splunkd journald-modinput '$@'"  journald-modinput -  starting (mode = Data Collection) Config -  serverHost=ip-172-31-49-163.us-west-2.compute.internal serverUri=https://127.0.0.1:8089 checkpointDir=/home/ec2-user/splunkforwarder/var/lib/splunk/modinputs/journald stanza=journald://s1 PropertiesMap: {host -> '$decideOnStartup' index -> 'default' interval -> '30' journalctl-exclude-fields -> '__MONOTONIC_TIMESTAMP,__SOURCE_REALTIME_TIMESTAMP' journalctl-include-fields -> 'PRIORITY,_SYSTEMD_UNIT,_SYSTEMD_CGROUP,_TRANSPORT,_PID,_UID,_MACHINE_ID,_GID,_COMM,_EXE' journalctl-quiet -> 'true'}

The cursor file, which is where the Splunk software keeps track of the data that has been ingested from the journal, might not be available for the first time. It will not affect journald input functionality. The data injection is normal. It generates this error log response.

Journald ID missing checkpoint file error

If you run the journald input the first, and receive the following error:

10-03-2020 18:29:35.241 -0600 ERROR ExecProcessor - message from "/opt/splunkforwarder/bin/splunkd journald-modinput '$@'" journald-modinput - unable to open cursor file /opt/splunkforwarder/var/lib/splunk/modinputs/journald/all.checkpoint for reading (No such file or directory)

No additional actions need to be taken. This is a checkpoint file that contains the journald ID of the last record the Splunk software already read. If you need to restart reading, the Splunk software restarts after the ID in this error.

When the journald reader starts, it first checks if the Splunk software needs to resume reading after a previous stop. The first time this process runs, no checkpoint file was created, so the reader will fail to find the checkpoint file, and will write the error to the log.