Get data with the Journald input
The journald input is a modular input that collects logs that Linux journald system logging produces for:
- 64-bit Intel x86 (x86-64) chipsets
- Linux on ARM architecture machines.
The journald process runs on versions of the Linux operating system that use systemd as the system management service. The systemd-journald.service
writes journal entries to the journald database, and they are read with the journalctl utility.
Your Splunk platform deployment accepts journald data from universal forwarders that capture the data and send it to other parts of your Splunk platform deployment. Both Splunk Cloud Platform and Splunk Enterprise accept journald data.
How the journald reader works
The journald input takes its configurations from defined stanzas in the inputs.conf
file. Each defined stanza contains the settings that define what data your Splunk Universal Forwarder collects. The number of defined stanzas determine the number of journalctl processes that run. For example, if you define five stanzas, the journalctl utility runs five times. If you configure five identical defined stanzas, the journalctl utility reports identical data to the Splunk software five times.
Every setting you define in a journald stanza is a filter for your data. If you do not configure filters, journald ingests the full contents of the journal, starting with the oldest entry. To avoid this, you define filters to refine your data ingestion. Journald configuration supports both prescriptive and restrictive filters. If one or more FIELD=VALUE
match arguments are passed, the output is filtered accordingly. For example, _SYSTEMD_UNIT=httpd.service
refers to the components of a structured journal entry.
Version and field compatibility
The journald input collector uses the systemd utility journalctl
. The older versions (versions 236 and lower) of journalctl do not support the journalctl-include-fields
configuration parameter.
To determine your version of journalctl, enter the following line into your command line interface (CLI):
journalctl --version
Journalctl version | Supported fields |
---|---|
Versions 236 and higher | journalctl-include-fields , journalctl-exclude-fields , journalctl-filter , journalctl-grep , journalctl-user-unit
|
Versions 236 and lower. | journalctl-exclude-fields , journalctl-filter , journalctl-grep , journalctl-user-unit
|
Configure the journald input
Use the inputs.conf file to configure your input.
- On your Splunk Universal Forwarder, navigate to
splunkforwarder/etc/apps/journald_input/default/
. - Copy the
inputs.conf
file. - Navigate to
splunkforwarder/etc/apps/journald_input/local/
. - Paste the copy of the
inputs.conf
file. - Open the
inputs.conf
file with a text editor. - Navigate to the
journald
stanza, and configure the stanza with the parameters that include or exclude the data fields that you want.[journald://my-stanza] # The following fields are included by default, and required: MESSAGE, _REALTIME_TIMESTAMP, _CURSOR journalctl-include-fields = PRIORITY,CMD,EXE journalctl-exclude-fields = journalctl-filter =_SYSTEMD_UNIT=my.service,_PID=232+_SYSTEMD_UNIT=sshd journalctl-grep =^WARN.*disk,.*errno=\d+\S+restarting journalctl-user-unit =unit1,unit2
If you are running old instances of systemd, you must define fields in the
stanza in the new installation and also clear that data from the older versions. For a full list of parameters, see the Reference section in this topic.
- Verify that your field configurations begin with underscores. The journald input stanza configuration requires underscores
- Save your changes.
- Restart your Splunk Universal Forwarder.
- (Optional) Use a deployment server to push the changes to your settings to other forwarders in your Splunk platform deployment. For more information, see Use forwarder management to manage apps topic in the Updating Splunk Enterprise Instances manual.
Reference
For a full list of available parameters, see the following table:
Parameter | Description |
---|---|
journalctl-include-fields = <string>
|
Filter which fields to send to the Splunk platform instance. |
journalctl-exclude-fields = <string>
|
Restrict the fields that sent to the Splunk platform instance. |
journalctl-filter = <string>
|
Filters fields using the matches concept in journalctl . For example, _SYSTEMD_UNIT=avahi-daemon.service _PID=28097 + _SYSTEMD_UNIT=dbus.servicewill show all messages from the Avahi service process with a PID of 28097, plus all messages from the D-Bus service. |
journalctl-unit = <string>
|
Show messages for the specified systemd unit. |
journalctl-identifier = <string>
|
Show messages for the specified syslog identifier SYSLOG_IDENTIFIER .
|
journalctl-priority = <string>
|
Filter output by message priorities or priority ranges. |
journalctl-boot = <string>
|
Messages from a specific boot. |
journalctl-facility = <string>
|
Filter output by syslog facility. |
journalctl-grep = <string>
|
Filter output to entries where the MESSAGE= field matches the specified regular expression.
|
journalctl-user-unit = <string>
|
Show messages for the specified user session unit. |
journalctl-dmesg = <string>
|
Show only kernel messages. |
journalctl-quiet = <string>
|
Suppresses all informational messages. |
journalctl-freetext = <string>
|
Reserved for future use. |
Troubleshoot the journald input
Best practices for the journald input
- Check for errors in your
splunkd.log
file. - Start with a simple configuration before you build something more complex.
- For more information on configurations, see the spec file for this product.
Verify data
To verify the data that your journald input is collecting, enter the following into your command line interface (CLI):
ps aux | grep journalctl
.
You need to define at least one stanza in your inputs.conf
for data to be collected.
Data duplication
When configuring journald, Each stanza will run a journal reader. Duplicate stanza definitions will result in data duplication.
Data collection efficiency
The include-fields
is more efficient than using exclude-fields
, include-fields
filtering happens at the journald level, instead of the Splunk Universal Forwarder level.
Fields are not available
Review your journalctl
version; the include-fields
and grep
are not available on earlier versions.
For deployment server scenarios where some machines use a newer version of journalctl
while some machines user the older version, the Splunk deployment server will push the same configurations to the new and older machines, and the older machines will fail to collect data.
Troubleshoot missing field options
The journald input uses an externally-developed utility to function. Because of this, older versions of journalctl
might not have all configuration options available. The following options might be missing in CentOS 7 filtering configurations:
- The
journalct-include-fields
option lets you declare which journald fields you want to retrieve. Without this option, you can only specify which fields you do not want to include. If, for example, you want to retrieve only theMESSAGE
field, you must list every field you do not want your Splunk platform instance to ingest. This method can generate multiple lengthy data fields.
- The
grep
option lets you apply a regular expression filter before data reaches the indexer.
journald logs unable to open cursor file for the first start on the Splunk Universal Forwarder
If, when starting the journald modular input for the first time on the universal forwarder, you receive the following journald log error:
09-04-2020 03:50:49.221 +0000 ERROR ExecProcessor - message from "/home/ec2-user/splunkforwarder/bin/splunkd journald-modinput '$@'" journald-modinput - unable to open cursor file /home/ec2-user/splunkforwarder/var/lib/splunk/modinputs/journald/s1.checkpoint for reading (No such file or directory) 09-04-2020 03:50:49.221 +0000 INFO ExecProcessor - message from "/home/ec2-user/splunkforwarder/bin/splunkd journald-modinput '$@'" journald-modinput - starting (mode = Data Collection) Config - serverHost=ip-172-31-49-163.us-west-2.compute.internal serverUri=https://127.0.0.1:8089 checkpointDir=/home/ec2-user/splunkforwarder/var/lib/splunk/modinputs/journald stanza=journald://s1 PropertiesMap: {host -> '$decideOnStartup' index -> 'default' interval -> '30' journalctl-exclude-fields -> '__MONOTONIC_TIMESTAMP,__SOURCE_REALTIME_TIMESTAMP' journalctl-include-fields -> 'PRIORITY,_SYSTEMD_UNIT,_SYSTEMD_CGROUP,_TRANSPORT,_PID,_UID,_MACHINE_ID,_GID,_COMM,_EXE' journalctl-quiet -> 'true'}
The cursor file might not be available the first time you run journald. This is where the Splunk software keeps track of the data that has been ingested from the journal. This does not affect journald input functionality. The data injection is normal and generates this error log response.
Journald ID is missing checkpoint file error
If you run the journald input for the first time and receive the following error:
10-03-2020 18:29:35.241 -0600 ERROR ExecProcessor - message from "/opt/splunkforwarder/bin/splunkd journald-modinput '$@'" journald-modinput - unable to open cursor file /opt/splunkforwarder/var/lib/splunk/modinputs/journald/all.checkpoint for reading (No such file or directory)
When the journald reader starts, it first checks whether the Splunk software needs to resume reading after a previous stop. The first time this process runs, no checkpoint file exists, so the journalid reader will fail to find the checkpoint file, and will write the error to the log.
No additional actions need to be taken, the Splunk software restarts after the ID in this error.
Get data from APIs and other remote data interfaces through scripted inputs | Forward data with the logd input |
This documentation applies to the following versions of Splunk Cloud Platform™: 9.2.2406, 8.2.2201, 8.2.2202, 8.2.2203, 9.0.2205, 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308, 9.1.2312, 9.2.2403 (latest FedRAMP release), 9.0.2208, 8.2.2112
Feedback submitted, thanks!