Resolve data quality issues
This topic helps you troubleshoot event-processing and data quality issues such as the following:
- Incorrect line breaking
- Incorrect event breaking
- Incorrect time stamp extraction
Line breaking issues
Indicators that you have line breaking issues include the following:
- You have fewer events than you expect and the events are very large, especially if your events are single-line events.
- Line breaking issues are present in the Monitoring Console Data Quality dashboard.
- In the Splunk Web Data Input workflow or in splunkd.log, an error message like the following.
To confirm that your Splunk software has line breaking issues, do one or more of the following:
- Visit the Monitoring Console Data Quality dashboard. Check the dashboard's table for line breaking issues. See About the Monitoring Console in Monitoring Splunk Enterprise.
- Look for messages in splunkd.log like the following:
- Search for events. Multiple events combined, or a single event broken into many, indicates a line breaking issue.
To resolve line breaking issues, in Splunk Web:
- Click Settings > Add data.
- Click add a file to test or monitor to redo the monitor input.
- Select a file with a sample of your data.
- Click Next.
- On the Set Source Type page, work with the options on the left until your sample data is correctly broken into events. To configure LINE_BREAKER or TRUNCATE, click Advanced.
- Complete the data input workflow or record the correct settings and use them to correct your existing input configurations.
While you are working with the options on the Set Source Type page, the LINE_BREAKER setting might not be properly set. LINE_BREAKER must have a capturing group and the group must match the events.
For example, you might have a value of LINE_BREAKER that is not matched (screenshot called linebreaker_mismatch). Look for messages with "Truncating line because limit of 10000 bytes has been exceeded" in splunkd.log or in Splunk Web:
If you find such a message, do the following:
- Check that LINE_BREAKER is properly configured to segment your data into lines as you expect. Make sure that the string exists in your data.
- If LINE_BREAKER is configured correctly, and you simply have very long lines, or if you are using LINE_BREAKER as the only method to define events, bypassing line merging later in the indexing pipeline,
make sure that TRUNCATE is set large enough to contain the entire data fragment delimited by LINE_BREAKER. The default value for TRUNCATE is 10,000. If your events are larger than the TRUNCATE value, you might want to increase the value of TRUNCATE. For performance and memory usage reasons, do not set TRUNCATE to unlimited.
If you do not specify a capturing group, LINE_BREAKER is ignored.
Event breaking, or aggregation, issues
Event breaking issues can pertain to BREAK_ONLY_BEFORE_DATE, MAX_EVENTS, and any props.conf setting that contains the keyword "BREAK".
Indicators that you have aggregation issues include:
- Aggregation issues present in the Monitoring Console Data Quality dashboard.
- An error in the Splunk Web Data Input work flow.
- Count events. If events are missing and are very large, especially if your events are single-line events, you might have event breaking issues.
To confirm that your Splunk software has event breaking issues, do one or more of the following:
- View the Monitoring Console Data Quality dashboard.
- Search for events, and find that they are multiple events mashed together.
- Check splunkd.log for messages like the following:
For line and event breaking, determine whether this is happening because either (1) your events are properly recognized but too large for the limits in place (MAX_EVENTS, which defines the maximum number of lines in an event), or (2) your events are not properly recognized.
If the cause is scenario 1, you can increase limits. But be aware that large events are not optimal for indexing performance, search performance, and resource usage. Large events can be costly to search. The upper values of both limits result in 10,000 characters per line, as defined by TRUNCATE, times 256 lines, as set by MAX_EVENTS. The combination of those two limits is a very large event.
If the cause is scenario 2, which is more likely, your Splunk software is not breaking events as it should. Check the following:
- Your event breaking strategy. The default is to break before the date, so if Splunk software does not extract a time stamp, it does not break the event. To diagnose and resolve, investigate time stamp extraction. See How timestamp assignment works.
- Your event breaking regex.
For more information:
- See How timestamp extraction works
- See Tune timestamp extraction for better indexing performance
- See Configure event line breaking
Time stamping issues
Time stamping issues can pertain to the DATETIME_CONFIG, TIME_PREFIX, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD, or TZ settings in props.conf. See props.conf.spec in the Admin Manual.
Indicators that you have time stamp parsing issues include:
- Timestamp parsing issues present in the Monitoring Console Data Quality dashboard.
- An error in the Splunk Web Data Input work flow.
- Count events. If you are missing events and have very large events, especially if your events are single-line events, parsing might be a problem.
- Less acute problems like time zone not properly assigned
- The value of
_timeassigned by Splunk software does not match the time in the raw data.
To confirm that you have a time stamping issue, do one or more of the following:
- Visit the Monitoring Console Data Quality dashboard. Check for timestamp parsing issues in the table. Time stamp assignment resorts to various fallbacks, as described in How timestamp assignment works. For most of the fallbacks, even if one of them successfully assigns a time stamp, you still get an issue in the Monitoring Console dashboard.
- Search for events, find that they are multiple events combined.
- Look in splunkd.log for messages like:
All events are indexed with the same time stamp, which makes searching that time range ineffective.
To resolve a time stamping issue:
- Make sure that each event has a complete time stamp, including a year, full date, full time, and a time zone.
- See Configure time stamp recognition for additional possible resolution steps.
Troubleshoot the input process
This documentation applies to the following versions of Splunk® Enterprise: 6.5.1612 (Splunk Cloud only), 6.6.0, 6.6.1, 6.6.2, 6.6.3