Docs » Get started with the Splunk Distribution of the OpenTelemetry Collector » Troubleshoot the Collector » Troubleshoot log collection

Troubleshoot log collection 🔗

This document describes common issues related to log collection with the Collector.

To troubleshoot the health and performance of the Collector see the OpenTelemetry Project troublehooting docs . It includes information about troubleshooting tools and debugging.

For more information on log collection see:

Platform

Docs

Kubernetes

Collect logs and events with the Collector for Kubernetes

Linux

Collect logs with the Collector for Linux

Windows

Collect logs with the Collector for Windows

My source isn’t generating logs 🔗

If using Linux, run the following commands to check if the source is generating logs:

tail -f /var/log/myTestLog.log
journalctl -u my-service.service -f

If using Windows, run the following command to check if the source is generating logs:

Get-Content myTestLog.log

Fluentd isn’t configured correctly 🔗

Do the following to check the Fluentd configuration:

  1. Check that td-agent is running. On Linux, run systemctl status td-agent. On Windows, run Get-Service td-agent.

  2. If you changed the configuration, restart Fluentd. On Linux, run systemctl restart td-agent. On Windows, run Restart-Service -Name td-agent.

  3. Check fluentd.conf and conf.d/*. @label @SPLUNK must be added to every source to activate log collection.

  4. Manual configuration might be required to collect logs off the source. Add configuration files to in the conf.d directory as needed.

  5. Activate debug logging in fluentd.conf (log_level debug), restart td-agent, and check that the source is generating logs.

While every attempt is made to properly configure permissions, it is possible that td-agent does not have the permission required to collect logs. Debug logging should indicate this issue.

It’s possible that the <parser> section configuration does not match the log events.

If you see a message such as “2021-03-17 02:14:44 +0000 [debug]: #0 connect new socket”, Fluentd is working as expected. You need to activate debug logging to see this message.

The Collector isn’t configured properly 🔗

Note

Fluentd is part of the Splunk Distribution of OpenTelemetry Collector, but deactivated by default for Linux and Windows. To activate it, use the --with-fluentd option when installing the Collector for Linux, or the with_fluentd = 1 option when installing the Collector for Windows.

Do the following to check the Collector configuration:

  1. Go to http://localhost:55679/debug/tracez to check zPages for samples. You might need to configure the endpoint.

  2. Activate logging exporter. See Logging exporter for more information.

  3. Run journalctl -u splunk-otel-collector.service -f to collect the logs for you to review.

  4. Review Troubleshoot the Splunk OpenTelemetry Collector if you can’t find what you need in the logs.

Test the Collector by sending synthetic data 🔗

You can manually generate logs. By default, Fluentd monitors journald and /var/log/syslog.log for events.

echo "2021-03-17 02:14:44 +0000 [debug]: test" >>/var/log/syslog.log
echo "2021-03-17 02:14:44 +0000 [debug]: test" | systemd-cat

Caution

Fluentd requires properly structured syslog to pick up the log line.

Unwanted profiling logs appearing in Splunk Observability Cloud 🔗

By default, the Splunk Distribution of the OpenTelemetry Collector sends AlwaysOn Profiling data using the Splunk HEC exporter. See Turn off logs or profiling data for more information.

Exclude log data in the Collector 🔗

Depending on its configuration, the Splunk Distribution of the OpenTelemetry Collector might collect and send logs to Splunk Observability Cloud through a logs pipeline that uses the Splunk HEC exporter.

To turn off logs colletion, see Turn off logs or profiling data for more information.

Send logs to Splunk Cloud Platform or Enterprise using the Collector 🔗

To send logs from the Collector to Splunk Cloud Platform or Splunk Enterprise, see Send logs to Splunk Cloud Platform or Splunk Enterprise.

This page was last updated on Nov 26, 2024.