Splunk® Enterprise

Troubleshooting Manual

Download manual as PDF

Download topic as PDF

I can't find my data!

Are you searching for events and not finding them, or looking at a dashboard and seeing "No result data"? Here are a few common mistakes to check.

Are you running Splunk Free?

Splunk Free does not support multiple user accounts, distributed searching, or alerting.

Saved searches that were previously scheduled by other users are still available, and you can run them manually as required. You can also view, move, or modify them in Splunk Web or in savedsearches.conf.

Review this topic about object ownership and this topic about configuration file precedence in the Admin Manual for information about where Splunk writes knowledge objects such as scheduled searches.

Was the data added to a different index?

Some apps, like the *nix and Windows apps, write input data to a specific index (in the case of *nix and Windows, that is the "os" index). If you're not finding data that you're certain is in Splunk, be sure that you're looking at the right index. See Retrieving events from indexes in the Search Manual for more information. You might want to add the os index to the list of default indexes for the role you're using. For more information about roles, refer to Add and edit roles with Splunk Web in the Securing Splunk Enterprise manual. For information about troubleshooting data input issues, see Troubleshoot the input process in the Getting Data In manual.

Do your permissions allow you to see the data?

Your permissions can vary depending on the index privileges or search filters. See Add and edit roles in Splunk Web in Securing Splunk Enterprise for more information.

What about issues related to time?

Double check the time range that you're searching. Are you sure the events exist in that time window? Try increasing the time window for your search.

You can try a time picker value of All time for some part of your data, like a source type or string. This is one of the few ways to show events that have been erroneously timestamped with a future timestamp.

If you are running a report, check the time zone of the user who created the report.

The indexer might be incorrectly timestamping for some reason. See How timestamp assignment works in the Getting Data In manual.

Are you using forwarders?

Check that your data is in fact being forwarded. Here are some searches to get you started. You can run all these searches, except for the last one, from the Splunk default Search app. The last search you run from the CLI to access the forwarder. A forwarder does not have a user interface:

  • Are my forwarders connecting to my receiver? Which IP addresses are connecting to Splunk as inputs, and how many times is each IP logged in metrics.log?

index=_internal source=*metrics.log* tcpin_connections | stats count by sourceIp

  • What output queues are set up?

index=_internal source=*metrics.log* group=queue tcpout | stats count by name

  • What hosts (not forwarder/TCP inputs) have logged an event to Splunk in the last 10 minutes? (Including rangemap.)

| metadata type=hosts index=netops | eval diff=now()-recentTime | where diff < 600 | convert ctime(*Time) | stats count | rangemap field=count low=800-2000 elevated=100-799 high=50-99 server=0-49

  • Where is Splunk trying to forward data to? From the Splunk CLI issue the following command:
$SPLUNK_HOME/bin/splunk search 'index=_internal source=*metrics.log* destHost | dedup destHost'
  • If you need to see if the socket is getting established you can look at the forwarder's log of this in splunkd.log "Connected to idx=<ip>:<port>" , and on the receiving side if you set the log category TcpInputConn to INFO or lower you can see messages "Connection in cooked mode from src=<ip>:<port>

Read up on forwarding in the Forwarding Data Manual.

Are you using search heads?

Check that your search heads are searching the indexers that contain the data you're looking for. Read about distributed search in the Distributed Search Manual.

Are you still logged in and under your license usage?

If you have several (3 for Splunk Free or 5 for Enterprise) license violations within a rolling 30 day window, Splunk will prevent you from searching your data.

Note, however, that Splunk will continue to index your data, and no data will be lost. You will also still be able to search the _internal index to troubleshoot your problem. Read about license violations in the Admin Manual.

Are you using a scheduled search?

Are you SURE your time range is correct? (You wouldn't be the first!) Search over all time to double check.

Are you sure the incoming data is indexed when you expect and not lagging? To determine if there is a lag between the event's timestamp and indexed time is to manually run the scheduled search with the added syntax of:

| eval time=_time | eval itime=_indextime | eval lag=(itime - time)/60 | stats avg(lag), min(lag), max(lag) by index host sourcetype

For example there is an indexing lag of up to 90 minutes, if you run a scheduled search every 20 minutes, you might not see the most recent data yet (but if you run the same search 70 minutes later, the data will be there).

It could also be a scheduler problem. The Knowledge Manager Manual has a topic on configuring priority of scheduled searches.

Other common problems with scheduled searches are searches getting rewritten, saved, run incorrectly, or run not as expected. Investigate scheduled searches in audit.log and the search's dispatch directory: read about these tools in "What Splunk logs about itself" in this manual.

Check your search query

  • Are you using NOT, AND, or OR? Check your logic.
  • How about double quotes? Read more about Search language syntax in the Search Reference Manual.
  • Are you using views and drilldowns? Splunk Web might be rewriting the search incorrectly via the intentions functionality.
  • Double check that you're using the correct index, source, sourcetype, and host.
  • Are you correctly using escape characters when needed?
  • Are your subsearches ordered correctly?
  • Are your subsearches being passed the correct fields?

Are you extracting fields?

  • Check your regex. One way to test regexes interactively is in Splunk using the rex command.
  • Do you have privileges for extracting and sharing fields? Read about sharing fields in the Knowledge Manager Manual.
  • Are your extractions applied for the correct source, sourcetype, and host?

Additional resources

Watch a video on troubleshooting missing forwarder data by a Splunk Support engineer.

Have questions? Visit Splunk Answers and see what questions and answers the Splunk community has.

If you get stuck at any point, contact Splunk Support. Don't forget to send a diag! Read about making a diag in this manual.

Command line tools for use with Support
Too many search jobs

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 7.0.0


It should also be mentioned that a way-too-small retention could have caused the events to be rolled to frozen shortly after being indexed. This search should show if that is the case:

index=_internal sourcetype=splunkd bucketmover | rex "[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db[\/\\\]db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+" | rex "db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+.*?[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db" | stats max(oldestTime) AS oldestTime BY indexname | eval retentionDays=(now()-oldestTime)/(60*60*24)

May 22, 2017

I would change this: "You might also want to try a real-time search over all time for some part of your data, like a source type or string." to this "You might also try a timepicker value of 'All time` for some part of your data, like a source type or string; this is one of the few ways to show events that have been mis-timestamped 'into the future' and force them to show up in your search results."

May 22, 2017

UlliPo, the link works for me in Chrome!

Jlaw splunk, Splunker
April 18, 2016

video on troubleshooting missing splunk forwarder data not available on YOUTUBE - link broken.

April 13, 2016

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters