I can't find my data!

Are you searching for events and not finding them, or looking at a dashboard and seeing "No result data"? Here are a few common mistakes to check.

Are you running Splunk Free?

Splunk Free does not support multiple user accounts, distributed searching, or alerting.

Saved searches that were previously scheduled by other users are still available, and you can run them manually as required. You can also view, move, or modify them in Splunk Web or in savedsearches.conf.

Review this topic about object ownership and this topic about configuration file precedence in the Admin Manual for information about where Splunk writes knowledge objects such as scheduled searches.

Was the data added to a different index?

Some apps, like the *nix and Windows apps, write input data to a specific index (in the case of *nix and Windows, that is the "os" index). If you're not finding data that you're certain is in Splunk, be sure that you're looking at the right index. See Retrieving events from indexes in the Search Manual for more information. You might want to add the os index to the list of default indexes for the role you're using. For more information about roles, refer to Add and edit roles with Splunk Web in the Securing Splunk Enterprise manual. For information about troubleshooting data input issues, see Troubleshoot the input process in the Getting Data In manual.

Do your permissions allow you to see the data?

Your permissions can vary depending on the index privileges or search filters. See Add and edit roles in Splunk Web in Securing Splunk Enterprise for more information.

What about issues related to time?

Double check the time range that you're searching. Are you sure the events exist in that time window? Try increasing the time window for your search.

You can try a time picker value of All time for some part of your data, like a source type or string. This is one of the few ways to show events that have been erroneously timestamped with a future timestamp.

If you are running a report, check the time zone of the user who created the report.

The indexer might be incorrectly timestamping for some reason. See How timestamp assignment works in the Getting Data In manual.

Are you using forwarders?

Check that your data is in fact being forwarded. Here are some searches to get you started. You can run all these searches, except for the last one, from the Splunk default Search app. The last search you run from the CLI to access the forwarder. A forwarder does not have a user interface:

• Are my forwarders connecting to my receiver? Which IP addresses are connecting to Splunk as inputs, and how many times is each IP logged in metrics.log?

index=_internal source=*metrics.log* tcpin_connections | stats count by sourceIp

• What output queues are set up?

index=_internal source=*metrics.log* group=queue tcpout | stats count by name

• What hosts (not forwarder/TCP inputs) have logged an event to Splunk in the last 10 minutes? (Including rangemap.)

| metadata type=hosts index=netops | eval diff=now()-recentTime | where diff < 600 | convert ctime(*Time) | stats count | rangemap field=count low=800-2000 elevated=100-799 high=50-99 severe=0-49

• Where is Splunk trying to forward data to? From the Splunk CLI issue the following command:

$SPLUNK_HOME/bin/splunk search 'index=_internal source=*metrics.log* destHost | dedup destHost'

• If you need to see if the socket is getting established you can look at the forwarder's log of this in splunkd.log "Connected to idx=<ip>:<port>" , and on the receiving side if you set the log category TcpInputConn to INFO or lower you can see messages "Connection in cooked mode from src=<ip>:<port>

Read up on forwarding in the Forwarding Data Manual.

Are you using search heads?

Check that your search heads are searching the indexers that contain the data you're looking for. Read about distributed search in the Distributed Search Manual.

Are you still logged in and under your license usage?

If you have a number of license violations within a range of days, you will be prevented from searching your data. See About license violations in the Admin Manual. The Splunk platform will continue to index your data, and no data will be lost. You can use the admin user to search the _internal index and troubleshoot the problem.

Are you using a scheduled search?

Your time range could be excluding the events. Search over all time to verify.

Are you sure the incoming data is indexed when you expect and not lagging? For example, indexing can lag for tens of minutes under certain conditions. If you run a scheduled search every 20 minutes, you might not see the most recent data yet. But if you run the same search 70 minutes later, the data will be there.

To identify a lag between the event's timestamp and indexed time, manually run the scheduled search with the following added syntax:

| eval time=_time | eval itime=_indextime | eval lag=(itime - time)/60 | stats avg(lag), min(lag), max(lag) by index host sourcetype

See Event indexing delay in this manual.

Missing data can also result from a scheduler problem. See Configure the priority of scheduled reports in the Reporting Manual.

Other common problems with scheduled searches are searches getting rewritten, saved, run incorrectly, or run not as expected. Investigate scheduled searches in audit.log and the search's dispatch directory. See What Splunk logs about itself in this manual and Dispatch directory and search artifacts in the Search Manual.

Check your search query

• Are you using NOT, AND, or OR? Check your logic.
• How about double quotes? Read more about Search language syntax in the Search Reference Manual.
• Are you using views and drilldowns? Splunk Web might be rewriting the search incorrectly via the intentions functionality.
• Double check that you're using the correct index, source, sourcetype, and host.
• Are you correctly using escape characters when needed?
• Are your subsearches ordered correctly?
• Are your subsearches being passed the correct fields?

Are you extracting fields?

• Check your regex. One way to test regexes interactively is in Splunk using the rex command.
• Do you have privileges for extracting and sharing fields? Read about sharing fields in the Knowledge Manager Manual.
• Are your extractions applied for the correct source, sourcetype, and host?