This topic walks you through simple searches using the Search interface. If you're not familiar with the search interface, go back to the Search app tutorial before proceeding.
It's your first day of work with the Customer Support team for the online Flower & Gift shop. You're just starting to dig into the Web access logs for the shop, when you receive a call from a customer who complains about trouble buying a gift for his girlfriend--he keeps hitting a server error when he tries to complete a purchase. He gives you his IP address, 10.2.1.44.
Typeahead for keywords
Everything in Splunk is searchable. You don't have to be familiar with the information in your data because searching in Splunk is free-form and as simple as typing keywords into the search bar and hitting Enter (or clicking that green arrow at the end of the search bar).
In the previous topic, you ran a search from the Summary dashboard by clicking on the Web access source type (
access_combined_wcookie). Use that same search to find this customer's recent access history at the online Flower & Gift shop.
1. Type the customer's IP address into the search bar:
As you type into the search bar, Splunk's search assistant opens.
Search assistant shows you typeahead, or contextual matches and completions for each keyword as you type it into the search bar. These contextual matches are based on what's in your data. The entries under matching terms update as you continue to type because the possible completions for your term change as well.
Search assistant also displays the number of matches for the search term. This number gives you an idea of how many search results Splunk will return. If a term or phrase doesn't exist in your data, you won't see it listed in search assistant.
|What else do you see in search assistant?
For now, ignore everything on the right panel next to the contextual help. Search assistant has more uses once you start learning the search language, as you'll see later. And, if you don't want search assistant to open, click "turn off auto-open" and close the window using the green arrow below the search bar.
More keyword searches
2. If you didn't already, run the search for the IP address. (Hit Enter.)
Splunk retrieves the customer's access history for the online Flower & Gift shop.
Each time you run a search, Splunk highlights in the search results what you typed into the search bar.
3. Skim through the search results.
You should recognize words and phrases in the events that relate to the online shop (flower, product, purchase, etc.).
The customer mentioned that he was in the middle of purchasing a gift, so let's see what we find by searching for "purchase".
4. Type purchase into the search bar and run the search:
sourcetype=access_combined_wcookie 10.2.1.44 purchase
When you search for keywords, your search is not case-sensitive and Splunk retrieves the events that contain those keywords anywhere in the raw text of the event's data.
Among the results that Splunk retrieves are events that show each time the customer tried to buy something from the online store. Looks like he's been busy!
Use Boolean operators
If you're familiar with Apache server logs, in this case the access_combined format, you'll notice that most of these events have an HTTP status of 200, or Successful. These events are not interesting for you right now, because the customer is reporting a problem.
5. Use the Boolean NOT operator to quickly remove all of these Successful page requests. Type in:
sourcetype=access_combined_wcookie 10.2.1.44 purchase NOT 200
You notice that the customer is getting HTTP server (503) and client (404) errors.
But, he specifically mentioned a server error, so you want to quickly remove events that are irrelevant.
| Splunk supports the Boolean operators: AND, OR, and NOT. When you include Boolean expressions in your search, the operators have to be capitalized.
The AND operator is always implied between search terms. So the search in Step 5 is the same as:
Another way to add Boolean clauses quickly and interactively to your search is to use your search results.
6. Mouse-over an instance of "404" in your search results and alt-click.
This updates your search string with "NOT 404" and filters out all the events that contain the term.
From these results, you see each time that the customer attempted to complete a purchase and received the server error. Now that you have confirmed what the customer reported, you can continue to drill down to find the root cause.
More about searching for keywords and phrases
When you run a search, you're implicitly using the search command. The search command enables you to use keywords, phrases, fields, boolean expressions, and comparison expressions to specify exactly which events you want to retrieve from a Splunk index(es).
To search with comparison expressions:
- You can use the "=" and "!=" operator with all field/value pairs.
- Other comparison operators, ("<", "<=", ">", and ">=") work only with fields that have numeric values.
Also, when specifying phrases to match, you can use the TERM() directive. TERM forces Splunk to match whatever is inside the parentheses as a single term in the index, even if it contains characters that are usually recognized as breaks or delimiters (such as underscores and spaces). Read more about this in the search command reference topic.
Splunk lets you highlight and select any segment from within your search results to add, remove, and exclude them quickly and interactively using your keyboard and mouse:
- To add more search terms, highlight and click the word or phrase you want from your search results.
- To remove a term from your search, click a highlighted instance of that word or phrase in your search results.
- To exclude events from your search results, alt-click on the term you don't want Splunk to match.
When you're ready to proceed, go to the next topic to learn how to investigate and troubleshoot interactively using the timeline in Splunk.
The Search app
Use the timeline
This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7