This topic walks you through simple searches using the Search interface. If you're not familiar with the search interface, go back to the search app tutorial before proceeding.
The Backstory: You are a member of the Customer Support team for the online Flower & Gift shop. This is your first day on the job. You want to learn some more about the shop. Some questions you want answered are:
- What does the store sell? How much does each item cost?
- How many people visited the site? How many bought something today?
- What is the most popular item that is purchased each day?
It's your first day of work with the Customer Support team for the online Flower & Gift shop. You're just starting to dig into the Web access logs for the shop, when you receive a call from a customer who complains about trouble buying a gift for his girlfriend--he keeps hitting a server error when he tries to complete a purchase. He gives you his IP address, 10.2.1.44.
Everything in Splunk is searchable. You don't have to be familiar with the information in your data because searching in Splunk is free-form and as simple as typing keywords into the search bar and hitting Enter (or clicking that green arrow at the end of the search bar).
Type ahead, or Search assistant
In the previous topic, you ran a search from the Summary dashboard by clicking on the Web access source type (
access_combined_wcookie). Use that same search to find this customer's recent access history at the online Flower & Gift shop.
1. Type the customer's IP address into the search bar:
As you type into the search bar, Splunk's search assistant opens.
Search assistant shows you typeahead, or contextual matches and completions for each keyword as you type it into the search bar. These contextual matches are based on what's in your data. The entries under matching terms update as you continue to type because the possible completions for your term change as well.
Search assistant also displays the number of matches for the search term. This number gives you an idea of how many search results Splunk will return. If a term or phrase doesn't exist in your data, you won't see it listed in search assistant.
For now, ignore everything on the right panel next to the contextual help. Search assistant has more uses once you start learning the search language, as you'll see later. And, if you don't want search assistant to open, click "turn off auto-open" and close the window using the green arrow below the search bar.
More keyword searches
2. If you didn't already, run the search for the IP address. (Hit Enter.)
Splunk retrieves the customer's access history for the online Flower & Gift shop. The timeline also updates, but we'll get to that later. For now, let's just take a look at the search results.
Each time you run a search, Splunk highlights in the search results what you typed into the search bar.
3. Skim through the search results.
You should recognize words and phrases in the events that relate to the online shop (flower, product, purchase, etc.).
The customer mentioned that he was in the middle of purchasing a gift, so let's see what we find by searching for "purchase".
4. Type purchase into the search bar and run the search:
sourcetype="access_combined_wcookie" 10.2.1.44 purchase
When you search for keywords, your search is not case-sensitive and Splunk retrieves the events that contain those keywords anywhere in the raw text of the event's data.
Among the results that Splunk retrieves are events that show each time the customer tried to buy something from the online store. Looks like he's been busy!
Use Boolean operators
If you're familiar with Apache server logs, in this case the access_combined format, you'll notice that most of these events have an HTTP status of 200, or Successful. These events are not interesting for you right now, because the customer is reporting a problem.
Splunk supports the Boolean operators: AND, OR, and NOT. When you include Boolean expressions in your search, the operators have to be capitalized.
5. Use the Boolean NOT operator to quickly remove all of these Successful page requests. Type in:
sourcetype="access_combined_wcookie" 10.2.1.44 purchase NOT 200
The AND operator is always implied between search terms. So the search in Step 5 is the same as:
sourcetype="access_combined_wcookie" AND 10.2.1.44 AND purchase NOT 200
You notice that the customer is getting HTTP server (503) and client (404) errors. But, he specifically mentioned a server error, so let's quickly remove events that are irrelevant.
Another way to add Boolean clauses quickly and interactively to your search is to use your search results. Splunk lets you highlight and select any segment from within your search results to add, remove, and exclude them quickly and interactively using your keyboard and mouse:
- To add more search terms, highlight and click the word or phrase you want from your search results. (This is demonstrated in Step 6.)
- To remove a term from your search, click a highlighted instance of that word or phrase in your search results.
- To exclude events from your search results, alt-click on the term you don't want Splunk to match.
6. Mouse-over an instance of "404" in your search results and alt-click.
This updates your search string with "NOT 404" and filters out all the events that contain the term.
From these results, you see each time that the customer attempted to complete a purchase and received the server error. Now that you have confirmed what the customer reported, you can continue to drill down to find the root cause.
Read more about searching
When you run a search, you're implicitly using the search command to retrieve events from a Splunk index(es). The search command enables you to use keywords, phrases, fields, boolean expressions, and comparison expressions to specify exactly which events you want to retrieve. This topic discussed searching with keywords and boolean expressions. Later topics in the tutorial will go over using time, fields, and the search language.
What's not discussed in this tutorial is using comparison expressions and operators for exact phrase matching, TERM() and CASE(). Read more about these methods in "Use the search command" in the Retrieve events chapter of the Search Manual.
When you're ready to proceed, go to the next topic to learn how to investigate and troubleshoot interactively using the timeline in Splunk.
The search app
Use the timeline
This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18