Add data to Splunk
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Add data to Splunk
This topic assumes that you have already downloaded, installed, and started a Splunk server. If you haven't yet, go back to the previous topic for instructions to do that.
Once you've started and logged into Splunk, you need to give it data that you can search. This topic walks you through downloading the sample dataset and adding it into Splunk.
Download the sample data file
This tutorial uses sample data from an fictitious online store, the Flower & Gift shop, to teach you about using Splunk. The sample data includes:
- Apache web server logs
- mySQL database logs
You can feed Splunk data from files and directories, network ports, and custom scripts, but for this tutorial, you will upload a compressed file directly to Splunk. Also, this tutorial is designed to be completed in a matter of hours. But, if you want to spread it out over a few days, just download a new sample data file and add it!
To proceed with this tutorial, download (but do not uncompress) the sample data from here: sampledata.zip This sample data file is updated daily.
Get the sample data into Splunk
Logging into Splunk should have taken you to Splunk Home. If it isn't the first view that you see, use the App list to select Home.
1. In Splunk Home, click Add data.
This takes you to the Add Data to Splunk dialogue where you can Choose a Data Type to add Or Choose a Data Source.
2. Under Or Choose a Data Source, click From files and directories.
This takes you to the Preview data dialogue, which enables you to see a preview of the data before you add it to a Splunk index. For the purposes of this tutorial, you won't need to do this. If you're interested in reading more about data preview, refer to "Overview of data preview" in the Getting Data In manual.
3. Select Skip preview and click Continue.
This takes you to the Home > Add data > Files & directories > Add new view. This is where you will upload the sample data file. Normally, this is all you need to do and Splunk handles the rest without any changes needed. For the purposes of this tutorial, however, you will also edit some of the properties.
4. Under Source, select Upload and index a file and browse for the sample data file that you just downloaded.
The source of an event tells you where it came from. If you collect data from files and directories, the "source" is the full pathname of the file or directory. In the case of a network-based source, the source is the protocol and port, such as UDP:514.
5. Select More settings.
This enables you to override Splunk's default settings for Host, Source type, and Index. For this tutorial, you're just going to change the Host settings.
| What about the Source type and Index settings?
The source type of an event tells you what kind of data it is, usually based on how it's formatted. Examples of source types are access_combined or cisco_syslog. This classification lets you search for the same type of data across multiple sources and hosts. For more information about how Splunk source types your data, read "Why source types matter" in the Getting Data In manual. The index setting tells Splunk where to put the data. By default, it's stored in main, but you might want to consider partitioning your data into different indexes if you have many types. For more information about creating custom indexes, read "Set up multiple indexes" in the Admin manual. |
6. Under Host and Set host, choose regex on path.
An event's host value is typically the hostname, IP address, or fully qualified domain name of the network host from which the event originated. If you take a look at the Sampledata.zip file, it contains four directories (folders): three of the folders are named for Apache web servers and one is a MySQL server. You want to set the host value to the names of these folders.
By selecting regex on path, you're telling Splunk to use a regular expression (regex) to match the segment of the path within the compressed file that you want to set as your host value.
7. Under Regular expression, copy and paste:
For Linux\Unix:
Sampledata.zip:./([^/]+)/
For Windows:
Sampledata.zip:.\\([^/]+)/
This regex should match any characters in the segment path under (Linux/Unix) Sampledata.zip/ or (Windows) Sampledata.zip\ .
8. Click Save.
When it's finished, Splunk displays a message saying the upload was successful.
Click Start searching and proceed to the next topic in this tutorial to look at your data in the Search app.
More about getting data in
This topic only discusses one type of input, uploading a local file, which is all you need to run through the tutorial. For information about all other type of data inputs Splunk can handle and how to add them, refer to the Getting Data In manual beginning with the topic, "What Splunk can index".
This documentation applies to the following versions of Splunk: 4.3 , 4.3.1 , 4.3.2 , 4.3.3 , 4.3.4 , 4.3.5 , 4.3.6 View the Article History for its revisions.
Comments
Sorry, IE + Win7 mangled the regex test program. The while loop line should be:
while ( ) {
and the matched line should be:
print "Matched: |$`$'|\n";
I have a nodding acquaintance with Regular Expressions from dabbling with Perl programs in a Unix environment, but the Unix/Linux Regex above:
./([^/]+)/
Makes no sense to me. It also throws syntax errors when I plug in into this little Regex test perl program:
=====
#!/usr/bin/perl
while () { # take one input line at a time
chomp;
if (/YOUR_PATTERN_GOES_HERE/) {
print "Matched: |$`$'|\n"; # the special match vars
} else {
print "No match: |$_|\n";
}
}
=====
Is there some additional syntax specific to Splunk in this Regex? Or is it a different flavor of Regex than Perl Regex?
Sorry for the delayed response, we have updated the topic.
+1 - had to use Rich's regex, too
Can confirm that Richprescotts example is working, the one originally provided in the tutorial not.
Please be so kind to rework this article.
Yes, that's correct! Richprescott's regular expression is the one which works on Windows.
The one specified in the tutorial (for Windows) only extracts a single log file.
~ rajeev.joshi13@gmail.com
The regex listed did not work when importing the data from a Windows machine. Had to use the following regex instead: Sampledata.zip:.\\([^/]+)/
*Sigh*, still wrong. I cannot write a pair of angle brackets in these comments...