Splunk® Enterprise

Search Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Export search results

This topic provides a technical overview of Splunk Enterprise data export methods. You can export search results directly out of Splunk Enterprise. You can also forward data to third party systems.

What are the available export methods?

Splunk Enterprise provides several export methods:

Splunk apps

Overview of Splunk Enterprise export options

Splunk Enterprise can export data in different ways. The export method you choose depends on the data volumes involved and your level of interactivity. For example, a single on-demand search export through Splunk Web might be appropriate for a low-volume export. Alternatively, if you want to set up a higher-volume, scheduled export, the SDK and REST options work best.

For large exports, the most stable method of search data retrieval is the Command Line Interface (CLI). From the CLI, you can tailor your search to external applications using the various Splunk Enterprise SDKs. The REST API works from the CLI as well, but is recommended only for internal use.

In terms of level of expertise, the Splunk Web and CLI methods are significantly more accessible than the SDKs and REST API, which require previous experience working with software development kits or REST API endpoints.

Method Volume Interactivity Remarks
Splunk Web Low On-Demand, Interactive Easy to obtain on-demand exports
CLI Medium On-Demand, Low Interactive Easy to obtain on-demand exports
REST High Automated, best for computer-to-computer Works underneath SDK
SDK High Automated, best for computer-to-computer Best for automation

Choose your export format

Splunk Enterprise lets you directly export your data into the following formats:

  • Raw Events
  • CSV
  • JSON
  • XML

Export data using Splunk Web

1. Run a search on your data.

2. Click the export button, located directly below the timeline.

Splunk Export Button.jpg

3. Select the Format that you want the search results to be exported in. You can select CSV, Raw Events, XML or JSON.

4. Choose the Number of Results you want (Limited or Unlimited).

5. Click Export to confirm.

Extend the session timeout when exporting large amounts of data

When you try to export large amounts of data using the export button, you can run into session timeout issues. Follow this procedure to extend the session timeout limit.

1. Click Settings and select Server Settings.

2. Under Splunk Web, increase the number in the Session timeout field.

Timeout.png

Increasing the timeout settings allows Splunk Web more time for the connection between your browser and splunkweb.

Archive search results

If you need to archive your search results, Splunk Enterprise can export your job data into third-party charting applications. See "Export job data to a file" in the Search Manual.

Schedule reports that send results to stakeholders

You can schedule reports to run on a regular interval and send their results to project stakeholders via email. The emails can present the report results in inline tables and CSV or PDF attachments. They can also include links to the report results in Splunk Enterprise.

See "Schedule Reports" in the Reporting Manual.

Export data using the CLI

The Command Line Interface (CLI) is easy to script, can handle automation, and can process volumes of data faster and more efficiently than Splunk Web.

To access Splunk Enterprise through the CLI, you either need shell access to a Splunk Enterprise server, or permission to access the correct port on a remote Splunk server.

Splunk Enterprise CLI exports use the following command structure:

 splunk search [eventdata] -preview 0 -maxout 0 -output [rawdata|json|csv|xml] > [myfilename.log] ... 

By default, CLI exports only export 100 events. To increase this number, use the -maxout argument. For example, if you include -maxout 300000 you can export 300,000 events. Set -maxout to 0 to export an unlimited number of events.

To learn more about the Splunk Enterprise CLI, read "About the CLI" in the Admin Manual.

CLI output command example

This CLI example takes events from the _internal index that occur within the time range specified by the search string and outputs 200,000 of them in raw data format to the file test123.dmp.

splunk search "index=_internal earliest=09/14/2014:23:59:00 latest=09/16/2014:01:00:00 " -output rawdata -maxout 200000 > c:/test123.dmp 

Export using the Splunk Enterprise REST API

You can bypass the Splunk Enterprise SDKs and gain access to Splunk Enterprise by directly using the REpresentational State Transfer (REST) model to make requests from a terminal or browser. The Splunk REST API lets you GET and POST data from a Splunk instance.

To export data using the REST API you need to GET your data from an endpoint. But before you can do that you need to run the search job.

1. Run a search job using the POST operation at /services/search/jobs/.

Set your search as the POST payload. Remember to include the date range in the search, if you want to include one.
curl -k -u admin:changeme \
     https://localhost:8089/services/search/jobs/ -d search="search sourcetype=access_* earliest=-7d"

2. Obtain the search job ID (SID) for the search.

The search returns an XML response that includes the search job ID in <sid> tags.
<?xml version='1.0' encoding='UTF-8'?>
<response>
  <sid>1423855196.339</sid>
</response>
You can also get the search job ID by viewing the job in the Search Job Inspector. Navigate to Activity > Jobs to open the Job Manager, locate the search job you just ran, and click Inspect. The Search Job Inspector opens in a separate window.

3. Create a GET operation to export the results of the search to a file.

The operation should:
  • Identify the search job user and app. The following example defines <user> as admin and <app> as search.
  • Use output_mode to identify your output format. Possible values are JSON, CSV, or XML The following example exports the search results to a JSON file.
curl -u admin:changeme \
     -k https://localhost:8089/servicesNS/admin/search/jobs/1423855196.339/results/ \
     --get -d output_mode=json count=5

For more information about object endpoints in Splunk Enterprise, read "Search endpoint descriptions" in the REST API Reference Manual.

For more information about using the REST API to work with searches, see "Creating searches using the REST API" in Rest API Tutorials.

For a detailed overview of the /services/search/jobs/export endpoint, see "Search endpoint descriptions" in the REST API Reference Manual.

Export using Splunk SDKs

Splunk provides Software Development Kits (SDKs) that help software developers create Splunk apps using common programming languages. Splunk SDKs let you integrate Splunk Enterprise with third-party reporting tools and portals, include search results in your application, and extract high volumes of data for archival purposes. Use of Splunk SDKs require proficiency in SDK knowledge and development.

Splunk offers SDKs for Python, Java, JavaScript, Ruby, and C#. Export searches in these SDKs run immediately, do not create a job for the search, and start streaming results immediately.

The Splunk SDKs are built on top of the Splunk Enterprise REST API. They provide a simpler interface for the REST API endpoints. With fewer lines of code, you can write applications that can:

  • Create and run authenticated searches
  • Add data
  • Index data
  • Manage search jobs
  • Configure Splunk

For more information about the Splunk SDKs, read "Overview of the Splunk SDKs" in the Splunk Developer Portal.

Python SDK

The Splunk SDK for Python lets you write Python applications that can interact with Splunk Enterprise. Export searches using the Python SDK can be run in historical mode and real-time mode. They start right away, and stream results instantly, letting you integrate them into your Python application.

Perform an export search using the Python SDK.

1. Set the parameters of what you wish to search. The following example sets the parameters as an export search of splunklib in the last hour.

import splunklib.client as client
import splunklib.results as results

2. Run a normal-mode search.

service = client.connect(…)
rr = results.ResultsReader(service.jobs.export("search index=_internal earliest= -1h"))

3. Get the results and display them using the ResultsReader.

 if isinstance(result, results.Message):
        # Diagnostic messages may be returned in the results
        print '%s: %s' % (result.type, result.message)
    elif isinstance(result, dict):
        # Normal events are returned as dicts
        print result
assert rr.is_preview == False

Java SDK

The Java SDK is able to conduct and export searches while using Java.

To perform an export search using the Java SDK, run the following example in the /splunk-sdk-java directory using the CLI:

java -jar dist/examples/export.jar main --username="admin" --password="changeme"

The Export application exports the "main" index to export.out, which is saved to the current working directory. If you want to run this application again, delete export.out before you try again. If you do not do this, you will get an error.

Here is a different CLI example of the Java SDK. It shows how to include a search query and change the output format to JSON.

java -jar dist/examples/export.jar main --search="search sourcetype=access_*" json

JavaScript Export

The Javascript Export endpoint can export data out of Splunk Enterprise within the Javascript framework.

Splunk Enterprise does not currently support the Javascript Export endpoint in its Javascript SDK. However, you can use a node javascript (.js) application request to export data.

To perform an export search using the Javascript Export endpoint:

1. Load the request module. Request is designed to be the simplest way to make an http/https call.

var request = require('request');

2. Call get to issue a GET request. Enter the following parameters:

  • strictSSL – When set to false, strictSSL tells the request to not validate the server certificate returned by Splunk Enterprise, which by default is not a valid certificate.
  • uri – Provide the uri of the Splunk Enterprise host along with the path for the export endpoint. A JSON response is specified in the query string.
  • qs – Set qs to supply the search parameter. By passing it this way, you do not have to URI encode the search string.
request.get(
    {
        strictSSL: false,
        uri: 'https://localhost:8089/servicesNS/admin/search/search/jobs/
              export?output_mode=json',
        qs: {
            search: 'search index=_internal'
        }
    }
)

3. Call auth to use HTTP Basic Auth and pass your Splunk Enterprise username and password.

.auth('admin', 'changeme', false)

4. Pipe the results to stdout.

.pipe(process.stdout);

C# SDK

An export search using the C# SDK runs asynchronously and immediately, does not create a job for the search, and starts streaming results right away. The C# SDK is useful when exporting large amounts of historical or real-time data.

To perform an export search using the C# SDK:

1. Create a preview search using StreamReader.

SearchPreviewStream searchPreviewStream;

2. Export the search result previews.

using (searchPreviewStream = service.ExportSearchPreviewsAsync("search index=_internal | head 100").Result)
{
    int previewNumber = 0;

3. Enumerate through each search result preview.


    foreach (var searchPreview in searchPreviewStream.ToEnumerable())
    {
        Console.WriteLine("Preview {0:D8}: {1}", ++previewNumber, searchPreview.IsFinal ? "final" : "partial");
        int recordNumber = 0;

        foreach (var result in searchPreview.Results)
        {
            Console.WriteLine(string.Format("{0:D8}: {1}", ++recordNumber, result));
        }
    }
}

Ruby SDK

The Ruby SDK helps developers build applications using Splunk Enterprise. The Splunk SDK for Ruby lets you write Ruby applications that can interact with the Splunk engine. The following instructions assume you have constructed your service class and connected to your Splunk server.

To perform an export search using the Ruby SDK:

1. Identify your search time parameters using the create_export method.

The create_export method starts a search query and returns the events found by the job before they are run through any transforming commands in the search string. This is equivalent to calling events on a job.
This search example returns events that have taken place within the last hour (-1h, now).
stream = service.create_export("search index=_internal | head 1",
                               :earliest_time => "-1h",
                               :latest_time => "now")

2. Identify the set of events that you want to export.

The following example is an export search of a streaming set of events that are output as raw data (_raw). To speed up performance it returns events before processing them through the transforming search commands in the search string, if any exist. This means that previews are skipped until the export search returns.
results = Splunk::ResultsReader.new(stream)
results.each do |result|
puts "#{result["_raw"]}"
end

3. Save your code to an .rb file (example.rb).

4. Run example.rb on your terminal.

Use the Dump search command

The dump search command allows large collections of events to be "dumped" onto a local disk. It can be used with the CLI, Splunk SDK and Splunk Web.

The basic syntax of the dump command is:

 dump basefilename=<string> [rollsize=<number>] [compress=<number>] [format=<string>] [fields=<comma-delimited-string>] 

The <format> is the data format of the dump file that you are creating. Your format options are raw, csv, tsv,xml, and json.

See the topic on the dump command in the Search Reference for search examples and full explanations of the dump command required and optional arguments.

Forward data to third party systems

Splunk Enterprise is able to forward data to third-party systems. It can send data:

  • Through a plain TCP socket
  • Packaged in a standard syslog

You configure heavy forwarders by editing outputs.conf, props.conf and transforms.conf. This export method is similar to routing your data to other Splunk Enterprise instances. You can filter the data by host, source, or source type. See "Forward data to third party systems" in the Forwarding Data manual.

PREVIOUS
Scheduling searches
  NEXT
About writing custom search commands

This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14


Comments

Thanks Felsherif. The changes were made in Oct when you posted the comment, we just neglected to add a comment here :-)

Lstewart splunk, Splunker
January 12, 2016

In step #1 above in "Export using the Splunk Enterprise REST API" the curl should be to /jobs only, not /export (this doesn't return the sid: "1. Run a search job using the POST operation at /services/search/jobs/export.")

So instead of:

curl -k -u admin:changeme \https://localhost:8089/services/search/jobs/export -d search="search sourcetype=access_* earliest=-7d"

Do this:

curl -k -u admin:changeme \https://localhost:8089/services/search/jobs -d search="search sourcetype=access_* earliest=-7d"

Felsherif splunk, Splunker
October 21, 2015

Hello and thanks for submitting this comment. After checking with our engineering team, they have confirmed that there's no problem with your search, and that the "INFO: Your timerange was substituted based on your search string" is an artifact from Splunk's old CLI search.

The info that returns back is simply to let you know that the search is complying with the time specified in your search string. I am presently working with our engineers to address this, in order to alleviate confusion.

But in short, dates do not need to be converted to unix time.

Please let me know if you have any questions.

Mglauser splunk
March 23, 2015

earliest and latest in the CLI command example don't appear to work. i believe if you are hard-coding dates, you need to first convert them to unix time.

/opt/splunk/bin/splunk search "index=foo earliest=12/01/2014:00:00:00 latest=12/02/2014:00:00:00" -output rawdata -maxout 20
INFO: Your timerange was substituted based on your search string

Awurster
March 17, 2015

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters