SOAP API

SOAP API

The Splunk SOAP API is being replaced with an XML API starting with 3.1. This documentation is being maintained to support existing SOAP integrations. For future use, please see either the REST API (new for 3.0) or look for the forthcoming XML API.

How Splunk uses SOAP

Splunk's SOAP API is how the splunkweb process communicates with splunkd process. Anything you can do through Splunk's Web interface, you can do through SOAP. You can make calls directly by building complete SOAP messages via the front-end of your choice, or use the Python Control Layer interface.

Python Control Layer

The Python Control Layer (PCL) allows you to send SOAP requests from your Python code without having to generate the full call yourself. You can still use SOAP directly, but many operations are available in the PCL and more will be added in the future. All of the data input methods in the PCL take a (more or less) standardized Python dictionary object and, in most cases, return a dictionary as well.

As for the methods themselves, they follow the convention:

def <module>_<operation>(args, fromCLI)

where <module> is the input module (e.g. "tail" or "fifo") and <operation> is the operation that is to be performed with the module (e.g. "addFile" or "list"). args is a Python dictionary object. fromCLI is a boolean variable that is set to True if the method is called from the command line tool, but is otherwise defaulted to False for calls from other consumers, such as the search interface.

The PCL Data Input Methods can be found in the control_api.py module, located in $SPLUNK_HOME/lib/python2.4/site-packages/splunk/clilib. To use this module, import the following into your Python code:

control_api.py

cli_common.py

control_exceptions.py

control_api.py defines most of the interface, cli_common.py contains authorization functions and control_exceptions.py contains the exceptions used by the PCL. This directory also contains many additional source files that actually implement the interface. All the Python code that Splunk uses is in these files for you to examine. As more functionality is folded into the PCL, methods will be appropriately exposed through control_api.py.

You can find the list of supported commands in cli.py (search for "cList" around line 250.) This file defines the interface exactly as Splunk uses it, so this is always the most accurate source for information. The list looks like this:

cList.addCmd(SplunkCmd("add",    "user",             user_add, "username" , proReq = True))
cList.addCmd(SplunkCmd("add",    "fifo",             fifo_add, "source"))

The argument "proReq = True" indicates that this function is only available with a Splunk enterprise license. Any request to a Splunk server with an enterprise license must be authenticated by passing a valid authentication token with the request. Additional details can be found by looking at the function definitions elsewhere in the file or the implementations in the *.py files.

All methods take a dictionary (set of key-value pairs). Methods usually have one or more required key-value pairs and zero or more optional key-value pairs. All keys are required to be in lower-case. There is no case restriction on the values. If a module is unable to find a required key-value pair in the incoming dictionary, an exception will be raised. If the method finds a key-value pair that is not in its set of required or optional key-values, an exception will be raised.

On error the methods will raise exceptions, so calls to the PCL should be wrapped in try/except blocks for error handling purposes.

Most methods will return a dictionary on completion. In most cases, an empty dictionary on return signifies a successful call. In certain cases, the dictionary will contain a "serverRestartReq" key-value signifying that splunkd must be restarted for changes to take effect.

Environment

Splunk requires a number of environment variables to be set, such as SPLUNK_HOME, and the correct PATH to be able to locate splunk files. Source the file setSplunkEnv in the splunk bin directory (usually /opt/bin/splunk) to set up the necessary environment. As long as the environment is correct, you can run your Python code from anywhere on your filesystem.

source /opt/splunk/bin/setSplunkEnv

Authentication

Every call requires an authentication token be passed in the dictionary, even for the free server which does not support users or authentication. In this situation, use a dictionary with an empty string for the authstr.

All requests to a Splunk server with an enterprise license must be authenticated, otherwise they will fail. To do this, first call getAuthInfo with a valid username and password. The server responds with an authentication token to be used for subsequent requests. This token is good until you submit a logout request or the next time splunkd is restarted.

Example

This is a minimal example of using the PCL to generate a SOAP call. It has no authentication or error handling but shows the basic structure of a PCL call (in this case to a free server.) It generates a SOAP request for function tail_list and prints the returned dictionary containing a list of all the tails currently configured in the server. Note that it calls the version of Python supplied with Splunk and not any that may be installed on the system. Use Splunk's version of Python to avoid problems with incompatible versions.

#!/opt/splunk/bin/python -u
import splunk.clilib.control_api as ca
argsDict = {
  "authstr" : ""
}
results = ca.tail_list(argsDict)
print results

The output looks like this:

{'dirs': ['/var/log/httpd'], 'dynamic': ['/var/log/httpd/error_log', '/var/log/httpd/access_log'], 'static': ['/var/log/system.log']}

For this minimal configuration, you can see there is one directory (/var/log/httpd) that contains two files (error_log and access_log) and one additional file specified by name (system.log.)

See the following page for more complete examples of making PCL calls from your Python code.

Using the PCL to Make SOAP Requests

Authentication Example

Below is a more complete example of the list_tails function. It authenticates against a Splunk server with an enterprise license, which could be elsewhere on the network, passes the authentication token with the list_tails request, and prints the contents of the returned dictionary.

#!/opt/splunk/bin/python -u
# splunk api
import splunk.clilib.control_api as ca
import splunk.clilib.cli_common  as comm
# splunk exception handling
# use "from ... import ..." to avoid namespace problems
from splunk.clilib import control_exceptions
# for exit()
import sys
# optionally connect to another server at hostname:port
# 8089 is the default Splunk SOAP port
# comm.setURI("https://testserver:8089")
try:
# send the username/password, get back the auth token
  authString = comm.getAuthInfo("admin", "changeme")
except control_exceptions.AuthError:
  print "Authentication failed."
  sys.exit(1)
# make a dictionary with the auth token
argsDict = {
  "authstr" : authString
}
# PCL call, returns list of tails
results = ca.tail_list(argsDict)
# output the results
for key, values in results.items():
  print "%s: %s" % (key, values)

Run against a Splunk server with an enterprise license, the output looks like this:

dirs: ['/var/log/httpd']

dynamic: ['/var/log/httpd/error_log', '/var/log/httpd/access_log']

static: ['/var/log/system.log']

If this example is run against a free server, an exception is raised because it can't handle the call to getAuthInfo.

Traceback (most recent call last):
  File "./listTails.py", line 18, in ?
    authString = comm.getAuthInfo("admin", "changeme")
  File "/opt/splunk/lib/python2.4/site-packages/splunk/clilib/cli_common.py", line 135, in getAuthInfo
    retStr = callAPI(block)
  File "/opt/splunk/lib/python2.4/site-packages/splunk/clilib/cli_common.py", line 180, in callAPI
    raise InvokeAPI, str(root.documentElement.firstChild.data)
splunk.clilib.control_exceptions.InvokeAPI: 'badparams'

The message "badparams" is a generic error. In this case it indicates that the requested operation, authentication against a free server, does not exist.

Search Example

Here is an example of a search. The key "terms" is the same as what you would type in the Splunk Box to do a search. For more on what you can specify here, see the Language Reference.

#!/opt/splunk/bin/python -u
# splunk api
import splunk.clilib.control_api as ca
import splunk.clilib.cli_common  as comm
# splunk exception handling
# use "from ... import ..." to avoid namespace problems
from splunk.clilib import control_exceptions
# for exit()
import sys
try:
# send the username/password, get back the auth token
  authString = comm.getAuthInfo("admin", "changeme")
except control_exceptions.AuthError:
  print "Authentication failed."
  sys.exit(1)
# make a dictionary with the auth token and the search string
argsDict = {
  "authstr" : authString,
  "terms"   : "404"
}
results = ca.search(argsDict)
print results

The output when run against the same minimal Splunk server with an enterprise license is this:

127.0.0.1 - - [07/Feb/2007:10:28:33 -0800] "GET /favicon.ico HTTP/1.1" 404 302

To get the full event, including metadata, specify you want XML by adding "format" to the dictionary:

argsDict = {
  "authstr" : authString,
  "terms"   : "404",
  "format"  : "xml"
}

Now the result is this:

<queryResult><ids>
</ids>
<results type="events"> <result cd="0:2891"><segtext xml:space="preserve">127.0.0.1 - - [07/Feb/2007:10:28:33 -0800] "GET /favicon.ico HTTP/1.1" 404 302<meta></meta></segtext>               <timestamp>1170872913</timestamp>               <source cd="2">/var/log/httpd/access_log</source>
                <host cd="1" name="test.splunk.com" >                 <tags></tags>
</host>         <sourcetype cd="2" base="too_small">too_small</sourcetype>
                <type cd="9" wob=" v:226e b1:110 a1:49 j1:4111519 k2:529728502 h2:2550214376 g1:2550123436  ">
                        <tags></tags>
</type>
</result></results></queryResult>

For more about the search function, you can look at the Python code in the file searchizzle.py. The function definition begins like this:

def search(args, fromCLI):
  paramsReq = ("terms", "authstr")
  paramsOpt = ("output", "format", "get")

which shows the required and optional parameters. "output" is the output type, the default is "splunkui" (and this is almost always the correct choice.) "format" was mentioned above, and "get" is the type of data to return.

The default for get is "events", to return matching events. You could also specify "hosts" to return a list of hosts with matching events and so on. The list of possible values is defined in validGetTypes a few lines below:

  validGetTypes     = ("events", "hosts", "sources", "sourcetypes", "types")

Additional PCL Search Examples

The search function has a number of optional parameters for determining what results are returned. In addition to the standard event results discussed in the previous example, here are some other ways you can use the PCL search function.

output

For a short report of selected matching event statistics, use "output":

argsDict = {
  "authstr" : authString,
  "terms"   : "404",
  "output"  : "scheduler"
}

If a regular search returns these events:

127.0.0.1 - - [27/Feb/2007:16:02:31 -0800] "GET /test.html HTTP/1.1" 404 305
127.0.0.1 - - [27/Feb/2007:14:00:50 -0800] "GET /favicon.ico HTTP/1.1" 404 307
127.0.0.1 - - [26/Feb/2007:16:16:35 -0800] "GET /test.html HTTP/1.1" 404 300
127.0.0.1 - - [26/Feb/2007:15:40:27 -0800] "GET /test.html HTTP/1.1" 404 300
127.0.0.1 - - [26/Feb/2007:15:37:04 -0800] "GET /test.html HTTP/1.1" 404 300
127.0.0.1 - - [26/Feb/2007:15:29:29 -0800] "GET /test.html HTTP/1.1" 404 300
127.0.0.1 - - [26/Feb/2007:15:27:07 -0800] "GET /favicon.ico HTTP/1.1" 404 302

Then specifying an output type of "scheduler" returns this:

eventCount:          7
hostCount:           1
sourceCount:         1
typeCount:           2
sourceTypeCount:     1
eventTagCount:       0
hostTagCount:        0
starttime:           12/31/1969:16:00:00
endtime:             03/06/2009:12:46:17

get

Use "get" to return a list of hosts, sources, sourcetypes or types that contain matching events. The dictionary shown below returns a list of the event types of the matching events.

argsDict = {
  "authstr" : authString,
  "terms"   : "404",
  "get"     : "types"
}

returns

eventtype::?9
eventtype::?11

With any of these, you can also specify a format of "xml" to get the output in XML with additional information:

<queryResult><ids>
</ids>
<results type="types">  <result c="5" cd="9"><segtext xml:space="preserve">5 results from eventtype::?9<meta></meta></segtext>          <type cd="9" wob=" v:cc9b b1:110 a1:49 j1:4111519 k2:529728502 h2:2550214376 g1:0  ">
                        <tags></tags>
</type>
</result>       <result c="2" cd="11"><segtext xml:space="preserve">2 results from eventtype::?11<meta></meta></segtext>                <type cd="11" wob=" v:cc9b b1:110 a1:49 j1:4111519 k2:529728502 h2:2550214376 g1:2550123436  ">
                        <tags></tags>
</type>
</result></results></queryResult>

Creating SOAP Requests

Observing PCL SOAP Calls

You can see exactly how Splunk creates SOAP calls with a small change to the existing PCL code to output the data sent and received. Use these examples to generate your own SOAP messages from the front-end of your choice and parse the data returned from splunkd for your application.

1. Open $SPLUNK_HOME/lib/python2.4/site-packages/splunk/clilib/cli_common.py
2. Find the following block in the callAPI function:

  try:
    if use_rpc == True:
      retStr = rpc(data)
    else:
      retStr = server.invokeAPI(apiData=data)
  except socket.error:
    raise SOAPConnectionException, "Could not connect to splunk server.  Please ensure that splunkd is running."

1. After the else: line, add (with 6 leading spaces):

      print "BEGIN SOAP CALL:\n%s\nEND SOAP CALL" % data

1. After the SOAPConnectionException line, add (with 2 leading spaces):

  print "BEGIN RETURNED XML:\n%s\nEND RETURNED XML" % retStr

1. Save the file
2. Run a few commands from the GUI to see what is being sent. The results can be found in $SPLUNK_HOME/var/log/splunk/web_access.log. If you prefer to send commands from the CLI, the output goes to stdout.

Remember to remove the added code later to avoid filling this file wtih SOAP calls.

Creating a SOAP Call

Here is an example of search via SOAP:

1. Make a request to https://localhost:8089/ (replace with your hostname:port) to generate an authentication token (valid for as long as you leave splunkd running, or until you switch authentication mechanisms):

<call name="userLogin"><params><login><username>admin</username><password>changeme</password></login></params></call>

This returns an authentication token, which you must include in your search call:

<auth>
  <userId>1</userId>
  <username>admin</username>
  <authToken>3105802749</authToken>
</auth>

1. Now run the search call, with a few important notes:

• the query in this call is "meta::all".
• it will fetch the first 3 matches (0 through 2).
• the query ID should be a randomly generated/unique number for each search.
• note that the authentication token generated above is included within the <params> block of the call.

<call name="executeQuery"><params><query>SEARCH  meta::all GET events::0-2 
OUTPUT splunkui::2.1 
format::raw</query><user>livesplunkuser</user><queryId>37627661733490256838</queryId>
<auth>
  <userId>1</userId>
  <username>admin</username>
  <authToken>3105802749</authToken>
</auth>
</params></call>

This results in the following xml (3 results, one in each <result> block),

which you can parse as you wish:

<queryResult>
<ids>
</ids>
<results type="events"><result cd="0:873197"><segtext xml:space="preserve">Feb 
19 14:25:49 VeeAte last message repeated 1964 times<meta><sg 
c="4003720337">format::l1_::_</sg> </meta></segtext>                
<timestamp>1171923949</timestamp>
                <source cd="1">/var/log/messages</source>
                <host cd="1" name="veeate" ><tags></tags></host>
                <sourcetype cd="1" base="syslog">syslog</sourcetype>
                <type cd="178" wob=" v:2af8 b1:97 a1:49 j1:1122 k2:5772 
h2:2550729191 g1:54764362 l2:0  ">
<tags></tags></type>
</result>
<result cd="0:873110"><segtext xml:space="preserve">Feb 19 14:24:49 VeeAte 
last message repeated 1956 times<meta><sg c="4003720337">format::l1_::_</sg> 
</meta></segtext>               <timestamp>1171923889</timestamp>
                <source cd="1">/var/log/messages</source>
                <host cd="1" name="veeate" ><tags></tags></host>
                <sourcetype cd="1" base="syslog">syslog</sourcetype>
                <type cd="178" wob=" v:2af8 b1:97 a1:49 j1:1122 k2:5772 
h2:2550729191 g1:54764362 l2:0  ">
<tags></tags></type>
</result>
<result cd="0:873023"><segtext xml:space="preserve">Feb 19 14:23:49 VeeAte 
last message repeated 1951 times<meta><sg c="4003720337">format::l1_::_</sg> 
</meta></segtext>               <timestamp>1171923829</timestamp>
                <source cd="1">/var/log/messages</source>
                <host cd="1" name="veeate" ><tags></tags></host>
                <sourcetype cd="1" base="syslog">syslog</sourcetype>
                <type cd="178" wob=" v:2af8 b1:97 a1:49 j1:1122 k2:5772 
h2:2550729191 g1:54764362 l2:0  ">
<tags></tags></type>
</result>
</results><performance>Queryid=3263159498132591711719240037627661733490256838 
user=livesplunkuser result="success" results_returned=9774 
submitted=02/19/2007:14:28:38 time_between_submission_and_execution=0.000 
execution_time=0.020 total_time=0.020</performance>
<query>SEARCH meta::all GET events::0-2 OUTPUT splunkui::2.1 
format::raw</query>
</queryResult>

executeQuery.pl

Here is a perl script that executes SOAP calls:

Note This requires XML::XPath and SOAP::Lite

#!/usr/bin/perl
use XML::XPath;
use SOAP::Lite;
$| = 1;
# URL to Splunk Server Management Port
$splunkd = "https://defiant.splunk.com:8089";
# Login to splunk instance supplied by argument and return auth token.
sub Splunk_SOAP_login {
    my ($url) = @_;
    $s = SOAP::Lite
        -> uri('urn:m2c-ManagementService')
        -> proxy($url);
    $apiData = "<call name=\"userLogin\"><params><login><username>admin</username><password>changeme</password></login></params></call>";
    $call = SOAP::Data->name('apiData' => $apiData);
    $authToken = $s->invokeAPI($call) -> result;
    return $authToken;
}
## Execute the SOAP invoke api call and return the result
sub Splunk_SOAP_invokeAPI {
    my ($url, $authToken, $callname, $data) = @_;
    $s = SOAP::Lite
        -> uri('urn:m2c-ManagementService')
        -> proxy($url);
    $apiData = "<call name=\"" . $callname . "\"><params>" . $authToken . $data . "</params></call>";
    $call = SOAP::Data->name('apiData' => $apiData);
    return $s->invokeAPI($call) -> result;
}
# Helper function to build the data portion of a query call.
sub buildQuery {
    my ($searchString) = @_;
    $qdata = "<query>SEARCH $searchString GET events::0-9 OUTPUT maxlines::14 splunkui::2.1 summary::2.1 format::all "timeformat::%m/%d/%Y, %H:%M:%S"</query><user>admin</user><queryId>1234567890</queryId>"; 
    return $qdata;
}
# Obtain an auth token.
$authToken = Splunk_SOAP_login($splunkd);
print("authToken: $authToken\n");
# Build the API call.
# change this to send a different call
# some possible choices are:
#     getLicenseInfo
#     getUserInfo
#     getHostname
#     deleteUser
$callname = "executeQuery";
# for queries, this is the search term
$term = "error";
$query = buildQuery($term);
# Make the call and print results.
$results = Splunk_SOAP_invokeAPI($splunkd, $authToken, $callname, $query);
print $results;

SOAP Message Examples

Here are examples of a full SOAP message, including the XML, SOAP body and envelope.

Login

POST https://defiant.splunk.com:18089 HTTP/1.1
Accept: text/xml
Accept: multipart/*
Accept: application/soap
Content-Length: 642
Content-Type: text/xml; charset=utf-8
SOAPAction: "urn:m2c-ManagementService#invokeAPI"
<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" 
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Body>
		<invokeAPI xmlns="urn:m2c-ManagementService">
			<apiData xsi:type="xsd:string"><call
			name="userLogin"><params><login><username>admin<
			/username><password>changeme</password></login><
		/	params></call></apiData>
		</invokeAPI>
	</soap:Body>
</soap:Envelope>

Execute Query

POST https://defiant.splunk.com:18089 HTTP/1.1
Accept: text/xml
Accept: multipart/*
Accept: application/soap
Content-Length: 925
Content-Type: text/xml; charset=utf-8
SOAPAction: "urn:m2c-ManagementService#invokeAPI"
<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Body>
		<invokeAPI xmlns="urn:m2c-ManagementService">
			<apiData xsi:type="xsd:string"><call name="executeQuery"><params>
			<auth><userId>1</userId><username>admin</username>
			<authToken>3135052749</authToken></auth>
			<query>SEARCH error GET events::0-9 OUTPUT maxlines::14
			splunkui::2.1 summary::2.1 format::all "timeformat::%m/%d/%Y,
			%H:%M:%S"</query><user>admin</user><
			queryId>1234567890</queryId></params></call>
			</apiData>
		</invokeAPI>
	</soap:Body>
</soap:Envelope>

SOAP error messages

A SOAP error message can be returned for any SOAP call. There are 3 error messages.

1.

Request Denied. You must provide correct credentials to perform this action.

You have a valid login but the action you just tried is beyond your permission level.

2.

User Session is not valid.

You provided a session token but it didn't match.

3.

No authentication provided for call.

You provided no auth on the call at all.

• Was this topic useful?
• Post a comment