Splunk® DB Connect

Deploy and Use Splunk DB Connect

Download manual as PDF

NOTE - Splunk DB Connect version 1.x reached its End of Life on July 28, 2016. Please see the migration information.
This documentation does not apply to the most recent version of DBX. Click here for the latest version.
Download topic as PDF

Troubleshooting

This topic describes how to troubleshoot common Splunk DB Connect issues.

Answers

Have questions? In addition to these troubleshooting tips, visit Questions related to Splunk DB Connect to see what questions and answers the Splunk community has about using Splunk DB Connect.

Java Bridge Server doesn't start after an upgrade of Java

An error appears in jdbrige.log indicating that Java cannot find the correct cipher suite

2015-03-18 09:37:40,269 ERROR Java process returned error code 1! Error: Initializing Splunk context... Environment: SplunkEnvironment{SPLUNK_HOME=D:Program FilesSplunk,SPLUNK_DB=D:Program FilesSplunkvarlibsplunk} Configuring Log4j... Exception in thread "main" com.splunk.config.SplunkConfigurationException: IO Error while reading configuration from Splunkd: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate) at com.splunk.config.rest.RESTAdapter.request(RESTAdapter.java:195) at com.splunk.config.rest.RESTAdapter.readConfig(RESTAdapter.java:203) at com.splunk.config.cache.CachedConfigurationAdapter.readConfig(CachedConfigurationAdapter.java:32) at com.splunk.config.cache.CachedConfigurationAdapter.readStanza(CachedConfigurationAdapter.java:40) at com.splunk.env.SplunkContext.getConfigStanza(SplunkContext.java:313) at com.splunk.env.SplunkContext.initialize(SplunkContext.java:128) at com.splunk.bridge.JavaBridgeServer.main(JavaBridgeServer.java:34) Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate) at sun.security.ssl.Handshaker.activate(Unknown Source) at sun.security.ssl.SSLSocketImpl.kickstartHandshake(Unknown Source) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source) at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source) at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source) at com.splunk.rest.Splunkd.request(Splunkd.java:212) at com.splunk.rest.Splunkd.request(Splunkd.java:98) at com.splunk.config.rest.RESTAdapter.request(RESTAdapter.java:193) ... 6 more

Fix: Starting with JDK 8u31 release, the SSLv3 protocol (Secure Socket Layer) has been deactivated and is not available by default. See the java.security.Security property jdk.tls.disabledAlgorithms in <JRE_HOME>/lib/security/java.security file.

If SSLv3 is absolutely required, the protocol can be reactivated by removing "SSLv3" from the jdk.tls.disabledAlgorithms property in the java.security file or by dynamically setting this Security property to "true" before JSSE is initialized.

http://www.oracle.com/technetwork/java/javase/8u31-relnotes-2389094.html

Java Bridge Server not running

A status error appears indicating that the Java Bridge Server is not running, and the dbx.log contains errors relating to REST keep-alive failed. Other symptoms might include searches that return column names with no or incomplete data. This typically occurs when Splunk DB Connect is running in a VM that has been suspended and restarted. Also, a stale state.xml file can prevent the Java Bridge Server from running.

Fix: If you suspend and restart a VM on which the Java Bridge Server is running, make sure to restart Splunk. Also, remove any stale state.xml files from $SPLUNK_HOME/var/lib/splunk/persistentstorage/dbx.

After upgrading to DBX 1.1.6, Java Bridge Server does not appear to be running. jbridge.log shows ERROR Java process returned error code 1! Error: Initializing Splunk context... Environment: SplunkEnvironment{SPLUNK_HOME=/opt/splunk,SPLUNK_DB=/opt/splunk/var/lib/spl unk} Configuring Log4j... Exception in thread "main" com.splunk.rest.SplunkdException: Unable to connect to Splunkd REST Service: Connection refused

Fix: Ensure the supported Oracle JRE is in use as specified in our requirements.

Note: Only the Oracle JRE is certified and supported for use with Splunk DB Connect. Customers have reported problems when starting the Java Bridge Server under alternate JREs or JDKs such as OpenJDK or IBM Java.

Input not updating

For a dbmon-tail, check the latest checkpoint value, which is stored in the $SPLUNK_DB/persistentstorage/dbx directory. ($SPLUNK_DB is the $SPLUNK_HOME/var/lib/splunk directory, if not otherwise defined.) Each input has its own directory, which is a hash of its name and a 32-character hexadecimal string. This directory typically contains these files:

  • manifest.properties has meta-information, such as the input name.
  • state.xml has the actual state in XML format.
  1. Identify the state directory.
  2. inspect the XML file.

The state file looks like this:

<list>
  <value key="latest.record_update">
    <value class="sql-timestamp">2012-12-07 04:22:25.703</value>
  </value>
</list>

Putting DBX.log in DEBUG mode

To enable debug-level logging for DBX, edit the [logging] stanza in java.conf:

[logging]
level = DEBUG
file = dbx.log
console = false
logger.com.splunk.dbx = DEBUG

Error creating PersistentValueStore

Note: You might encounter this error after upgrading from earlier DB Connect versions.

If this error appears in the jbridge.log file, you might have a corrupted persistent store file.

ERROR Java process returned error code 1! Error: Initializing Splunk context... 
Environment:
SplunkEnvironment{SPLUNK_HOME=/opt/splunk,SPLUNK_DB=/opt/splunk/var/lib/splunk} 
Configuring Log4j... [Fatal Error] :1:1: Premature end of file. Exception in thread "main" 
com.splunk.config.SplunkConfigurationException: Error creating PersistentValueStore type xstream: 
com.thoughtworks.xstream.io.StreamException:  : Premature end of file. 

To resolve this issue, remove $SPLUNK_DB/persistentstorage/dbx/global, recursively.

Issues with bad line breaking/line merging

The problem is caused by Splunk linebreak heuristics. Typically, log file data includes event timestamps, which Splunk understands. If you have timestamps in your database rows, you'll avoid linebreak issues. Be sure to set output timestamp and specify that the timestamp column is the actual timestamp column.

If you don't have timestamps in your db rows

If you don't have timestamps in your database rows, you have two options:

  • Click output timestamp and leave the timestamp column blank. Splunk outputs the current time when indexing.
  • Use the default sourcetype in the input config. Leave it blank because Splunk DB Connect uses dbmon:kv as the sourcetype (in the normal case where you're using the key-value output format). But, if you put something custom in the sourcetype field, you must tell Splunk how to linebreak for that sourcetype. Copy the props.conf settings for the default stanzas - specifically, add "SHOULD_LINEMERGE = false".

If your timestamp is not of type datetime/timestamp

Splunk DB Connect expects the timestamp column in your database to be of type datetime/timestamp. If it is not (for example, it is in format char/varchar/etc.), you can first try to convert the SQL statement into the correct type using CAST or CONVERT functions. If this method doesn't work, you can use the following workaround:

Check the Output timestamp box and specify the output.timestamp.parse.format so DB Connect can obey the timestamp output format setting. For example, if the database column EVENT_TIME has strings, such as CHAR, VARCHAR, or VARCHAR2, with values like 01/26/2013 03:03:25.255 you must specify the parse format in the appropriate copy of inputs.conf.

output.timestamp = true
output.timestamp.column = EVENT_TIME
output.timestamp.parse.format = MM/dd/yyyy HH:mm:ss.SSS

Unexpected session key expiration

A system clock change or suspend/resume cycle can cause unexpected session key expiration. To remedy the problem, restart the Splunk system using DB Connect. If it does not come back cleanly, delete the state file, $SPLUNK_DB/persistentstorage/dbx/global/state.xml, and restart the Splunk system, again.

Java bridge log file settings

By default the Python Java bridge process logs INFO-level events to the jbridge.log file, using a rolling file appender over five files for a maximum of 100M bytes.

You can create a jbridge_server.conf file in the $SPLUNK_HOME/etc/apps/dbx/local directory to override those settings.

Example jbridge_server.conf file entry:

[log]
filename=jbridge_conf.log
maxCount=10
fileSize=1000000000000
logLevel=debug

The DB Connect homepage keeps refreshing

This issue is caused by a known bug (DBX-317), which affects users that are logged in with the capabilities of a role that inherits from the default dbx_user role.

To workaround, give users the dbx_user role directly, rather than assign them a role that inherits from dbx_user

Cannot connect to any database: SSL Errors

If you cannot connect to a database and see similar errors in jbridge.log to the following, check to ensure that you are not running in FIPS mode. FIPS mode is not compliant with the jbridge service and is not supported. About FIPS Mode

ERROR Java process returned error code 1! Error: Initializing Splunk context... Environment: SplunkEnvironment{SPLUNK_HOME=/u01/splunk,SPLUNK_DB=/u01/splunk/var/lib/splunk} Configuring Log4j... Exception in thread "main" com.splunk.config.SplunkConfigurationException: IO Error while reading configuration from Splunkd: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure at com.splunk.config.rest.RESTAdapter.request(RESTAdapter.java:195) at com.splunk.config.rest.RESTAdapter.readConfig(RESTAdapter.java:203) at com.splunk.config.cache.CachedConfigurationAdapter.readConfig(CachedConfigurationAdapter.java:32) at com.splunk.config.cache.CachedConfigurationAdapter.readStanza(CachedConfigurationAdapter.java:40) at com.splunk.env.SplunkContext.getConfigStanza(SplunkContext.java:313) at com.splunk.env.SplunkContext.initialize(SplunkContext.java:128) at com.splunk.bridge.JavaBridgeServer.main(JavaBridgeServer.java:34) Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1781)

Cannot connect to Microsoft SQL server

If you cannot connect to a Microsoft SQL server, verify that you are using the correct driver, host, and port.

  • Driver: MSSQL is the correct driver to use for Microsoft SQL servers. ODBC does not work as effectively.
  • Host: To specify a host for Microsoft SQL, use a fully qualified domain name, a short name, or an IP address. Do not use the Microsoft SQL convention of <SERVERNAME\DATABASE> for the host field.
  • Port: Many Microsoft SQL Servers use dynamic ports instead of TCP/1433. Work with your database administrator to identify the correct port, or see "Verifying the port configuration of an instance of SQL Server" here.

Cannot connect to Oracle SQL Server

If you receive an error attempting to connect to an Oracle DB, note the following:

Oracle Error Codes The most common error codes are:

  • ORA-12504: TNS:listener was not given the SID in CONNECT_DATA

This error means that the SID was missing from the CONNECT_DATA configuration. To troubleshoot, check that the connect descriptor corresponding to the service name in TNSNAMES.ORA also has an SID component in the CONNECT_DATA stanza.

  • ORA-12505: TNS:listener does not currently know of SID given in connect descriptor

You are receiving this error because the listener received a request to establish a connection to the Oracle DB, but the SID for the instance either has not yet dynamically registered with the listener or has not been statically configured for the listener. Typically, this is a temporary condition that occurs after the listener has started, but before the database instance has registered with the listener. To troubleshoot, try waiting a few moments and try the connection again. You should also check which instances are currently known by the listener by executing: lsnrctl services <listener name>

  • ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

This error is because the listener received a request to establish a connection to the database. The connection descriptor received by the listener specified a service name for a service that either has not yet dynamically registered with the listener or has not been statically configured for the listener. To troubleshoot, try waiting a few moments and try the connection again. You should also check which instances are currently known by the listener by executing: lsnrctl services <listener name>

Explanation of Oracle TNS Listener and Service Names

TNS is a proprietary protocol developed by Oracle. It provides a common interface for all industry-standard protocols and enables peer-to-peer application connectivity without the need for any intermediary devices.

DBX utilizes Java (via the JDBC driver) to connect Splunk to a TNS Listener, which in turn connects to the Oracle Database. You can configure DBX to connect via the Service Name or the Oracle SID. Typically, most connectivity issues with DBX and Oracle Databases are caused by misconfiguration of the TNS Listener.

Database login error from search head pool

If you receive a database login error when using search head pooling with DB Connect:

  • Hit the distributed REST endpoints and confirm that each database connection on each search head has a unique password set. The REST endpoints are as follows:
https://<search_head_hostname>:8089/servicesNS/nobody/dbx/dbx/databases
https://<search_head_hostname>:8089/servicesNS/nobody/dbx/dbx/distributed

Problem upgrading from earlier DB Connect versions

To ensure a successful upgrade of DB Connect, we recommend that you stop the Java Bridge Server and make a backup of your local directory, as follows:

1. Stop the Splunk instance on which the Splunk DB Connect app is running.

2. Make a backup of your $SPLUNK_HOME/etc/apps/dbx directory.

3. Delete the original $SPLUNK_HOME/etc/apps/dbx directory.

4. Start the Splunk Enterprise instance.

5. Perform a fresh install of the latest version of Splunk DB Connect, as shown in steps 1-4 of Install the Splunk DB Connect App.

6. Once you have successfully installed the DB Connect app, copy all .conf files from your $BACKUP_DIR/etc/apps/dbx/local directory into your new $SPLUNK_HOME/etc/apps/dbx/local directory.

7. Start Splunk.

Note: After upgrading DB Connect, you might encounter this error creating PersistentValueStore.

Renaming rising_column breaks database input

Renaming the rising_column causes a "catch-22." If you rename the rising column, DB Connect returns an exception stating that no such column exists in the original table. If you set the rising_column to the unrenamed column name that is in the table, DB Connect returns an exception stating that there is no such field in the final output. Workaround: Do not rename the rising_column field.

Java Bridge Server does not work after upgrading JDK

If you update the version of JDK running on your Linux server without updating the version of JDK that Splunk DB Connect references in Java Home, the Java Bridge Server will not work, and an "Error getting database connection: Pool not open" message appears in the dbx.log file. Fix:

1. Open the Splunk DB Connect App.

2. Go to Settings > Splunk DB Connect configuration.

The Java setup page opens.

3. In the Java Home field, update the version of JDK in the path name, so that it matches the version of JDK currently running on the server.

inputs.conf stanzas must match in default and local directories

inputs.conf stanzas in /default and /local directories must match for DB Connect to function properly. This is because entries in default/inputs.conf are disabled by default and overridden by the corresponding stanza in local/inputs.conf.

For example, on Windows, if a user has script stanzas, such as [script://.\bin\jbridge_server.py] in default/inputs.conf and [script://D:\Splunk\etc\apps\dbx\bin\jbridge_server.py] in local/inputs.conf the passAuth parameter specified in default/inputs.conf is not inherited to local/inputs.conf and a failure occurs.

Timestamp column displays in epoch time instead of datetime

When using dbquery to query a database table in preview mode, the timestamp column displays in epoch time, instead of human readable and Splunk-recognized datetime (MM/DD/YYYY HH:MM:SS).

Workaround: Use a SQL statement in your query to convert epoch time to datetime. The specific SQL command depends on the database. For example, to convert epoch time to datetime, use the following statement:

  SELECT FROM_UNIXTIME(epoch timestamp, optional output format)

The default output is YYYY-MM-DD HH:MM:SS (DBX-748)

For more information on configuring timestamps in DB Connect, see About timestamps and database output.

Queries containing AS do not change column names as expected

When using the AS keyword in a query such as the following:

   SELECT xyz AS abc.xyz FROM jkl

DB Connect returns the column name as xyz instead of as abc.xyz. This is because the JDBC specification states that a column name is not changed by the AS keyword; it always returns the actual name of the column.

PREVIOUS
Use database search commands
  NEXT
Configuration file reference

This documentation applies to the following versions of Splunk® DB Connect: 1.0.6, 1.0.7, 1.0.8, 1.0.9, 1.0.10, 1.0.11, 1.1, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.1.6, 1.1.7, 1.2.0, 1.2.1, 1.2.2


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters