Troubleshoot the Splunk Add-on for JMX

For troubleshooting tips that you can apply to all add-ons, see Troubleshoot add-ons in Splunk Add-ons. For additional resources, see Support and resource links for add-ons in Splunk Add-ons.

Log files

Most runtime errors are written to $SPLUNK_HOME/var/log/splunk/jmx.log. Some errors can be found in $SPLUNK_HOME/var/log/splunk/splunkd.log.

You can customize the log4j settings by changing the contents in $SPLUNK_HOME/etc/apps/Splunk_TA_jmx/bin/lib/log4j.xml.

Warning messages if provided configuration might result in data duplication

The following error message appears in the ta_jmx_rh_input_data_duplication.log file when you use a server and template pair that another input is already using:

2019-03-13 16:19:01,767 WARNING pid=7704 tid=MainThread file=splunk_ta_jmx_utility.py:check_data_duplication:76 | Selected templates: [<List_of_Templates>] and servers: [<List_of_Servers>] are already present in enabled stanza: <Input_Name>. This may cause data duplication

JMX Server credential validation

The following error can be observed in UI for one of the below cases.:

• The JMX is not enabled on a Java machine.
• You create the server with invalid server credentials.
• Your Splunk platform deployment is not able to reach the service where JMX is running.

Failed to make connection with JMX server. Please ensure correct values are entered.

Third-party library warning logger messages

A third-party library initiates an error message when the Java modular input initializes. You can ignore this warning message:

org.exolab.castor.mapping.Mapping loadMapping

Missing jmx_tasks error logger messages

The following error appears in splunkd.log when you first install the add-on, initially there are no tasks configured. Once the Add-on is configured for data collection, this error no longer appears:

ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" ConfManagerException: Config file: jmx_tasks does not exist.

OutOfMemory Error messages in logs

An OutOfMemory Error in splunkd.log files when using the IBM java is thrown when the Java Virtual Machine cannot allocate an object because it is out of memory and no more memory could be made available by the garbage collector. Increase the heap size to resolve this issue:

1. Navigate to $SPLUNK_HOME$/etc/apps/Splunk_TA_jmx/bin/java_const.py

2. Change the code from:

JAVA_COMMON_ARGS = [JAVA_EXECUTABLE ,"-classpath", CLASSPATH, "-Dconfighome=" + CONFIG_HOME,"-Dsplunkhome="+ SPLUNK_HOME]

to

JAVA_COMMON_ARGS = [JAVA_EXECUTABLE, "-Xmx<maxheapsize>m" ,"-classpath", CLASSPATH, "-Dconfighome=" + CONFIG_HOME,"-Dsplunkhome="+ SPLUNK_HOME]

3. Replace the <maxheapsize> with a suitable Maximum heap size value in Megabytes (m denotes Megabytes). Do not allocate more memory to heap than the available physical memory on the machine.

4. Restart Splunk.

UnknownHost Exception logged in jmx.log

The following error might appear when you configure a new server:

com.splunk.jmx.ServerConfigValidator -22697 [P=70438:O=0:CT] ERROR [] - Failed to make connection with JMX server. Reason- defaultLocalHost:java.net.UnknownHostException: ibmwebsphere: ibmwebsphere: System error
org.omg.CORBA.INTERNAL: defaultLocalHost:java.net.UnknownHostException: ibmwebsphere: ibmwebsphere: System error
at com.ibm.rmi.corba.ORB.defaultLocalHost(ORB.java:2041) ~[?:?]
at com.ibm.rmi.corba.ORB.defaultBootstrapHost(ORB.java:2018) ~[?:?]
at com.ibm.rmi.corba.ORB.parseProperties(ORB.java:2146) ~[?:?]
at com.ibm.rmi.corba.ORB.orbParameters(ORB.java:1347) ~[?:?]
at com.ibm.rmi.corba.ORB.set_parameters(ORB.java:1295) ~[?:?]
at com.ibm.CORBA.iiop.ORB.set_parameters(ORB.java:1695) ~[?:?]
at org.omg.CORBA.ORB.init(ORB.java:401) ~[?:?]
at [jmxmodinput.jar:?]

The issue can be resolved if you add the server host to the list of the hosts in the machine where your Splunk platform instance is configured for data collection:

1. For Linux systems, the host file is present at the following path: /etc/hosts
2. For Windows systems, the host file is usually present at the following path: C:\Windows\System32\drivers\etc\hosts
3. Navigate to the host file and add the following line to add the host of the server:

<host_ip> <host_name>

4. Save your changes.