Splunk® IT Service Intelligence

Modules

Download manual as PDF

Download topic as PDF

Application Server Module troubleshooting

Verify Data Collection

Verify that the add-ons in your deployment are installed and configured correctly by checking the add-on's indices, sources or source types.

Add-on Data verification search
Tomcat index = * sourcetype = tomcat:*
Websphere index = * sourcetype = index=* sourcetype = ibm:was*

Data population can also be verified by checking to see if your tags have data.

Tag Search
appserver tag=appserver
inventory tag=appserver tag=inventory
inventory-->application tag=appserver tag=inventory tag=application
performance tag=appserver tag=performance
performance-->transactions tag=appserver tag=performance tag=transactions
performance-->runtime tag=appserver tag=performance tag=runtime
performance-->throughput tag=appserver tag=performance tag=throughput
performance-->session tag=appserver tag=performance tag=session

Verify inventory data

Index Search Data Model Search
tag=appserver
tag=inventory
|search application_server=*
|table application_server,role,vendor_product,version_number,vendor_product_runtime,vendor_product_runtime_version,full_host_name
|dedup application_server
|tstats latest(Inventory.ip_address) as ip_address,
latest(Inventory.product_family) as product_family,
latest(Inventory.vendor_product) as vendor_product,
latest(Inventory.version_number) as version_number,
latest(Inventory.vendor_product_runtime) as vendor_product_runtime,
latest(Inventory.runtime_vendor) as runtime_vendor,
latest(Inventory.vendor_product_runtime_version) as vendor_product_runtime_version,
latest(Inventory.appserver_port_number) as appserver_port_number,
latest(Inventory.root_role) as role,
latest(Inventory.full_host_name) as full_host_name,
latest(Inventory.application_server) as application_server
from datamodel=AppServer.Inventory
where Inventory.ip_address=* 
by host
| fields + application_server,role,vendor_product,version_number,vendor_product_runtime,vendor_product_runtime_version,full_host_name

Verify KPI data

Search Expected Results
tag=appserver
tag=performance
tag=session
| timechart avg(active_sessions_count) as "Active Sessions (count)"
Numbers populate on the Statistics tab with the field Active Sessions(count)
tag=appserver
tag=performance
tag=runtime
| timechart avg(cpu_utilization_percent) as cpu_util_pct
| eventstats avg(cpu_util_pct) as average
| eventstats min(cpu_util_pct) as minimum
| eventstats max(cpu_util_pct) as maximum
| eval _stats = "Range: ".round(minimum, 1)." - ".round(maximum, 1)."%, "."Average: ".round(average, 1)."%"
| table cpu_util_pct,average, minimum, maximum
Numbers populate on the Statistics tab in a table with columns cpu_util_pct, average, minimum, and maximum


Entities not found

When you are unable to locate operating system entities, see the ITSI module troubleshooting section on entities.

Troubleshooting entity appearance speed

It will take about an hour to start seeing a full sparkline chart and for KPIs to become active. Learn about entity appearance speed.

Metrics not collected

If you are unable to view your data, run the data model audit to make sure your data models are processing your data.

If you are running into issues seeing your KPI data, see ITSI module troubleshooting section on KPIs.

User Permissions

If a user encounters a permission-related obstacle, the issue could be related to their assigned role.

ITSI permissions are determined by the role that each user has. Each role offers a different set of user permissions.

To locate an existing user or role in Splunk web, use the Search bar at the top of the Users or Roles page in the Access Controls section by selecting Settings > Access Controls.

See ITSI module troubleshooting to learn about permissions within ITSI.

Learn about adding navigation to a Splunk app.

The Drilldown page stops functioning

If the drilldown page stops working while using ITSI:

  1. Launch a web browser and
  2. Use your console to debug the JavaScripts,
  3. Examine the results to see if any KPI templates are corrupted.

Tomcat lookup field error

If you have both the Tomcat and WebSphere add-ons installed on the same server, and encounter the following error about Tomcat.

    
Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'tomcat:access:log' and lookup table 'tomcat_rootdir_jmxhost_lookup'.
Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'tomcat:runtime:log' and lookup table 'tomcat_rootdir_jmxhost_lookup'. [tag-asnixdev3.sv.splunk.com]
Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'tomcat:access:log' and lookup table 'tomcat_rootdir_jmxhost_lookup'. [tag-asnixdev3.sv.splunk.com]
Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'tomcat:runtime:log' and lookup table 'tomcat_rootdir_jmxhost_lookup'.

Check the $splunk/etc/apps/Splunk_TA_ibm-was/lookups/ folder to see if the file ibm_was_inventory.csv is present. If it is not there, open the WebSphere add-on installation package, and copy the ibm_was_inventory.csv file into the $splunk/etc/apps/Splunk_TA_ibm-was/lookups/ folder.

PREVIOUS
Application Server Module data model reference table
  NEXT
About the Database Module

This documentation applies to the following versions of Splunk® IT Service Intelligence: 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.1.1, 3.1.2, 3.1.3, 3.1.4, 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.0.4, 4.1.0, 4.1.1, 4.1.2, 4.1.5, 4.2.0, 4.2.1, 4.2.2, 4.2.3, 4.3.0, 4.3.1, 4.4.0, 4.4.1


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters