Share performance and usage data in Splunk Enterprise

Splunk LLC collects critical data so that we can enhance the value of your investment in Splunk software.

We use this data to optimize your deployment, prioritize our features, improve your experience, notify you of patches, and develop high quality product functionality.

Changes in version 9.1.0

There are minor changes to Splunk data collection practices in version 9.1.0, mainly for the Splunk Assist service. Data collection remains on by default. For more information on why Splunk changed its policy to enable the collection of usage data, see the 8.0 version of this topic.

The support usage data that Splunk collects for Splunk Assist and for telemetry are the same. The targets for these data sources, however, are different. You might need to update any firewall settings that you have before you can use Splunk Assist, even though the Splunk platform can send support usage data back to Splunk.

You can still opt out of data sharing at any time, but if you do, you cannot use the Splunk Assist service, which requires that data sharing is active. See How to opt out.

To learn more about Splunk Assist, see About Splunk Assist in the Monitoring Splunk Enterprise Manual.

Benefits of sharing data with Splunk

When you share data with Splunk, you receive the following benefits:

Improved product quality. By collecting accurate information about the topology decisions and deployment scale used by our customers, we can replicate those topology configurations and scale in our internal testing, helping us improve your product experience.
Timely notification of known bugs, version incompatibilities, and configuration issues. When you share data about the product versions you have deployed, we can provide accurate messages and support to help you with bugs, upgrade tasks, version compatibility problems, and other configuration issues you might experience.
Relevant feature enhancements. We prioritize what features to develop and enhance first based on the features customers use the most. By sharing your data, you influence these data-driven decisions in favor of the features you use at your organization.
You can use the Splunk Assist service to monitor your deployment in accordance with Splunk best practices for security, performance, and configuration.

For more information, see How Splunk uses the data it collects.

What data Splunk collects

The following table summarizes the data that your Splunk platform deployment sends to Splunk when you enable data collection. Follow the links to see examples of this data.

Type of data	Description	Examples
Aggregated usage data	Includes features used, deployment topology, and performance metrics in both the platform and apps. This data is not associated with your license ID. You must enable Aggreated usage data to use the Splunk Assist service.	Aggregated usage data examples App usage data examples
Support usage data	Support usage data is the same as the aggregated usage data, but the license ID remains associated with your data when it reaches Splunk. You must enable support usage data to use the Splunk Assist service.	Aggregated usage data examples App usage data examples
License usage data	Includes your license ID, active license group and subgroup, total license stack quota, total license pool consumption, license stack type, license pool quota, license pool consumption.	License usage data examples
Software version data	Includes the version of Splunk Enterprise and of each installed app, along with relevant metadata about deployment architecture.	Software version data examples

Splunk does not collect the contents of your indexed data.

Some cloud and hybrid products modify the kinds of data that Splunk collects. When that happens, a separate agreement or notification states how the data collection differs for that product.

For instructions on how to view the data that your deployment collects and sends to Splunk, see View what data is sent from your deployment.

Examples of data sent to Splunk

Aggregated usage, support usage, and license usage data is sent to Splunk as a JSON packet that includes information like the component name and deployment ID, in addition to the data for the specific data collection component. The deploymentID is unique to a deployment and does not change on upgrade or even after uninstall and reinstall of Splunk Enterprise on the same machine.

Here is an example of a complete JSON packet:

{
component: deployment.app
   data: { [-]
     enabled: true
     host: 878e7b21bf98580dbdb4ed3baf6c35d78aa5bc3d3c824eb8714a313c
     name: search
     version: 8.0.0
   }
   date: 2019-09-23
   deploymentID: d6d8e776-a8d3-5467-a03b-375577646cbb
   executionID: 2FC293C59049AC0D44B677D3A9D786
   timestamp: 1569294102
   transactionID: 4E1CFC7E-BE9F-355D-7DDE-D4F8D5E4852D
   version: 3
   splunkVersion: 8.1.2
   visibility: anonymous,support
}

The following tables list the component names, descriptions, and an example of what data is collected for that component. For ease of use, the examples for aggregated usage and license data show examples of only the data field from the JSON object.

Aggregated usage data examples

The following example demonstrates the data sent to Splunk when sharing of aggregated usage data is enabled.

Component	Description	Example
`app.RapidDiag.cliAccessMetrics`	RapidDiag CLI interface usage statistics.	{ [-] app: splunk_rapid_diag component: app.RapidDiag.cliAccessMetrics data: { [-] action: 'run' count: 2 mode: 'templates' result: 0 } deploymentID: 654b5421-eec2-5229-9fc6-5f065e00f9f5 eventID: 8BEB3B43-FC9E-47F3-8FFF-BA6E1D2CF425 executionID: C7212C53-51C7-4CB5-9316-1A3F6815594F optInRequired: 3 timestamp: 1605611221 type: aggregate visibility: [ [-] anonymous support ]
`app.RapidDiag.uiAccessMetrics`	RapidDiag UI interface usage statistics.	{ [-] app: splunk_rapid_diag component: app.RapidDiag.uiAccessMetrics data: { [-] count: 1 status: 200 uri_path: /en-GB/app/splunk_rapid_diag/data_collection user: 8c6976e5b541 } deploymentID: 654b5421-eec2-5229-9fc6-5f065e00f9f5 eventID: 4A5E61B6-C5C8-47F7-A6C9-AA4409E3AB5D executionID: 07237CFC-6663-44D6-9F12-82D273A4AF06 optInRequired: 3 timestamp: 1605540721 type: aggregate visibility: [ [-] anonymous support ]
`app.RapidDiag.executionMetrics`	RapidDiag task execution statistics.	{ [-] app: splunk_rapid_diag component: app.RapidDiag.executionMetrics data: { [-] count: 10 metricName: dd1cd3d60a28 status: Success type: collector } deploymentID: 654b5421-eec2-5229-9fc6-5f065e00f9f5 eventID: AA2EA083-F71C-473A-B19D-0C0993FCB520 executionID: B0FFB679-2745-4AA6-AF99-71999ED514BF optInRequired: 3 timestamp: 1605611641 type: aggregate visibility: [ [-] anonymous support ] app: splunk_rapid_diag component: app.RapidDiag.executionMetrics data: { [-] count: 10 name: Slow search performance status: Success type: task } deploymentID: 654b5421-eec2-5229-9fc6-5f065e00f9f5 eventID: A6253B1F-7C26-4656-AE8F-848AC125783F executionID: B0FFB679-2745-4AA6-AF99-71999ED514BF optInRequired: 3 timestamp: 1605611641 type: aggregate visibility: [ [-] anonymous support ]
`app.session.coreLibrarySettings.save`	Tracks if certain core library settings are toggled on or off.	{ [-] component: app.session.coreLibrarySettings.save data: { [-] app: search page: core_library_settings setting: enable_jQuery2 value: False } deploymentID: 942a8692-dce5-9b6f-4bd4-f4811c20328f eventID: 899f8692-dce5-9b6f-4bd4-f4811c20328f experienceID: a6c7710b-6822-394e-3292-812eef0d265a optInRequired: 3 timestamp: 1617218044 userID: 40babbddf86516c5864e524a6e3b66f38ca835e56a112d0ab0407857ffd0e45c version: 4 visibility: anonymous,support }
`app.session.createNewDashboardDialog.interact`	General telemetry collected when a new dashboard is created.	{ [-] "component": "app.session.createNewDashboardDialog.interact", "data": { "action": "createNewDashboard", "editId": true, "hasDescription": false, "dashboardType": "udf", "layout": "absolute", "sharing": "user", "status": "success", "app": "search", "page": "dashboards" }, }
`app.session.dashboard.load`	Dashboard characteristics, generated as session data when a dashboard loads.	{ [-] app: search dashboard: { [-] autoRun: false hideAppBar: false hideChrome: false hideEdit: false hideExport: false hideFilters: false hideSplunkBar: false hideTitle: false isScheduled: false isVisible: true numCustomCss: 0 numCustomJs: 0 refresh: 0 submitButton: false theme: light version: 1.0 isDeprecatedXMLDashboard: true } elementTypeCounts: { [-] area: 1 column: 1 line: 1 singlevalue: 8 statistics: 10 } formInputTypeCounts: { [-] } layoutType: row-column-layout numElements: 21 numFormInputs: 0 numPanels: 21 numPrebuiltPanels: 0 numSearches: 21 page: network_insights searchTypeCounts: { [-] inline: 21 } }
`app.session.dashboard.interact`	Whether a user pressed Cancel or Continue for the URL warning modal.	{ [-] "component":"app.session.dashboard.interact", "data":{ "type":"urlWarningModal", "action":"cancel", "app":"search", "page":"giulia_sxml" },
`app.session.dashboard.error`	If an asynchronous error occurred in a CustomJS script used by a dashboard.	{ [-] data: { [-] app: search errorType: customJSError page: kieran123 }
`app.session.dashboard.telemetry`	General telemetry collected when adding and configuring dashboard elements.	{ [-] "component": "app.session.dashboard.telemetry", "data": { "pageAction": "scheduledExport.save", "success": true, "enabledInitially": false, "enabledAtSave": true, "cronSchedule": "0 18 * * *", "emailCountTo": 1, "emailCountCC": 0, "emailCountBCC": 0, "emailSubjectLength": 22, "emailMessageLength": 17, "includeLinkInitially": false, "includeLinkAtSave": false, "app": "search", "page": "dashboards" } }
`app.session.dataactions.interact`	User interactions in the dataactions UI.	{ [-] component:app.session.dataactions.interact data: { [-] action: save app: $SPLUNK_PLATFORM editType: new externalDestinationCount: 0 name: 9dd8c74a33ee89cb4fbe82deee2273ec6b8262370225b377188d3cad8f8c1376 page: manager/search/ingest_rulesets ruleCount: 1 ruleCountsByAction: { [-] filter: 1 } sourcetype: 65935aef8944a30f5046ba0159cfa2ddcb2992846bc1efad3a53a432427e8279 } deploymentID: 825dc0d6-5430-5ef8-9b69-2c54adad7f1a eventID: 7f49d8ee-5b9c-c401-2cd0-81bc811a25a1 experienceID: 9b66912a-86df-efa0-e099-ee9ccf4e835e optInRequired: 3 splunkVersion: 9.0.0 timestamp: 1652730582 userID: d2bb23947441c280c5cf8fee0df81614294fcd9131aa21638b8137d655e71a68 version: 4 visibility: anonymous,support }
`app.session.dataactions.load`	Number of rulesets and type of deployment.	{ [-] component: app.session.dataactions.load data: { rulesetCount: 2 deploymentType: cluster-master } date: 2018-10-26 deploymentID: 99b6ffd8-2e80-5e3b-905c-8c6f6fd743a0 executionID: F0AE995E8653D768A360E73BE3F544 timestamp: 1540570045 transactionID: 89F7329E-86AD-BBFD-034F-209CB8A06F05 version: 3 visibility: anonymous,support }
`app.session.datainteractions.load`	Apps installed per Splunk instance.	{ [-] data: { [-] rulesetCount: 2 deploymentType: cluster-master } date: 2018-10-26 deploymentID: 99b6ffd8-2e80-5e3b-905c-8c6f6fd743a0 executionID: F0AE995E8653D768A360E73BE3F544 timestamp: 1540570045 transactionID: 89F7329E-86AD-BBFD-034F-209CB8A06F05 version: 3 visibility: anonymous,support }
`app.session.globalBanner.error`	Unexpected error responses from GET/POST requests to the global banner endpoint, and the status code.	{ [-] app: $SPLUNK_PLATFORM page: manager/launcher/global_banner responseText: {"messages":[{"type":"ERROR","text":"Argument \"unknown\" is not supported by this handler."}]} status: 400 }
`app.session.globalBanner.interact`	Tracks when a user clicks a banner link.	{ [-] action: link click app: $SPLUNK_PLATFORM page: manager/launcher/global_banner }
`app.session.html_dashboard`	Count the number of HTML dashboards in the Splunk Enterprise instance.	{ [-] component: app.session.html_dashboard data: { [-] app: search page: jquery_staging count: 21 } deploymentID: 942a8692-dce5-9b6f-4bd4-f4811c20328f eventID: 899f8692-dce5-9b6f-4bd4-f4811c20328f experienceID: a6c7710b-6822-394e-3292-812eef0d265a optInRequired: 3 timestamp: 1617218044 userID: 40babbddf86516c5864e524a6e3b66f38ca835e56a112d0ab0407857ffd0e45c version: 4 visibility: anonymous,support }
`app.session.html_dashboard.load`	Track the number of times an HTML dashboard is loaded.	{ [-] component: app.session.html_dashboard.load data: { [-] app: search page: network_insights } deploymentID: 942a8692-dce5-9b6f-4bd4-f4811c20328f eventID: 899f8692-dce5-9b6f-4bd4-f4811c20328f experienceID: a6c7710b-6822-394e-3292-812eef0d265a optInRequired: 3 timestamp: 1617218044 userID: 40babbddf86516c5864e524a6e3b66f38ca835e56a112d0ab0407857ffd0e45c version: 4 visibility: anonymous,support }
`app.session.metrics.interact`	Track the type of filter the user set on a chart.	{ [-] accessor: METRICS action: SERIES_FILTER_ADD app: search chartType: line context: analysis customInfo: { [-] app: metrics-analysis commitHash: 5b0687f037c02ab76c3adc2391e80d84887d2b3e version: 2.28.0 } numCustomFilters: 1 numFilters: 1 numHostFilters: 0 numIndexFilters: 0 numIndexRefLines: 0 numMeasures: 1 numSeries: 1 numSourceTypeFilters: 0 numStaticRefLines: 0 numTimeRangeRefLines: 0 numTimeShiftRefLines: 0 page: analytics_workspace seriesHasSplit: false seriesId: 264aa232-2d23-47c0-8a0e-9ee641465d44 type: view/UPDATE_SERIES value: { [+] } viewId: v27f16248-701c-4fe2-b79e-27462e15861c }
`app.session.metrics.process`	De-identified chart configuration data related to the queries sent by workspace charts.	{{ [-] action: EXECUTE_QUERY app: search context: analysis customInfo: { [-] app: metrics-analysis commitHash: 50bd435d736fd97bb0a7125221bab4bce3b14975 splunkVersion: 8.1.0 version: 2.28.0 } elapsed: 232 page: analytics_workspace query: { [-] series: [ [-] { [-] accessor: METRICS aggregation: avg axis: left filters: 1 refLines: [ [-] { [-] aggregation: max includeValueInLabel: true timeRange: null timeShift: -1d type: indexDataAggregation } ] span: 10s split: { [-] limit: 5 type: top } timeshift: -30m } ] timeRange: { [-] earliest: 1596751969.139 latest: 1596755569.139 } } requestId: 00961132-3d15-45a2-9d69-0624b16a9009 status: completed viewId: v69289f5f-c33c-4161-9281-53724a9aa768 }
`app.session.page.interact`	Tracks user interactions with search, reports, alerts, data models, tags, lookups, and search macros.	{ [-] action: Edit Permissions - Save app: search custom: { [+] } page: dataset }
`app.session.page.load`	Tracks loads and whether web services are supported, generated as session data when a page loads.	{ [-] allowWebService: true app: $SPLUNK_PLATFORM page: manager/search/adddata }
`app.session.pageview`	Page view session data, generated whenever a user visits a new page.	{ [-] app: launcher page: home }
`app.session.pivot.interact`	Changes to pivots, generated as session data when a user makes a change to a pivot.	{ [-] app: search context: pivot eventAction: change eventCategory: PivotEditorReportContent eventLabel: Pivot - Report Content eventValue: { [-] transient: true } numAggregations: 1 numColumnSplits: 0 numCustomFilters: 0 numRowSplits: 1 page: pivot reportProps: { [-] display.general.type: visualizations display.statistics.show: 1 display.visualizations.charting.chart: area display.visualizations.charting.chart.rangeValues: [0,30,70,100] display.visualizations.charting.gaugeColors: ["0x53a051","0xf8be34","0xdc4e41"] display.visualizations.charting.legend.placement: none display.visualizations.show: 1 display.visualizations.singlevalue.rangeColors: ["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"] display.visualizations.singlevalue.trendInterval: auto display.visualizations.type: charting earliest: -24h@h latest: now windowedEarliest: 2019-09-23T03:00:00.000+00:00 windowedLatest: 2019-09-24T03:58:52.000+00:00 } }
`app.session.pivot.load`	Pivot characteristics, generated as session data when a pivot loads.	{ [-] app: search context: pivot eventAction: load eventCategory: PivotEditor eventLabel: Pivot - Page numAggregations: 1 numColumnSplits: 0 numCustomFilters: 0 numRowSplits: 1 page: pivot reportProps: { [-] display.general.type: visualizations display.statistics.show: 1 display.visualizations.charting.chart: area display.visualizations.charting.chart.rangeValues: [0,30,70,100] display.visualizations.charting.gaugeColors: ["0x53a051","0xf8be34","0xdc4e41"] display.visualizations.charting.legend.placement: none display.visualizations.show: 1 display.visualizations.singlevalue.rangeColors: ["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"] display.visualizations.singlevalue.trendInterval: auto display.visualizations.type: charting earliest: -24h@h latest: now windowedEarliest: 2019-09-23T03:00:00.000+00:00 windowedLatest: 2019-09-24T03:58:52.000+00:00 } }
`app.session.roles.srchFilter`	Event actions on the authoritzation/roles page of Splunk Web	{ [-] app: $SPLUNK_PLATFORM context: authorization/roles eventAction: CreateEditRole eventCategory: SrchFilterInRoles eventLabel: Search Filter in role - admin eventValue: * page: manager/launcher/authorization/roles }
`app.session.rum.mark`	Track performance of the first meaningful paint for the global banner settings page and the view itself, when enabled.	{{ [-] app: $SPLUNK_PLATFORM hero: Global Banner Settings - First meaningful paint page: manager/launcher/global_banner sourceLocation: Global Banner Settings - First meaningful paint timeSinceOrigin: 6917.774999994435 transactionId: 2da6cc30-6880-11ea-a7ac-5ff240bf600d }
`app.session.rum.measure`	Track performance of the first meaningful paint for the global banner settings page and the view itself, when enabled.	{ [-] app: $SPLUNK_PLATFORM duration: 6917.774999994435 fromSourceDurations: { [+] } fromSourceLocation: origin hero: Global Banner Settings - First meaningful paint page: manager/launcher/global_banner timeSinceOrigin: 6917.774999994435 toSourceLocation: Global Banner Settings - First meaningful paint transactionId: 2da6cc30-6880-11ea-a7ac-5ff240bf600d }
`app.session.search.interact`	Search page interactions, session data generated by each user interaction with the search page.	{ [-] app: search context: search eventAction: submit eventCategory: CreateReportDialog eventLabel: Search App - Actions eventValue: success page: search reportProps: { [-] dispatch.sample_ratio: 1 display.events.table.sortDirection: asc display.general.type: statistics display.page.search.mode: smart display.prefs.events.offset: 0 display.prefs.statistics.offset: 0 display.statistics.format.0: display.statistics.format.0.colorPalette: display.statistics.format.0.colorPalette.colors: display.statistics.format.0.field: display.statistics.format.0.scale: display.statistics.format.0.scale.thresholds: display.statistics.sortColumn: Number of Users display.statistics.sortDirection: asc display.visualizations.charting.chart: bar earliest: -24h@h latest: now workload_pool: } }
`app.session.session_start`	Session data generated when a user is first authenticated. Contains the deploymentID (identifier for deployment), eventID (identifier for this specific event), experienceID (identifier for this session), userID (hashed username), data.guid (GUID for instance serving the page).	{ [-] app: launcher browser: Chrome browserVersion: 68.0.3440.106 device: Linux x86_64 guid: 0C4C7528-375A-4DA5-ABF8-09189051BB51 locale: en-US os: Linux osVersion: not available page: home splunkVersion: 8.0.0 }
`app.session.tableUI.interact`	Tracks interactions on the Table UI page.	{ [-] action: create_table_view app: search location: datasets listing page page: datasets }
`app.session.template.load`	Tracks the number of times users access HTML template files that Splunk Enterprise no longer uses.	{ [-] app: asdf page: search template: test-example }
`app.session.udf.telemetry`	General telemetry collected on visualization usage and settings.	{ [-] "component": "app.session.udf.telemetry", "data": { "pageAction": "dashboard.initialize", "metadata": {}, "udfVersion": "20.3.1", "definition": { "visualizations": { "viz_2aae822a03cb3f7c58a43c04652ee908": { "type": "viz.column", "options": {}, "titleLength": 13, "descriptionLength": 26 }, "viz_3a1a36fecbc0b5b46b5cb8777756ea6c": { "type": "viz.singlevalueicon", "options": { "showValue": false, "icon": true } }, "viz_cf5bd9532cfe6d8619132f9bb11cefd5": { "type": "viz.rectangle" }, "viz_36b6e66b1475b0e0677676b947f1d884": { "type": "viz.singlevalue", "options": {}, "titleLength": 13, "descriptionLength": 24 }, "viz_f3479a853843e0e72405cc99fc9fc810": { "type": "viz.text", "options": { "content": true } } }, "inputs": {}, "layout": { "globalInputs": [], "type": "absolute", "options": {}, "structure": [ { "item": "viz_2aae822a03cb3f7c58a43c04652ee908", "type": "block", "position": { "x": 0, "y": 0, "w": 300, "h": 300 } }, { "item": "viz_3a1a36fecbc0b5b46b5cb8777756ea6c", "type": "block", "position": { "x": 330, "y": 0, "w": 250, "h": 250 } }, { "item": "viz_cf5bd9532cfe6d8619132f9bb11cefd5", "type": "block", "position": { "x": 640, "y": 40, "w": 150, "h": 160 } }, { "item": "viz_36b6e66b1475b0e0677676b947f1d884", "type": "block", "position": { "x": 10, "y": 340, "w": 250, "h": 250 } }, { "item": "viz_f3479a853843e0e72405cc99fc9fc810", "type": "block", "position": { "x": 370, "y": 270, "w": 310, "h": 60 } } ] }, "descriptionLength": 0, "titleLength": 44 }, "app": "splunk-dashboard-studio", "page": "_do_not_edit_delete_telemetryreviewdashboard" } }
`app.splunk_monitoring_console`	Determines whether splunk_monitoring_console is enabled. If enabled, determines whether the mode is standalone or distributed.	{ [-] component: app.splunk_monitoring_console.info data: { disabled: 1 mode: standalone mc_auto_config: disabled role_list: ["license_master", "license_manager", "cluster_master", "cluster_manager", "search_head", "kv_store"] } date: 2018-10-26 deploymentID: 99b6ffd8-2e80-5e3b-905c-8c6f6fd743a0 executionID: F0AE995E8653D768A360E73BE3F544 timestamp: 1540570045 transactionID: 89F7329E-86AD-BBFD-034F-209CB8A06F05 version: 3 visibility: anonymous,support }
`assist-app.appVersion.<appId>`	Splunk Assist - App Assist	{ [-] "name": "assist-app.appVersion.<appId>", "category": "apps", "entityID": "<search-head>" "entityType": "search-head", "status": "critical" \| "warning" \| "conform", "updatedAt": "<timestamp>", "details": { "installedVersion": "<release version of app>", "latestVersion": "<latest version on Splunkbase>", } } }
`assist-certificate.expiry`	Splunk Assist - Certificate Assist	{ [-] name: assist-certificate.expiry displayName: "Certificate expiration" category: "availability" \| "security" \| "performance" \| "apps" entityID: "data_034" entityType: "indexer" status: "critical" \| "warning" \| "conform" updatedAt: timestamp previousStatus: "critical" \| "warning" \| "conform" version: <version> details: { "expiry" : <timestamp>, "subject" : <subject dn>, "serial" : <serial number>, "fingerprint" : <fingerprint>, "issuer" : <issuer dn> } }
`assist-app.appVersion.<appId>`	Splunk Assist - Config Assist	{ [-] "name": "assist-config.<file>.<stanza>.<property>", "entityID": "<splunk_server>", "entityType": "search-head", "status": "critical" \| "warning" \| "conform", "details": { "file": "<file>", "stanza": "<stanza>", "property": "<property>", "valueType": "bool" \| "string" \| "int" \| "list", "currentValue": "<current_property_value>", "expectedValue": "<expected_property_value>" } } }
`assist-app.telemetry.deployment_info`	Splunk Assist - Information about the customer's deployment.	{ "deployment_info":[ { "os_name":"Linux", "splunk_version":"9.2.4", "cpu_arch":"x86_64", "host_name":"mysearchhead", "server_roles":[ "indexer", "license_master" ] } ], "event_time":1697747489184875, "event_type":"deployment_info" }
`assist-app.telemetry.error`	Splunk Assist - Unexpected package errors.	{ "event_time":1697746213447146, "event_type":"error", "error":"assist service call returned with non-success response code", "message":"Indicator send failed (SCS)" }
`assist-app.telemetry.panic`	Splunk Assist - Package crash information. Stack trace.	{ "event_time":1697746213447146, "event_type":"panic", "panic":"invalid memory address or nil pointer dereference", "trace":"goroutine 19 [running]:\ncd.splunkdev.com/beam/go-package/plugin.runPackagePeriodically.func1.1()\n\t/Users/abbys/go/src/cd.splunkdev.com/beam/go-package/plugin/package_plugin.go:112 +0x6e\npanic({0x15e9b00?, 0x1b94b90?})\n\t/usr/local/Cellar/go/1.21.1/libexec/src/runtime/panic.go:914 +0x21f\ncd.splunkdev.com/beam/go-package/splunkd.SearchWithAuth({0x17922d0, 0xc0000ec150}, {0x0, 0x0}, {{0x16ceeb3?, 0x104ec06?}, {0x0?, 0x2?}}, {0x15aa840, 0xc0000ac6f0}, ...)\n\t/Users/abbys/go/src/cd.splunkdev.com/beam/go-package/splunkd/search.go:46 +0x2a5\ncd.splunkdev.com/beam/go-package/splunkd.Search(...)\n\t/Users/abbys/go/src/cd.splunkdev.com/beam/go-package/splunkd/search.go:27\ncd.splunkdev.com/beam/go-package/telemetry.DeploymentInfoEvent({0x17922d0, 0xc0000ec150}, 0xc0000f61c0)\n\t/Users/abbys/go/src/cd.splunkdev.com/beam/go-package/telemetry/telemetry.go:97 +0xaa\ncd.splunkdev.com/beam/go-package/plugin.runPackagePeriodically.func1(0xc0000aa540, {0x17922d0, 0xc0000ec150})\n\t/Users/abbys/go/src/cd.splunkdev.com/beam/go-package/plugin/package_plugin.go:129 +0x65\ncd.splunkdev.com/beam/go-package/plugin.runPackagePeriodically({0x17922d0, 0xc0000ec150}, 0xc0000aa540)\n\t/Users/abbys/go/src/cd.splunkdev.com/beam/go-package/plugin/package_plugin.go:132 +0x58\ncd.splunkdev.com/beam/go-package/plugin.TestLoggingPanicInfoFromPackageCodeInDebugMode.func1()\n\t/Users/abbys/go/src/cd.splunkdev.com/beam/go-package/plugin/package_plugin_test.go:147 +0x1f\ngithub.com/stretchr/testify/assert.didPanic(0x100e15d?)\n\t/Users/abbys/go/src/cd.splunkdev.com/beam/go-package/vendor/github.com/stretchr/testify/assert/assertions.go:1158 +0x82\ngithub.com/stretchr/testify/assert.NotPanics({0x49b1a908, 0xc00009aea0}, 0xc0000d60a0, {0xc0001c5f28, 0x1, 0x1})\n\t/Users/abbys/go/src/cd.splunkdev.com/beam/go-package/vendor/github.com/stretchr/testify/assert/assertions.go:1229 +0x72\ngithub.com/stretchr/testify/require.NotPanics({0x178eda0, 0xc00009aea0}, 0xc00030cf28?, {0xc00030cf28, 0x1, 0x1})\n\t/Users/abbys/go/src/cd.splunkdev.com/beam/go-package/vendor/github.com/stretchr/testify/require/require.go:1582 +0x85\ncd.splunkdev.com/beam/go-package/plugin.TestLoggingPanicInfoFromPackageCodeInDebugMode(0x0?)\n\t/Users/abbys/go/src/cd.splunkdev.com/beam/go-package/plugin/package_plugin_test.go:146 +0x23a\ntesting.tRunner(0xc00009aea0, 0x16ee4b8)\n\t/usr/local/Cellar/go/1.21.1/libexec/src/testing/testing.go:1595 +0xff\ncreated by testing.(*T).Run in goroutine 1\n\t/usr/local/Cellar/go/1.21.1/libexec/src/testing/testing.go:1648 +0x3ad\n" }
`assist-app.telemetry.run_stats`	Splunk Assist - Metrics on a single package run.	{ "event_time":1697748088496056, "event_type":"run_stats", "indicator_count":23, "memory":21902, "memory_delta":5, "run_duration_ms":534 }
`scripted_inputc.telemetry`	Describes how much data is ingested through scripted input.	app: "scripted_input" version: none bytes: number of bytes ingested { [-] app: component: scripted_inputc.telemetry data: { [-] app: scripted_input bytes: 7645634 version: no version } deploymentID: 18393d55-3552-546c-a5ab-61a96a04ae04 eventID: 367E743C-D629-4B25-B46A-78447116F3A4 executionID: 319FB159-0B47-4CA0-B29D-4CD0EDDF0DCF optInRequired: 1 timestamp: 1586974636 type: event userID: 574f5debd4e54c49ef018a6e1bde0379df499a23a865ab83e8d23d1170256f40 visibility: [ [-] anonymous support ] }
`cherrypy.load`	How frequently CherryPy routes are used.	{ [-] app: search component: cherrypy.route.load data: { [-] class: ViewController file_path: controllers/view.py route: /:app/:view_id splunkVersion: 20220805 } deploymentID: 06b3d792-4e28-5402-97d4-54bab09eee0f eventID: 55E95327-6505-423A-B1F8-EA90946977C3 executionID: 8ED4CAB4-DACF-4F9D-A631-4084DDD5FC4E optInRequired: 3 timestamp: 1663796731 type: event userID: da69a92a70c0997c4db33654b7621445d38aab4c8423f2efdc4249f0b4675e94 visibility: [ [-] anonymous support ] }
`deployment.app`	Apps installed on search head and peers.	{ [-] enabled: true host: 878e7b21bf98580dbdb4ed3baf6c35d78aa5bc3d3c824eb8714a313c name: search version: 8.0.0 }
`deployment.clustering.indexer`	Host name of an indexer, replication factor, and search factor for indexer cluster.	{ [-] enabled: false host: 06d3392e0644587c3c3131833c81bfa6a7be78361e35e2ba8edf9c92 timezone: -0700 }
`deployment.clustering.member`	Indexer cluster member status.	{ [-] master: 1b83dc9e131f02b53329dfc1d3700aea92dd8223a22325d274e5aa3a member: { [-] guid: 14B1E1C3-ABD1-4D02-88D5-3A6964EF8376 host: 942796f349f59b3ae64b47e507299b64b9a638fc9fc7a2580863f951 status: Up } site: default }
`deployment.clustering.searchhead`	Indexer cluster and search head connection status.	{ [-] master: 1b83dc9e131f02b53329dfc1d3700aea92dd8223a22325d274e5aa3a searchhead: { [-] guid: 141D5E4A-3C5C-4051-B2DB-E679027A0D57 host: f7724a2690f17f0fe3ea97418c92fffde62a890b517261377b1060f4 status: Connected } site: default }
`deployment.distsearch.peer`	Distributed search peer status.	{ [-] host: 33b1957bfe1d0f7d3aac34e8655cf49f74375fb5043cb756f9a48405 peer: { [-] guid: 676F6738-BA57-44EC-94F0-A6821739DF8C host: 76e4ed3636a6f4dc9737d119fde51e0007713c7f87af7acf0dc057a7 status: Up } }
`deployment.forwarders`	Forwarder architecture: Number of hosts, number of forwarder instances, OS/version, CPU architecture, Splunk Enterprise version, distribution of forwarding volume	{ [-] architecture: x86_64 bytes: { [-] avg: 632367800 max: 689339847 min: 602231091 p10: 602891365 p20: 603551640 p30: 604211914 p40: 604872189 p50: 605532463 p60: 622293940 p70: 639055417 p80: 655816893 p90: 672578370 } hosts: 3 instances: 3 os: Linux splunkVersion: 8.0.0 type: full }
`deployment.httpEventCollector`	Describes how much data is ingested through HEC for Splunk apps, add-ons, and connectors.	{ [-] app: component: deployment.httpEventCollector data: { [-] app: stream333 bytes: 50 version: 3.1 } deploymentID: 18393d55-3552-546c-a5ab-61a96a04ae04 eventID: 367E743C-D629-4B25-B46A-78447116F3A4 executionID: 319FB159-0B47-4CA0-B29D-4CD0EDDF0DCF optInRequired: 1 timestamp: 1586974636 type: event userID: 574f5debd4e54c49ef018a6e1bde0379df499a23a865ab83e8d23d1170256f40 visibility: [ [-] anonymous support ] }
`deployment.index`	Index type and configuration. Includes indicator of whether a metrics index has subsecond search capability.	{ [-] app: search buckets: { [-] cold: { [-] count: 0 events: 0 sizeGB: 0 } coldCapacityGB: unlimited homeCapacityGB: unlimited homeEventCount: 871 hot: { [-] count: 0 max: 3 sizeGB: 0 } thawed: { [-] count: 0 events: 0 sizeGB: 0 } warm: { [-] count: 6 sizeGB: 0 } } host: 6aac2d36b0f11492299b161a6c5a4f79451708e195b98a5dbaa47b9b name: uba_alarms timeResolution: sec total: { [-] buckets: 6 currentDBSizeGB: 0 events: 871 maxDataSizeGB: 500 maxTime: 1568987048 minTime: 1567603567 rawSizeGB: 0 } type: event }
`deployment.licensing.slave`	License slaves.	{ [-] master: 33b1957bfe1d0f7d3aac34e8655cf49f74375fb5043cb756f9a48405 slave: { [-] guid: 1E7D1EA4-9E76-410B-825F-36CDA037F377 host: 33b1957bfe1d0f7d3aac34e8655cf49f74375fb5043cb756f9a48405 pool: auto_generated_pool_enterprise } }
`deployment.node`	GUID, host, number of virtual and physical cores, CPU architecture, memory size, storage (partition) capacity, OS/version, Splunk Enterprise version	{ [-] cpu: { [+] } guid: 991BECEF-7F25-442D-B388-FF5A5AED16C3 host: cbefb1beb9ca9908007643320dec0ab0b345b51fd2f85ab7eec38370 memory: { [-] capacity: 32655630402 utilization: { [-] avg: 0.67 max: 0.74 min: 0.5 p10: 0.6 p20: 0.62 p30: 0.64 p40: 0.66 p50: 0.67 p60: 0.69 p70: 0.7 p80: 0.71 p90: 0.72 } } os: Linux osExt: Linux osVersion: 4.15.0-1031-aws partitions: [ [-] { [-] capacity: 208111882207 fileSystem: ext4 utilization: 0.91 } ] splunkVersion: 8.0.0 }
`deployment.shclustering.member`	Search cluster member status.	{ [-] captain: 208999515adad3c46696443afe61049c8f8bfe56b6330feadbc64b48 member: { [-] guid: 45B3EA5E-4868-4243-9BEA-109C2F76F02A host: 258a814c13167915bedd945acd0f5e16c058a8b1bab8972206f82120 status: Up } site: default }
`htmlcleaner.dashboard`	General telemetry collected on CSS tag usage.	{ [-] data: { app: search page: network_insights sanitizedTags: [ "DIV", "H1", "SPAN" ], inlineStyles: [ { type: "StyleAttribute", element: "div", properties: [ "background-color", "width" ] }, { type: "StyleElement", rulesets: [ { properties: [ "background-color", "content", "color" ] }, { properties: [ "width" ] } ] } ] } }
`instrumentation.performance`	Performance of instrumentation queries.	{ [-] instance_type: Single queries: [ [-] { [-] component: deployment.app isFailed: 0 resultCount: 145 runDuration: 0.843 scanCount: 0 searchProviders: 3 sid: 1569294993.84 } { [-] component: deployment.app isFailed: 0 resultCount: 145 runDuration: 1.079 scanCount: 0 searchProviders: 3 sid: 1569294995.85 } { [-] component: deployment.distsearch.peer isFailed: 0 resultCount: 2 runDuration: 0.211 scanCount: 0 searchProviders: 3 sid: 1569294996.86 } { [-] component: deployment.licensing.slave isFailed: 0 resultCount: 1 runDuration: 0.781 scanCount: 0 searchProviders: 3 sid: 1569294997.87 } { [-] component: usage.search.report_acceleration isFailed: 0 resultCount: 1 runDuration: 0.387 scanCount: 0 searchProviders: 3 sid: 1569294998.88 } { [-] component: usage.search.report_acceleration isFailed: 0 resultCount: 1 runDuration: 0.36 scanCount: 0 searchProviders: 3 sid: 1569294998.89 } { [-] component: usage.search.searchTelemetry isFailed: 0 resultCount: 1 runDuration: 1.2650000000000001 scanCount: 14 searchProviders: 3 sid: 1569294999.90 } { [-] component: usage.lookups.lookupDefinitions isFailed: 0 resultCount: 1 runDuration: 0.28700000000000003 scanCount: 0 searchProviders: 1 sid: 1569295000.91 } { [-] component: performance.bundleReplication isFailed: 0 resultCount: 3 runDuration: 1.238 scanCount: 2784 searchProviders: 3 sid: 1569295001.92 } { [-] component: performance.indexing isFailed: 0 resultCount: 8 runDuration: 6.098 scanCount: 35273 searchProviders: 3 sid: 1569295010.93 } { [-] component: performance.search isFailed: 0 resultCount: 3 runDuration: 21.253 scanCount: 213234 searchProviders: 3 sid: 1569295016.94 } { [-] component: usage.search.concurrent isFailed: 0 resultCount: 8 runDuration: 8.671 scanCount: 167724 searchProviders: 3 sid: 1569295038.96 } { [-] component: usage.users.active isFailed: 0 resultCount: 3 runDuration: 9.34 scanCount: 56960 searchProviders: 3 sid: 1569295047.97 } { [-] component: deployment.node isFailed: 0 resultCount: 15 runDuration: 9.965 scanCount: 1166 searchProviders: 3 sid: 1569295056.98 } { [-] component: deployment.index isFailed: 0 resultCount: 113 runDuration: 14.809000000000001 scanCount: 0 searchProviders: 3 sid: 1569295067.99 } { [-] component: usage.search.type isFailed: 0 resultCount: 3 runDuration: 17.365000000000002 scanCount: 167724 searchProviders: 3 sid: 1569295082.100 } { [-] component: licensing.stack isFailed: 0 resultCount: 5 runDuration: 1.772 scanCount: 10 searchProviders: 3 sid: 1569295100.101 } { [-] component: deployment.forwarders isFailed: 0 resultCount: 28 runDuration: 8.309000000000001 scanCount: 268106 searchProviders: 3 sid: 1569295102.102 } { [-] component: usage.indexing.sourcetype isFailed: 0 resultCount: 1373 runDuration: 45.673 scanCount: 735929 searchProviders: 3 sid: 1569295111.103 } { [-] component: deployment.clustering.indexer isFailed: 0 resultCount: 1 runDuration: 3.157 scanCount: 0 searchProviders: 1 sid: 1569295160.104 } { [-] component: usage.app.page isFailed: 0 resultCount: 9 runDuration: 0.795 scanCount: 65 searchProviders: 3 sid: 1569295163.105 } ] roles: { [-] cluster_master: false in_cluster: false indexer: true kv_store: true lead_node: true license_master: true search_head: true } timezone: +0000 }
`licensing.stack`	Licensing quota and consumption.	{ consumption: 127025471 guid: C131C257-98FE-4E8B-9595-CB4D93246F98 host: Splunk name: enterprise pools: [ { consumption: 127025471 quota: 6442450944 } ] product: enterprise quota: 6442450944 subgroup: Production type: enterprise }
`modinputc.telemetry`	Describes how much data is ingested through Splunk apps, add-ons, and connectors.	{ [-] app: component: modinputc.telemetry data: { [-] app: stream333 bytes: 50 version: 3.1 } deploymentID: 18393d55-3552-546c-a5ab-61a96a04ae04 eventID: 367E743C-D629-4B25-B46A-78447116F3A4 executionID: 319FB159-0B47-4CA0-B29D-4CD0EDDF0DCF optInRequired: 1 timestamp: 1586974636 type: event userID: 574f5debd4e54c49ef018a6e1bde0379df499a23a865ab83e8d23d1170256f40 visibility: [ [-] anonymous support ] }
`performance.bundleReplicationCycle`	Metrics for the bundle replication cycle.	{ [-] avgBundleBytes: 0 avgPeerCount: 1 avgPeerSuccessCount: 1 avgReplicationTimeMsec: 1 cycleCount: 144 replicationPolicy: classic }
`performance.indexing`	Indexing performance: Core utilization, storage utilization, memory usage, indexing throughput, search latency.	{ [-] host: 3c4681a5be1881de8554c8bab7be78e8d151557ef571e6a72bdad589 thruput: { [-] avg: 1903 max: 7854 min: 4 p10: 1419 p20: 1433 p30: 1452 p40: 1806 p50: 1860 p60: 1865 p70: 1878 p80: 2046 p90: 2326 total: 7138077 } }
`performance.search`	Search performance: Core utilization, storage utilization, memory usage, indexing throughput, search latency.	{ [-] buckets: { [-] avg: 1.9 max: 27 min: 0 p10: 0 p20: 0 p30: 0 p40: 0 p50: 0 p60: 0.88 p70: 2 p80: 6 p90: 6 } dayRange: { [-] avg: 876.81 max: 18162.29 min: 0 p10: 0 p20: 0 p30: 0 p40: 0 p50: 0 p60: 0.01 p70: 0.01 p80: 0.01 p90: 0.03 } latency: { [-] avg: 2.31 max: 19744.69 min: 0.01 p10: 0.02 p20: 0.02 p30: 0.09 p40: 0.47 p50: 1.6 p60: 1.85 p70: 2.05 p80: 2.23 p90: 2.64 } scanCount: { [-] avg: 344030.32 max: 38060408 min: 0 p10: 0 p20: 0 p30: 0 p40: 0 p50: 1.59 p60: 90.32 p70: 1156.18 p80: 25454.25 p90: 308440.56 } searches: 30576 slices: { [-] avg: 5034.33 max: 219740 min: 0 p10: 0 p20: 0 p30: 0 p40: 0 p50: 0 p60: 0 p70: 2246.06 p80: 11491.43 p90: 14170.42 } }
`preactivation.activate_button.click`	Splunk Assist: Click of the 'Turn on Splunk Assist' button on the preactivation page	{ [-] app: splunk_instrumentation component: otel data: { [-] duration: 100 id: e885d2195fc2da5f name: preactivation.activate_button.click parentId: 384a56ef157fdb04 severity: info source: splunk-assist-telemetry tags: { [-] analyticsSessionId: ac692c428bdd9b311982b2fe4c47ec75-407a7b72cb5c2f6caa9bc0a5e8262f6a2a47c26ed0572f32d77cd5d48280c961 app: splunk-assist-telemetry browser.name: edge-chromium browser.version: 98.0.1108 environment: play isInternalUser: false location.href: http://f746a79e790051f6d1c546e40fd2392cd155ddac1d79a01616751a40f91b1852/en-US/app/splunk_monitoring_console/assist#/onboarding os.name: Windows 10 preferred.color.scheme: light screen.size: {"width":1024,"height":768} splunk.telemetry: skinny-web-opentelemetry splunk.telemetryType: manual splunk.telemetryVersion: 1.15.2 tenant: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 user: 407a7b72cb5c2f6caa9bc0a5e8262f6a2a47c26ed0572f32d77cd5d48280c961 useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36 Edg/98.0.1108.55 } timestamp: 1681813688910000 traceId: 0c656f44e52f4b033452388f258883b0 } deploymentID: c51647a9-5c49-551c-8c13-1b38b69a7b89 eventID: AF1B33F7-C6AF-495B-AD3F-1D2976E333E2 executionID: 27B43822-E2F0-4653-B9D0-016AF5107533 optInRequired: 0 original_timestamp: 1681813703 timestamp: 1681813703 type: event userID: 8c77775e05bdf6989c89d88a47de57bf5df6bf10d0c987ff9337f1a75c33446d visibility: [ [-] anonymous support ] }
`preactivation.support_button.click`	Splunk Assist: Click of the 'Contact Splunk support' button on the preactivation page	{ [-] app: splunk_instrumentation component: otel data: { [-] duration: 100 id: 1cc5688f07374baa name: preactivation.support_button.click parentId: eec194a91c38640f severity: info source: splunk-assist-telemetry tags: { [-] analyticsSessionId: a7960cbdea4fd0e70c587cf2385b166f-407a7b72cb5c2f6caa9bc0a5e8262f6a2a47c26ed0572f32d77cd5d48280c961 app: splunk-assist-telemetry browser.name: chrome browser.version: 111.0.0 environment: prod isInternalUser: false location.href: http://f746a79e790051f6d1c546e40fd2392cd155ddac1d79a01616751a40f91b1852/en-US/app/splunk_monitoring_console/assist#/onboarding os.name: Mac OS preferred.color.scheme: dark screen.size: {"width":3440,"height":1440} splunk.telemetry: skinny-web-opentelemetry splunk.telemetryType: manual splunk.telemetryVersion: 1.15.2 tenant: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 user: 407a7b72cb5c2f6caa9bc0a5e8262f6a2a47c26ed0572f32d77cd5d48280c961 useragent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 } timestamp: 1681844042684000 traceId: 28bc43c6508bf218ee6f5bbb795360af } deploymentID: 09446c1e-b100-526e-923d-c3eabfa52a5d eventID: 392DCE6C-FAA5-491E-A8E3-B485588B6D25 executionID: 71882FFB-EC35-40F3-B63F-BA98F8D16E89 optInRequired: 0 original_timestamp: 1681844042 timestamp: 1681844042 type: event userID: b4d2583bdeef806f1721ca20e001767e2155d5e10ed9d55db68ae3b6db847e2e visibility: [ [-] anonymous support ] }
`onboarding.activate_button.click`	Splunk Assist: Click of the 'Turn on Splunk Assist' button on the onboarding page (landing page of Assist)	{ [-] app: splunk_instrumentation component: otel data: { [-] duration: 200 id: e1f05f657184f226 name: onboarding.activate_button.click parentId: cd1ecae930ad4399 severity: info source: splunk-assist-telemetry tags: { [-] analyticsSessionId: ac692c428bdd9b311982b2fe4c47ec75-407a7b72cb5c2f6caa9bc0a5e8262f6a2a47c26ed0572f32d77cd5d48280c961 app: splunk-assist-telemetry browser.name: edge-chromium browser.version: 98.0.1108 environment: play isInternalUser: false location.href: http://f746a79e790051f6d1c546e40fd2392cd155ddac1d79a01616751a40f91b1852/en-US/app/splunk_monitoring_console/assist#/onboarding os.name: Windows 10 preferred.color.scheme: light screen.size: {"width":1024,"height":768} splunk.telemetry: skinny-web-opentelemetry splunk.telemetryType: manual splunk.telemetryVersion: 1.15.2 tenant: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 user: 407a7b72cb5c2f6caa9bc0a5e8262f6a2a47c26ed0572f32d77cd5d48280c961 useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36 Edg/98.0.1108.55 } timestamp: 1681813688790000 traceId: 0f0cd0643fc0fa8ac0ec107da09ca4a3 } deploymentID: c51647a9-5c49-551c-8c13-1b38b69a7b89 eventID: D1A93DAE-EE3E-4428-832E-2445AAD67DC4 executionID: 27B43822-E2F0-4653-B9D0-016AF5107533 optInRequired: 0 original_timestamp: 1681813702 timestamp: 1681813702 type: event userID: 8c77775e05bdf6989c89d88a47de57bf5df6bf10d0c987ff9337f1a75c33446d visibility: [ [-] anonymous support ] }
`overview.category_card.click`	Splunk Assist: The mouse click event of the category cards at the top of the overview page	{ [-] app: splunk_instrumentation component: otel data: { [-] duration: 900 id: 60c53cbc98c3a193 name: overview.category_card.click severity: info source: splunk-assist-telemetry tags: { [-] analyticsSessionId: 3fc143888a31bfc23a654238a5b4d404-407a7b72cb5c2f6caa9bc0a5e8262f6a2a47c26ed0572f32d77cd5d48280c961 app: splunk-assist-telemetry browser.name: chrome browser.version: 112.0.0 category: availability conforming: 5 critical: 1 environment: play isInternalUser: false location.href: http://a7aa8e2b90a79dd144265ec0e9d9908fd3a12fdc1f0d616d5268408f09bd5ba0/en-US/app/splunk_monitoring_console/assist#/overview os.name: Mac OS preferred.color.scheme: light screen.size: {"width":1792,"height":1120} splunk.telemetry: skinny-web-opentelemetry splunk.telemetryType: manual splunk.telemetryVersion: 1.18.0 tenant: a7aa8e2b90a79dd144265ec0e9d9908fd3a12fdc1f0d616d5268408f09bd5ba0 user: 407a7b72cb5c2f6caa9bc0a5e8262f6a2a47c26ed0572f32d77cd5d48280c961 useragent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 warning: 0 } timestamp: 1683157870140000 traceId: 31a3c8660a709198f3f1597872e23ee2 } deploymentID: 7912c58c-b6bd-5142-9183-bb1aa01723f3 eventID: 00644935-0ABB-458D-8773-1C9425D7A0D1 executionID: EFDE4AC1-1E12-404E-AA7E-1FB81A4C30CC optInRequired: 0 original_timestamp: 1683157871 timestamp: 1683157871 type: event userID: 06bf24a855997aa495e698d0e87ea164ddf8f943a372f7355b97e3e56e6a138c visibility: [ [+] ] }
`overview.topology.node.click`	Splunk Assist: The mouse click event of the topology cards in the Indicators breakdown panel on the overview page	{ [-] app: splunk_instrumentation component: otel data: { [-] duration: 200 id: 20ab9500a297af6c name: overview.topology.node.click severity: info source: splunk-assist-telemetry tags: { [-] analyticsSessionId: afde1287a145bb32b2da0e473dedbe51-407a7b72cb5c2f6caa9bc0a5e8262f6a2a47c26ed0572f32d77cd5d48280c961 app: splunk-assist-telemetry browser.name: chrome browser.version: 113.0.0 count: 11 environment: play isInternalUser: false location.href: http://a7aa8e2b90a79dd144265ec0e9d9908fd3a12fdc1f0d616d5268408f09bd5ba0/en-US/app/splunk_monitoring_console/assist#/overview nodeType: search_head os.name: Mac OS preferred.color.scheme: light screen.size: {"width":1792,"height":1120} splunk.telemetry: skinny-web-opentelemetry splunk.telemetryType: manual splunk.telemetryVersion: 1.18.0 status: warning tenant: a7aa8e2b90a79dd144265ec0e9d9908fd3a12fdc1f0d616d5268408f09bd5ba0 user: 407a7b72cb5c2f6caa9bc0a5e8262f6a2a47c26ed0572f32d77cd5d48280c961 useragent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 } timestamp: 1683225546025000 traceId: a1865c6b31a78db19a61050915c5c267 } deploymentID: 7912c58c-b6bd-5142-9183-bb1aa01723f3 eventID: 6CB91A1F-E898-425C-B4B9-76292542F632 executionID: 8DE5F611-B2A5-47A8-BBD4-BBA50AB6A4F5 optInRequired: 0 original_timestamp: 1683225547 timestamp: 1683225547 type: event userID: 06bf24a855997aa495e698d0e87ea164ddf8f943a372f7355b97e3e56e6a138c visibility: [ [+] ] }
`overview.overview_list.open_assist.click`	Splunk Assist: The mouse click event of the action button on Overview table. Clicking on this button will open up the assist page for the indicator indicated by indicatorName	{ [-] app: splunk_instrumentation component: otel data: { [-] duration: 200 id: 22e35157c73c06ea name: overview.overview_list.open_assist.click severity: info source: splunk-assist-telemetry tags: { [-] analyticsSessionId: c4f2271527ab0a4497abc1d66d29fcd0-407a7b72cb5c2f6caa9bc0a5e8262f6a2a47c26ed0572f32d77cd5d48280c961 app: splunk-assist-telemetry browser.name: chrome browser.version: 112.0.0 conforming: 3 critical: 0 environment: play indicatorName: assist-certificate.expiry isInternalUser: false location.href: http://559f8b32939d9748b4e512a6c69050be140e8c3b8ce74a4575d1890e2a030c64/en-US/app/splunk_monitoring_console/assist#/overview os.name: Mac OS preferred.color.scheme: light screen.size: {"width":1792,"height":1120} splunk.telemetry: skinny-web-opentelemetry splunk.telemetryType: manual splunk.telemetryVersion: 1.18.0 tenant: 559f8b32939d9748b4e512a6c69050be140e8c3b8ce74a4575d1890e2a030c64 user: 407a7b72cb5c2f6caa9bc0a5e8262f6a2a47c26ed0572f32d77cd5d48280c961 useragent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 warning: 0 } timestamp: 1683227439182000 traceId: c4974a407db142f4dcb30f4ab1f6767e } deploymentID: a3d7ae8d-f11a-56f7-a4d1-fd455b087616 eventID: 6A3486A8-D5BF-4669-AF08-0F48D83BD683 executionID: 5712114F-1486-45E6-A907-5C6CA02F8EEF optInRequired: 0 original_timestamp: 1683227446 timestamp: 1683227446 type: event userID: 7503b69d9c6e85f659d04ab0582e4fab2aee90b91d74f67e9dcaf2f9a9de919b visibility: [ [+] ] }
`usage.admissionRules.report`	Admission rules: Status, list of rules enabled and rules triggered for filtered searches.	{ [-] app: splunk_instrumentation component: usage.admissionRules.report data: { [-] admissionRulesEnabled: 1 guid: 13E5506A-4C0F-4BB9-B468-B5F977A00FDE host: e521fc4eebd5e93b2cadcced3e03f699c86f2b5c rules: { [-] allindex_alltime: { [-] predicate: index=df58248c414f342c81e056b40bee12d17a08bf61 AND search_time_range=alltime } audit: { [-] predicate: index=cb4ed408dd9f3497da0bcbece65f847423927e85 AND app=3559d7accf00360971961ca18989adc0614089c0 AND role=d033e22ae348aeb5660fc2140aec35850c4da997 } internal: { [-] predicate: index=f1b1f1f40216ee2e2b5a526eec43c8f71cccef5d AND user=d033e22ae348aeb5660fc2140aec35850c4da997 AND search_time_range=alltime } totalCount: 3 } rulesTriggered: [ [-] { [-] filteredSearchesCount: 1 searchFilterRule: allindex_alltime } { [-] filteredSearchesCount: 3 searchFilterRule: audit } { [-] filteredSearchesCount: 1 searchFilterRule: internal } ] serverRoles: indexer, license_master } deploymentID: dc739253-34a9-5b44-afd8-ea73e9066dc5 eventID: DE0063AE-31F5-42FA-AE92-0F62913EF42E executionID: 8B45C62A-0D0B-4689-B1BD-F29BFA3D9255 optInRequired: 3 timestamp: 1587004320 type: aggregate visibility: [ [-] anonymous support ] }
`usage.app.page`	App name, page name, locale, number of users, number of page loads, generated as session data.	{ [-] app: search locale: en-US occurrences: 1 page: users: 1 }
`usage.authMethod.config`	Authentication method: Hashed host and GUID, authentication method (Splunk, LDAP, or SAML), MFA type (none, Duo, or RSA).	{ [-] authentication method: Splunk guid: C099BFA3-E5B5-4AB1-AB64-471703C54388 host: 8cd44b23a1bd3ae283f21a7d9c5434163181efc8 mfa type: none }
`usage.bucketmerge.clustered`	Usage of cluster bucket merge command, cluster bucket list command, and cluster bucket merge command with -dryrun option.	{ [-] component: usage.bucketmerge.clustered data: { command: merge newBucketsCount: 5 oldBucketsCount: 50 bucketsFailedToMergeCount: 2 indexersCount: 10 } date: 2018-10-26 deploymentID: 99b6ffd8-2e80-5e3b-905c-8c6f6fd743a0 executionID: F0AE995E8653D768A360E73BE3F544 timestamp: 1540570045 transactionID: 89F7329E-86AD-BBFD-034F-209CB8A06F05 version: 3 visibility: anonymous,support }
`usage.bucketmerge.standalone`	Usage of bucket merge command, bucket list command, and bucket merge command with --dryrun option.	{ [-] component: usage.bucketmerge.standalone data: { command: merge newBucketsCount: 5 oldBucketsCount: 50 durationSec: 7.5 } date: 2018-10-26 deploymentID: 99b6ffd8-2e80-5e3b-905c-8c6f6fd743a0 executionID: F0AE995E8653D768A360E73BE3F544 timestamp: 1540570045 transactionID: 89F7329E-86AD-BBFD-034F-209CB8A06F05 version: 3 visibility: anonymous,support }
`usage.configtracker.config`	Whether or not the feature is enabled or disabled. What "mode" the feature is in (e.g. - diff, track_only, auto.) And what kinds of file paths, and/or fields are added to the denylist.	{ [-] component: usage.configtracker.config data: { disabled: false mode: auto denylist: someregexfilterhere uses_inotify: true exclude_fields: server.conf:general:pass4SymmKey, ui-prefs.conf:general:* } date: 2018-10-26 deploymentID: 99b6ffd8-2e80-5e3b-905c-8c6f6fd743a0 executionID: F0AE995E8653D768A360E73BE3F544 timestamp: 1540570045 transactionID: 89F7329E-86AD-BBFD-034F-209CB8A06F05 version: 3 visibility: anonymous,support }
`usage.configtracker.introspection`	Configuration file change logs made on a Splunk instance.	{ [-] component: usage.configtracker.introspection data: { count: 102 path: $SPLUNK_HOME/etc/system/local/transforms.conf stanza: hostoverride prop: DEST_KEY, REGEX, FORMAT } date: 2018-10-26 deploymentID: 99b6ffd8-2e80-5e3b-905c-8c6f6fd743a0 executionID: F0AE995E8653D768A360E73BE3F544 timestamp: 1540570045 transactionID: 89F7329E-86AD-BBFD-034F-209CB8A06F05 version: 3 visibility: anonymous,support }
`usage.configtracker.searches`	Configuration file change SPL queries that were run on an environment, and their corresponding results.	{ [-] component: usage.configtracker.searches data: { user_count: 20 total_search_count: 754 } date: 2018-10-26 deploymentID: 99b6ffd8-2e80-5e3b-905c-8c6f6fd743a0 executionID: F0AE995E8653D768A360E73BE3F544 timestamp: 1540570045 transactionID: 89F7329E-86AD-BBFD-034F-209CB8A06F05 version: 3 visibility: anonymous,support }
`usage.durableSearch`	Number of users of the durable search feature, how durable search is being used (for scheduled searches? for summary indexing?), and commonly-used durable search setting values.	{ [-] durableBackfillType: auto durableLagTime: 60 durableMaxBackfillIntervals: 100 durableTrackTimeType: _indextime enableSummaryIndex: Yes name: 8a4d0e8816a25ed813c5f40dbfc34d0bd46d9c49 } date: 2020-06-02 deploymentID: 87402ea1-6505-59d5-b04a-c12dcf7b0a06 executionID: ED6EF443C5FC863A9AABA6B89A1839 timestamp: 1591117572 transactionID: 0B2234FD-2D78-7939-75B1-B5BECABD5FD3 version: 4 visibility: anonymous,support
`usage.healthMonitor.currentState`	Distributed health report: Enabled status, number of clicks, node status (node path, current color, worst color in last 24 hours), Splunk version.	{ [-] enabled: 1 } healthReportClicks: 10 nodeStatus: [ [-] { [-] color: green nodePath: splunkd worstColorInLast24Hours: green } { [-] color: green nodePath: splunkd.file_monitor_input worstColorInLast24Hours: green } { [-] color: green nodePath: splunkd.file_monitor_input.batchreader-0 worstColorInLast24Hours: green } { [-] color: green nodePath: splunkd.file_monitor_input.tailreader-0 worstColorInLast24Hours: green } { [-] color: green nodePath: splunkd.index_processor worstColorInLast24Hours: green } { [+] } { [+] } { [+] } { [+] } { [+] } { [+] } { [+] } ] splunkVersion: 8.1.0 }
`usage.healthMonitor.report`	Health report manager: Alert actions and enabled status, feature thresholds and enabled status.	{ [-] alert: { [-] alert_action:email: { [-] action/ action.to/ action.url/ action.integration_url_override: empty disabled: 0 } alert_action:webhook: { [-] action/ action.to/ action.url/ action.integration_url_override: empty disabled: 0 } health_reporter: { [-] action/ action.to/ action.url/ action.integration_url_override: email disabled: 0 } } feature:batchreader: { [-] enabled: 1 threshold: { [-] indicator:data_out_rate:red: 2 indicator:data_out_rate:yellow: 1 } } feature:buckets: { [-] enabled: 1 threshold: { [-] indicator:buckets_created_last_60m:red: 60 indicator:buckets_created_last_60m:yellow: 40 indicator:percent_small_buckets_created_last_24h:red: 50 indicator:percent_small_buckets_created_last_24h:yellow: 30 } } feature:cluster_bundles: { [-] enabled: 1 threshold: { [-] indicator:cluster_bundles:yellow: 1 } } feature:data_durability: { [-] enabled: 1 threshold: { [-] indicator:cluster_replication_factor:red: 1 indicator:cluster_search_factor:red: 1 } } feature:data_searchable: { [-] enabled: 1 threshold: { [-] indicator:data_searchable:red: 1 } } feature:ddaa_archived_buckets: { [-] enabled: 1 threshold: { [-] indicator:archived_buckets_failed_last_24h:red: 80 indicator:archived_buckets_failed_last_24h:yellow: 40 } } feature:disk_space: { [-] enabled: 1 threshold: { [-] indicator:disk_space_remaining_multiple_minfreespace:red: 1 indicator:disk_space_remaining_multiple_minfreespace:yellow: 2 } } feature:indexers: { [-] enabled: 1 threshold: { [-] indicator:detention:red: 1 indicator:detention:yellow: 1 indicator:missing_peers:red: 1 indicator:missing_peers:yellow: 1 } } feature:indexing_ready: { [-] enabled: 1 threshold: { [-] indicator:indexing_ready:red: 1 } } feature:master_connectivity: { [-] enabled: 1 threshold: { [-] indicator:master_connectivity:red: 1 } } feature:replication_failures: { [-] enabled: 1 threshold: { [-] indicator:replication_failures:red: 10 indicator:replication_failures:yellow: 5 } } feature:s2s_autolb: { [-] enabled: 1 threshold: { [-] indicator:s2s_connections:red: 70 indicator:s2s_connections:yellow: 20 } } feature:search_lag: { [-] enabled: 1 threshold: { [-] indicator:count_extremely_lagged_searches_last_hour:red: 1 indicator:count_extremely_lagged_searches_last_hour:yellow: 0 indicator:percent_searches_lagged_high_priority_last_24h:yellow: 10 indicator:percent_searches_lagged_non_high_priority_last_24h:yellow: 40 } } feature:searches_delayed: { [-] enabled: 1 threshold: { [-] indicator:percent_searches_delayed_high_priority_last_24h:red: 10 indicator:percent_searches_delayed_high_priority_last_24h:yellow: 5 indicator:percent_searches_delayed_non_high_priority_last_24h:red: 20 indicator:percent_searches_delayed_non_high_priority_last_24h:yellow: 10 } } feature:searches_skipped: { [-] enabled: 1 threshold: { [-] indicator:percent_searches_skipped_high_priority_last_24h:red: 10 indicator:percent_searches_skipped_high_priority_last_24h:yellow: 5 indicator:percent_searches_skipped_non_high_priority_last_24h:red: 20 indicator:percent_searches_skipped_non_high_priority_last_24h:yellow: 10 } } feature:searchheadconnectivity: { [-] enabled: 1 threshold: { [-] indicator:master_connectivity:red: 1 indicator:master_version_compatibility:yellow: 1 } } feature:shc_captain_common_baseline: { [-] enabled: 1 threshold: { [-] indicator:common_baseline:red: 1 } } feature:shc_captain_connection: { [-] enabled: 1 threshold: { [-] indicator:captain_connection:red: 1 indicator:captain_existence:red: 1 } } feature:shc_captain_election_overview: { [-] enabled: 1 threshold: { [-] indicator:dynamic_captain_quorum:yellow: 1 } } feature:shc_members_overview: { [-] enabled: 1 threshold: { [-] indicator:detention:red: 1 indicator:detention:yellow: 1 indicator:replication_factor:yellow: 1 indicator:status:red: 1 indicator:status:yellow: 1 } } feature:shc_snapshot_creation: { [-] enabled: 1 threshold: { [-] indicator:snapshot_creation:red: 20 indicator:snapshot_creation:yellow: 10 } } feature:slave_state: { [-] enabled: 1 threshold: { [-] indicator:slave_state:red: 1 indicator:slave_state:yellow: 1 } } feature:slave_version: { [-] enabled: 1 threshold: { [-] indicator:slave_version:red: 1 } } feature:splunkoptimize_processes: { [-] enabled: 1 threshold: { [-] indicator:concurrent_optimize_processes_percent:yellow: 100 } } feature:tailreader: { [-] enabled: 1 threshold: { [-] indicator:data_out_rate:red: 2 indicator:data_out_rate:yellow: 1 } } feature:wlm_configuration_check: { [-] enabled: 1 threshold: { [-] indicator:configuration_check:red: 0 } } feature:wlm_system_check: { [-] enabled: 1 threshold: { [-] indicator:system_check:red: 0 } } }
`usage.indexing.sourcetype`	Indexing volume, number of events, number of hosts, source type name.	{ [-] bytes: 90962 events: 354 hosts: 1 name: splunk_telemetry }
`usage.ingestactions.deletions`	Count of destination and ruleset deletions	{ [-] data: { destinationDeletions: 3 rulesetDeletions: 1 } date: 2018-10-26 deploymentID: 99b6ffd8-2e80-5e3b-905c-8c6f6fd743a0 executionID: F0AE995E8653D768A360E73BE3F544 timestamp: 1540570045 transactionID: 89F7329E-86AD-BBFD-034F-209CB8A06F05 version: 3 visibility: anonymous,support }
`usage.ingestactions.destinations`	Characteristics of routing destinations	{ [-] data: { destinations: { [ { batchSizeThresholdKB: 131072 batchTimeout: 5 compression : none dropEventsOnUploadError: false encryption: none signatureVersion: v1 supportsVersioning: true urlVersion: v1 destinationType: s3 authMethodAccesskey: true authMethodIAM": false } }] } date: 2018-10-26 deploymentID: 99b6ffd8-2e80-5e3b-905c-8c6f6fd743a0 executionID: F0AE995E8653D768A360E73BE3F544 timestamp: 1540570045 transactionID: 89F7329E-86AD-BBFD-034F-209CB8A06F05 version: 3 visibility: anonymous,support }
`usage.ingestactions.rulesets`	Count of routing destinations, ruletset types, and ruleset conditions	{ [-] data: { s3: 2 filter: 3 mask: 3 route: 5 set_index: 2 clone: 2 maskRegexCount: 3 filterRegexCount: 2 filterEvalExprCount: 1 routeRegexCount: 3 routeEvalExprCount: 2 uniqueIndexCount: 2 } date: 2018-10-26 deploymentID: 99b6ffd8-2e80-5e3b-905c-8c6f6fd743a0 executionID: F0AE995E8653D768A360E73BE3F544 timestamp: 1540570045 transactionID: 89F7329E-86AD-BBFD-034F-209CB8A06F05 version: 3 visibility: anonymous,support }
`usage.kvstore`	Metrics and performance data about KV store.	{ [-] usage.flushAverageMs: 5.3538461538461535 usage.instanceType: primary usage.memRamMb: 0 usage.memVirtualMb: 0 usage.oplogEndTime: 1569301264 usage.oplogStartTime: 1569222045 usage.oplogTimeRange: 79219 usage.readLatencyToUpTime: 0.000153653421585191 usage.readLatencyUsPerOp: 0.02158053280617528 usage.storageEngine: mmapv1 usage.upTime: 3956 usage.version: 3.6.12-splunk usage.writeLatencyToUpTime: 0.000153653421585191 usage.writeLatencyUsPerOp: 0.00048009036995199094 }
`usage.lookups.lookupDefinitions`	Lookup definition metadata with hashed lookup names.	{ [-] lookups: [ [-] { [-] _timediff: is_temporal: 0 name: 96117ed21e74f16d452027ed8e16c5d32fddd229 sharing: system size: type: external } { [-] _timediff: is_temporal: 0 name: 256d0fae9448acc55cd2e5cbabe7dbec576158c2 sharing: global size: 18053 type: file } { [-] _timediff: is_temporal: 0 name: 88767984d9dc6308309ffde5dc3591fa3865e7f2 sharing: global size: 832 type: file } { [-] _timediff: is_temporal: 0 name: 1b0131dbc851786586e269a2ba8b2f08bbd6834f sharing: global size: type: geo } { [-] _timediff: is_temporal: 0 name: 6d47b91d0c0753e9332ec2c0f8c956151c9b1e16 sharing: global size: type: geo } ] }
`usage.passwordPolicy.config`	Password policy management: hashed host and GUID, attribute configurations.	{ [-] constant login time: 0.000 days until password expires: 90 enable lockout users: false enable password expiration: false enable password history: false enable verbose login fail message: true expiration alert in days: 15 failed login attempts: 5 force existing users to change weak passwords: false guid: 32BEE8DE-E64D-4B02-B2FE-4F13F18A0CAE host: b8758da2f94fd58e648bce573fa3d9dc5797566d lockout duration in minutes: 30 lockout threshold in minutes: 5 minimum number of characters: 1 minimum number of digits: 0 minimum number of lowercase letters: 0 minimum number of special characters: 0 minimum number of uppercase letters: 0 password history count: 24 }
`usage.python`	Default setting for Python version in the app, path of the script with its name hashed, version of Python used in the script.	{ [-] pythonDefault: python2 scriptPath: /usr/local/bamboo/splunk-install/current/etc/apps/SplunkEnterpriseSecuritySuite/bin/D7A80DE23601F645B8A06995DF910A3D08AB9EAA scriptPythonVersion: python2 }
`usage.rest`	Usage of an endpoint, HTTP method, status code, and user agent in a REST request made from a Splunk Enterprise SDK. The data that is collected includes the partial endpoint URL of the target feature. Any user-identifiable data or resource names in the URL are discarded.	{ [-] endpointUri: search/jobs method: POST status: 200 userAgent: splunk-sdk-python/1.6.3 }
`usage.savedSearches.alert`	Usage of the saved search alerting functionality: triggering conditions and modes, alert actions, alert suppression, schedules, and so on.	{ [-] actionList: script alertConditionType: number of hosts alertSeverity: 3 alertSuppress: No alertSuppressGroup: 58e7079db82d48abfcdda002ce09d3f371c8bad1 alertTrackable: Yes cronSchedule: 0 0 * * * name: 831ee1f249cf286c2065e7ba7e38b0b5228c738d triggerMode: Once }
`usage.search.concurrent`	Distribution of concurrent searches.	{ [-] host: 3c4681a5be1881de8554c8bab7be78e8d151557ef571e6a72bdad589 searches: { [-] avg: 2 max: 2 min: 1 p10: 1 p20: 1 p30: 1 p40: 1 p50: 2 p60: 2 p70: 2 p80: 2 p90: 2 } }
`usage.search.report_acceleration`	Report acceleration metrics.	{ [-] existing_report_accelerations: 0 }
`usage.search.searchTelemetry`	List of commands and corresponding counts for all searches run on the system in the span of one day.	{ [-] commands: [ [-] { [-] count: 1 name: addinfo } { [-] count: 5 name: eval } { [-] count: 6 name: external_command } { [-] count: 9 name: fields } { [-] count: 1 name: inputlookup } { [-] count: 1 name: join } { [-] count: 1 name: litsearch } { [-] count: 2 name: makemv } { [-] count: 1 name: mvcombine } { [-] count: 2 name: mvexpand } { [-] count: 2 name: noop } { [-] count: 4 name: prerest } { [-] count: 1 name: prestats } { [-] count: 4 name: presummarize } { [-] count: 2 name: rename } { [-] count: 4 name: rest } { [-] count: 1 name: search } { [-] count: 3 name: stats } { [-] count: 4 name: summarize } { [-] count: 6 name: timeliner } { [-] count: 1 name: where } ] }
`usage.search.searchtelemetry.type`	Search type, count, average bytes read, max bytes read, duration.	{ [-] searchTypeInformation: [ [-] { [-] avg(bytes_read): 90531.02683363149 count: 559 duration: 1488.45949719 max(bytes_read): 46382154 type: adhoc } { [-] avg(bytes_read): 0 count: 3224 duration: 199.042348043 max(bytes_read): 0 type: scheduled } ] }
`usage.search.searchtelemetry.sourcetypeUsage`	Sourcetype usage.	{ [-] sourcetypeUsage: [ [-] { [-] http_event_collector_metrics: 1 kvstore: 1 mongod: 3 search_telemetry: 1 splunk_disk_objects: 1 splunk_resource_usage: 1 splunk_web_service: 3 splunkd: 11 splunkd_remote_searches: 3 splunkd_ui_access: 2 } ] }
`usage.search.type`	Number of searches of each type.	{ [-] ad-hoc: 3619 datamodel acceleration: 1 other: 2 report acceleration: 1 scheduled: 34412 summary index: 506 }
`usage.smartStore.Config`	SmartStore global configuration, per index configuration, hashed internal and external index names.	{ [-] global config: { [-] cachemanager: { [-] eviction_padding: 5120 hotlist_bloom_filter_recency_hours: 360 hotlist_recency_secs: 86400 max_cache_size: 0 } clustering: { [-] mode: disabled } diskUsage: { [-] minFreeSpace: 5000 } } list of indexes: { [-] non-SmartStore enabled: ea9f4255e269599dd961c3efd8775ab5ac1d3948,f1b1f1f40216ee2e2b5a526eec43c8f71cccef5d,302a11446cd560395417c9e2d2177a7a0fa8d74d,568b2f85dcc1c8608d713a66a0eabd5b88956547,d140ef99de26b2f8b6f54081084d0b8b2f59f36f,5a74588fcf73bdd06619007f6659c41827885700,66f79d8a6327c82c9033e6d65ff03322a3766c87,b28b7af69320201d1cf206ebf28373980add1451,f4f77578164d1b03fb4c931f727a3e2966e541d4,0d176ba3aa7be325bcaeaf13ea2da4d155f04e33,87da723b9f33eb0f1bcad8ea3405d8c2d248f862,05535ecff78ef61038725b6ed3016b8c9a037496,f397214775e4f8191c17e838b4d518cb90051672 } per index config: { [-] external_05535ecff78ef61038725b6ed3016b8c9a037496: { [-] frozenTimePeriodInSecs: 188697600 hotlist_bloom_filter_recency_hours: none hotlist_recency_secs: none maxGlobalDataSizeMB: 0 maxHotSpanSecs: 7776000 } external_0d176ba3aa7be325bcaeaf13ea2da4d155f04e33: { [-] frozenTimePeriodInSecs: 188697600 hotlist_bloom_filter_recency_hours: none hotlist_recency_secs: none maxGlobalDataSizeMB: 0 maxHotSpanSecs: 7776000 } external_66f79d8a6327c82c9033e6d65ff03322a3766c87: { [-] frozenTimePeriodInSecs: 604800 hotlist_bloom_filter_recency_hours: none hotlist_recency_secs: none maxGlobalDataSizeMB: 0 maxHotSpanSecs: 7776000 } external_87da723b9f33eb0f1bcad8ea3405d8c2d248f862: { [-] frozenTimePeriodInSecs: 188697600 hotlist_bloom_filter_recency_hours: none hotlist_recency_secs: none maxGlobalDataSizeMB: 0 maxHotSpanSecs: 7776000 } external_b28b7af69320201d1cf206ebf28373980add1451: { [-] frozenTimePeriodInSecs: 188697600 hotlist_bloom_filter_recency_hours: none hotlist_recency_secs: none maxGlobalDataSizeMB: 0 maxHotSpanSecs: 7776000 } external_f397214775e4f8191c17e838b4d518cb90051672: { [-] frozenTimePeriodInSecs: 188697600 hotlist_bloom_filter_recency_hours: none hotlist_recency_secs: none maxGlobalDataSizeMB: 0 maxHotSpanSecs: 7776000 } external_f4f77578164d1b03fb4c931f727a3e2966e541d4: { [-] frozenTimePeriodInSecs: 188697600 hotlist_bloom_filter_recency_hours: none hotlist_recency_secs: none maxGlobalDataSizeMB: 0 maxHotSpanSecs: 7776000 } internal_302a11446cd560395417c9e2d2177a7a0fa8d74d: { [-] frozenTimePeriodInSecs: 1209600 hotlist_bloom_filter_recency_hours: none hotlist_recency_secs: none maxGlobalDataSizeMB: 0 maxHotSpanSecs: 7776000 } internal_568b2f85dcc1c8608d713a66a0eabd5b88956547: { [-] frozenTimePeriodInSecs: 1209600 hotlist_bloom_filter_recency_hours: none hotlist_recency_secs: none maxGlobalDataSizeMB: 0 maxHotSpanSecs: 7776000 } internal_5a74588fcf73bdd06619007f6659c41827885700: { [-] frozenTimePeriodInSecs: 2419200 hotlist_bloom_filter_recency_hours: none hotlist_recency_secs: none maxGlobalDataSizeMB: 0 maxHotSpanSecs: 7776000 } internal_d140ef99de26b2f8b6f54081084d0b8b2f59f36f: { [-] frozenTimePeriodInSecs: 63072000 hotlist_bloom_filter_recency_hours: none hotlist_recency_secs: none maxGlobalDataSizeMB: 0 maxHotSpanSecs: 7776000 } internal_ea9f4255e269599dd961c3efd8775ab5ac1d3948: { [-] frozenTimePeriodInSecs: 188697600 hotlist_bloom_filter_recency_hours: none hotlist_recency_secs: none maxGlobalDataSizeMB: 0 maxHotSpanSecs: 7776000 } internal_f1b1f1f40216ee2e2b5a526eec43c8f71cccef5d: { [-] frozenTimePeriodInSecs: 2592000 hotlist_bloom_filter_recency_hours: none hotlist_recency_secs: none maxGlobalDataSizeMB: 0 maxHotSpanSecs: 432000 } } total storage capacity: { [-] 0: { [-] available: 130459.672 capacity: 476802.039 free: 142405.105 fs_type: apfs } } }
`usage.streamingMetricAlerts`	Usage of the streaming metric alerting functionality: group by alerts, triggering evaluations and thresholds, alert suppression, result enrichment or filtering, and alert actions.	{ [-] actionList: email,rss alertSeverity: 2 alertTrackable: No hasComplexCondition: Yes hasDescription: Yes hasFilter: No hasGroupby: Yes hasLabels: Yes hasMultipleMetricIndexes: Yes name: 227a3ad2631f5a7fe8709f7cac3308580f532d75 triggerActionPerGroup: Yes triggerEvaluationPerGroup: Yes triggerExpires: 48h triggerMaxTracked: 10 triggerPrepare: No triggerSuppress: No triggerThreshold: once after 5m }
`usage.users.active`	The number of active users per day.	{ [-] active: 1 }
`usage.workloadManagement.report`	Workload management: Hashed host and GUID, OS/version, server roles, WLM support and enable status, pool configurations, rule configurations.	{ [-] categories: { [-] ingest: { [-] allocated cpu percent: 20.00 allocated mem limit: 100.00 } misc: { [-] allocated cpu percent: 10.00 allocated mem limit: 10.00 } search: { [-] allocated cpu percent: 70.00 allocated mem limit: 70.00 } } guid: F3DC7C6B-DF89-4585-A7A6-B4A3510D957D host: eadc124359ea492c6b04c079dcf3bec3be2fb32c os: Linux osVersion: 4.9.184-linuxkit pools: { [-] total count: 0 } rules: { [-] total count: 0 } server roles: indexer, license_master, kv_store wlm enabled: 0 wlm supported: 1 }

Support usage data examples

Support usage data is the same data as the aggregated usage data, but if you opt to send support usage data, Splunk can use the license GUID to identify usage data from a specific customer account to help troubleshoot support cases.

See Aggregated usage data examples.

Support usage data is distinct from diagnostic file data. Diagnostic files are never automatically generated and can only be sent to Splunk Support manually by a user with the appropriate permissions. For more about diagnostic files, see Generate a diag in the Troubleshooting Manual.

License usage data examples

The following example demonstrates the type of data sent to Splunk when sharing of license usage data is enabled.

Component Description Example

licensing.stack

Licensing quota and consumption

{ [-]
     consumption: 14462827
     guid: 47798245-85D7-4DCA-A303-D49910F40ED1
     host: fecaab81b0934386719a161bfe3656ca782ec6d14806ae15d4ec4dc5
     name: enterprise
     pools: [ [-]
       { [-]
         consumption: 14462827
         quota: 53687091200
       }
     ]
     product: enterprise
     quota: 53687091200
     subgroup: Production
     type: enterprise
   }

Software version data examples

The following example demonstrates the software version data sent to Splunk for Splunk Enterprise when sharing of software version data is enabled.

Description	Example
CPU architecture	x86_64
Operating system	Linux
Product	enterprise
Splunk roles	admin
License group, subgroup, and hashed GUID	Enterprise, Production, <GUID>
Splunk software version	7.0.0

The following example demonstrates the software version data sent to Splunk for each app when sharing of software version data is enabled for that app.

Description	Example
App ID, name, and version	gettingstarted, Getting Started, 1.0
Splunk version	7.0
Platform, architecture	Darwin, x86_64

App usage data examples

In addition to the data enumerated in this topic, certain apps collect usage data. See the documentation for each app for details and examples.

Splunk Add-on Builder: Share data in Splunk Add-on Builder
Splunk App for AWS: Share data in the Splunk App for AWS
Splunk Business Flow: Share data in Splunk Business Flow
Splunk DB Connect: Share data in Splunk DB Connect
Splunk Enterprise Security: Share data in Splunk Enterprise Security
Splunk Industrial Asset Intelligence: Share data in Splunk Industrial Asset Intelligence
Splunk IT Service Intelligence: Share data in Splunk IT Service Intelligence
Splunk Machine Learning Toolkit: Share data in the Splunk Machine Learning Toolkit
Splunk Security Essentials: Splunk Security Essentials Telemetry

How Splunk collects the data

If aggregated, support, or license usage data collection is enabled, a few instances in your Splunk Enterprise deployment collect data through scheduled searches. Most of the searches run in sequence, starting at 3:05 AM on the node that runs the searches, unless you change the schedule. All searches are triggered with a scripted input.

In addition, when aggregated or support data collection is enabled, session data about user activity transmits from the browser directly to the Splunk telemetry API.

Which instance runs the searches and sends data to Splunk

One primary instance in your deployment runs distributed searches that collect most of the usage data. This primary instance is also responsible for sending the data to Splunk. The instance that acts as the primary instance depends on the details of your deployment:

If indexer clustering is enabled, the cluster manager is the primary instance. If you have more than one indexer cluster, each cluster manager is a primary instance.
If search head clustering is enabled but not indexer clustering, each search head captain is a primary instance.
If your deployment does not use clustering, the searches run on a search head.

If you opt out of instrumentation, the searches from the primary instance do not run.

Additional instances in your deployment run a smaller number of searches, depending on colocation details. If data collection is enabled, the data from these searches is collected by the primary node and sent to Splunk. If you opt out, these searches still run, but no data is sent.

For your deployment to send data to Splunk, the primary instance responsible for the searches must be connected to the internet with no firewall rules or proxy server configurations that prevent outbound traffic to https://quickdraw.splunk.com/telemetry/destination or https://*.api.splkmobile.com. If necessary, add these URLs for outbound traffic to your firewall allow list.

Instrumentation in the Splunk Enterprise file system

After the searches run, the primary instance packages the searched data and sends it to Splunk. It also indexes the data to the _telemetry index. Session data is transmitted directly to the telemetry API from the browser. It does not go to the _telemetry index. The _telemetry index retains the data for two years by default and is limited in size to 256 MB.

The instrumentation app resides in the file system at $SPLUNK_HOME/etc/apps/splunk_instrumentation.

How Splunk uses the data it collects

If you share aggregated usage data, Splunk collects data about your Splunk software usage and aggregates it together with similar data from other deployments so Splunk can understand what features and workflows are most important to users and improve its products and services over time. Collected license IDs are used only to verify that data is received from a valid Splunk product and persisted only for deployments opting into license or support usage reporting. These license IDs help Splunk analyze how different Splunk products are being deployed across the population of customers and are not attached to any aggregated usage data.

If you share support usage data, Splunk links the data about your software usage to your installed license ID so that Splunk can provide improved support and services for your deployment. The Splunk Assist service uses support usage data to identify and provide insights to let you align your Splunk Enterprise deployment with Splunk best practices for security, performance, and configuration. The Support and Customer Success teams use this data to identify and troubleshoot support issues that you file and improve your Splunk software implementation.

If you share license usage data, Splunk uses the data to ensure compliance with your purchased offering.

If you share Splunk product version data, Splunk uses the data to track how many deployments use particular versions of Splunk software offerings and to provide in-product notifications when updates are available. For apps, version data is correlated with information about app downloads to populate app analytics views on Splunkbase provided to the app's developer, and to compute the number of installs on the app details page.

How Splunk transmits and stores the data it collects

When you enable aggregated, support, and license usage data sharing, Splunk Enterprise runs searches to collect this data and sends the search summaries to a collection endpoint. Session data and Splunk software version data is not included in the searches. Session data is sent from your browser as the events are generated. Version data about Splunk Enterprise is sent to Splunk by your browser after you log into Splunk Web. Version data about your Splunk apps is sent to Splunk daily through a REST call from splunkd to splunkbase.splunk.com. Data is transmitted to Splunk from a single primary instance in your deployment. See Which instance runs the searches and sends data to Splunk.

The Splunk platform encrypts telemetry data with transport layer security (TLS) before it leaves your deployment, and verifies authentication before it stores the data securely on Splunk cloud infrastructure. The infrastructure that customer telemetry uses has strict access controls that are subject to regular audit. For more information about how Splunk collects, uses, and discloses information about the data collected, see the Splunk Privacy Policy. For more information about Splunk's data privacy, security, and compliance practices, see Splunk Protects.

View the data your Splunk Enterprise deployment sends to Splunk

You can view aggregated usage, support usage, and license usage data that your deployment has recently sent in Splunk Web.

Navigate to Settings > Instrumentation.
Click the category of data you wish to view in Search.

This log is available only after the first run of the collection. To inspect the type of data that gets sent before you opt in on your production environment, you can opt in on your sandbox environment.

To view the browser session data, use JavaScript logging in your browser. Look for network events sent to a URL containing splkmobile. Events are triggered by user actions such as navigating to a new page in Splunk Web.

To view version data that is sent for Splunk Enterprise, watch JavaScript network traffic as you log into Splunk Web. The data is sent inside a call to quickdraw.splunk.com.

How to opt out

Splunk collects support usage, aggregated usage, license data, and software version data by default. You can opt in or out at any time.

Prerequisite
To enable or disable collection of usage data, the user that you use to log into Splunk Enterprise must hold a role that includes the edit_telemetry_settings capability.

Opt out of sharing aggregated or support usage data

To change your aggregated or support usage data sharing settings, follow these steps:

Click Settings > Instrumentation in Splunk Web.
Click the gear icon next to Usage Data.
Adjust the sliders to enable or disable sharing aggregated or support usage data.

Opt out of sharing license data automatically

By default, Splunk collects license usage data based on your installed license to ensure compliance with your purchased offering. To disable sharing license data automatically, edit your local copy of the telemetry.conf configuration file and set sendLicenseUsage = false.

Certain license programs require that you report your license usage. The easiest way to do this is to automatically send this information to Splunk. If you disable automatic license data sharing, you can send license data manually. Follow these steps each time you want to send data manually:

On a search head, log into Splunk Web.
Select Settings > Instrumentation.
Click Export.
Select a date range and data type.
Click Send to send data to Splunk directly or click Export to export the data to your local machine and send the data to Splunk using another mechanism.

Opt out of sharing software version data

To stop sending Splunk data about the version of Splunk Enterprise you have installed, edit the web.conf configuration file and set the value of the updateCheckerBaseURL setting to 0.

In addition, you can turn off version data sharing for each Splunk app. To disable notifications of new versions and stop sending Splunk data about the app version, set check_for_updates to false in the local copy of the app.conf file for each app.

Opt out of sharing data and prevent future admins from opting in

To opt out from all collection of usage, support, and license data and prevent other admins from enabling it in the future, do the following on one search head in each cluster and on each non-clustered search head:

Click Settings > Instrumentation in Splunk Web.
Click the gear icon next to Usage Data.
Disable all options.
Click Settings > Roles.
Remove the edit_telemetry_settings capability from the admin role. Users with this role no longer receive notifications about data collection, nor can they access Settings > Instrumentation in Splunk Web.

If you want to disable collection of usage information across multiple deployments of the Splunk platform that are not centrally managed, block DNS resolution of e1345286.api.splkmobile.com.

How to adjust your data collection schedule

If you share data, the collection process begins daily at 3:00 AM by default. You can change the frequency and timing of this collection.

If all instances in your deployment are running Splunk Enterprise version 7.1.0 or later, you can schedule instrumentation to run starting at any hour of the day on a daily or a weekly schedule. The collection process runs a few searches in sequence on several instances in your deployment. Depending on the size of your deployment and whether you run instrumentation daily or weekly, it can take a few minutes before the final searches run on the primary instance to package and send the data to Splunk. See Which instance runs the searches.

Changing the instrumentation collection schedule has trade-offs. Scheduling the collection to run weekly instead of daily might decrease the total search load for the week. A weekly collection takes longer than a daily collection, because it gathers data from all seven days. If you choose weekly collection, set it for a day and time when you expect the search load to be low.

Change the collection schedule using Splunk Web

On a search head, in Splunk Web, navigate to Settings > Instrumentation.
Next to Usage Data, click the gear icon.
Click Edit usage data schedule.
Select a frequency, day, and time.
Click Save.

You do not need to restart the search head.

Change the collection schedule using configuration files

You can change the collection schedule by editing the telemetry.conf file. For guidelines on editing this file, see telemetry.conf.

At the command line on any search head, navigate to $SPLUNK_HOME/etc/apps/splunk_instrumentation/local/.
Create or edit telemetry.conf.
Edit the values for any of scheduledHour, scheduledDay, and reportStartDate according to the guidelines in telemetry.conf.spec.

Impacts on performance during collection of shared data

Aggregated usage, support usage, and license usage data is summarized and sent once per day at around 03:00 (3 am) by default. Splunk tested the performance impact on a deployment of one search head and three indexers and found the following performance impacts during the time that the searches were running:

4.5% increase in CPU overhead
Negligible effects on memory, disk, and network overhead
Up to 5% increase on the search time of regular search workloads

Session data and update checker data is sent from your browser as the events are generated. The performance implications are negligible.

How to enable data sharing for Splunk Assist

If you want to use the Splunk Assist service to monitor your Splunk Enterprise deployment according to Splunk best practices, or need to turn data sharing back on after you have opted out, use this procedure to confirm that data sharing is active.

Log into your Splunk Enterprise instance.
From the system bar, click Settings > Instrumentation.
On the "Instrumentation" page, click the gear icon next to Usage Data.
In the pop-up window that appears, review the Aggregated Usage Data and Support Usage Data toggle switches. Ensure that both toggle switches are set to "Enabled".
Click the gear icon again to close the Usage Data settings popup.

Data sharing is now on.

Related answers from Splunk Community

Share performance and usage data in Splunk Enterprise

Changes in version 9.1.0

Benefits of sharing data with Splunk

What data Splunk collects

Examples of data sent to Splunk

Aggregated usage data examples

Support usage data examples

License usage data examples

Software version data examples

App usage data examples

How Splunk collects the data

Which instance runs the searches and sends data to Splunk

Instrumentation in the Splunk Enterprise file system

How Splunk uses the data it collects

How Splunk transmits and stores the data it collects

View the data your Splunk Enterprise deployment sends to Splunk

How to opt out

Opt out of sharing aggregated or support usage data

Opt out of sharing license data automatically

Opt out of sharing software version data

Opt out of sharing data and prevent future admins from opting in

How to adjust your data collection schedule

Change the collection schedule using Splunk Web

Change the collection schedule using configuration files

Impacts on performance during collection of shared data

How to enable data sharing for Splunk Assist

Comments

Share performance and usage data in Splunk Enterprise

Was this topic useful?