Troubleshoot knowledge bundle replication
Several tools are available to view various aspects of the replication process:
- The Settings facility in Splunk Web
- The monitoring console
- CLI commands
- REST API endpoints
- The
metrics.log
file
Use Splunk Web to view replication status
After you add search peers to the search head, as described in Add search peers to the search head, you can view the replication status of the knowledge bundle:
1. On the search head, click Settings at the top of the Splunk Web page.
2. Click Distributed search in the Distributed Environment area.
3. Click Search peers.
There is a row for each search peer. The column Replication status indicates whether the search head is successfully replicating the knowledge bundle to the search peer.
Note: In the case of a search head cluster, you must view replication status from the search head cluster captain. This is because only the captain replicates the knowledge bundle to the cluster's search peers. The other cluster members do not participate in bundle replication. If you view the search peers' status from a non-captain member, the Replication status column might read "Initial" instead of "Successful."
Use the monitoring console to view current and historical replication activity
You can use the monitoring console to monitor most aspects of your deployment. This section discusses the console dashboards that provide insight into knowledge bundle replication.
The primary documentation for the monitoring console is located in Monitoring Splunk Enterprise.
To view the dashboards, look on the search head, under the monitoring console's Search menu and Knowledge Bundle Replication submenu. There are these dashboards:
- Knowledge Bundle Replication. This dashboard provides information on knowledge bundle replication, including issues such as current configuration and historical bundle replication activity.
- Cascading Replication. This dashboard provides insight into the cascading bundle replication process, both current and historical. As its name suggests, this dashboard is valid only if the replication policy is set to "cascading".
To view the dashboards, ensure that the monitoring console is configured in "Distributed Mode".
Use the CLI to view bundle replication configuration and status
View bundle replication configuration
To check the bundle replication configuration, run the following command from the search head:
splunk show bundle-replication-config
The following examples show results for both classic and cascading replication policies:
***Knowledge Bundle Replication Configuration*** Replication Policy: classic Replication Threads: 9 Max Bundle Size: 2147483648 bytes Status Queue Size: 5 Replication Period: 60s File Size Warning Threshold: 524288000 bytes Connection Timeout: 60s Send Timeout: 60s Receive Timeout: 60s ***Knowledge Bundle Replication Configuration*** Replication Policy: cascading Replication Threads: 9 Max Unchanged Status Periods: 5 Max Bundle Size: 2147483648 bytes Status Queue Size: 5 Replication Period: 60s File Size Warning Threshold: 524288000 bytes Connection Timeout: 60s Send Timeout: 60s Receive Timeout: 60s
View bundle replication cycle information
To check the progress of bundle replication, run the following command from the search head:
splunk show bundle-replication-status
The following examples show results for both classic and cascading replication policies:
***Knowledge Bundle Replication Cycle Status*** Replication Policy: classic Replication In Progress: 0 Bundle ID: qa-centos7x64-056-1566601708 Cycle ID: F4971121-0C6E-4A88-9B9B-4F273AACF58B Current Bundle: /root/splunk_install/var/run/qa-centos7x64-056-1566601708.bundle Current Replication Start Time: 1566601710 Peers: Peer URI: https://10.140.126.37:8089 Peer State: succeeded Peer URI: https://10.140.127.113:8089 Peer State: succeeded Peer URI: https://10.140.127.79:8089 Peer State: succeeded ***Knowledge Bundle Replication Cycle Status*** Replication Policy: cascading Replication In Progress: 0 Bundle ID: qa-centos7x64-056-1566591363 Cycle ID: 6ADFDA1F-5342-4AD8-8277-F8E5F017F39B Current Bundle: /root/splunk_install/var/run/qa-centos7x64-056-1566591363.bundle Current Replication Start Time: 1566591365 Status Unchanged Periods: 1 Plans: PlanID: 6C216817-A3C4-4004-922D-4262C0FBCACD Peers: Peer URI: https://10.140.126.37:8089 Peer State: cascade_replication_succeeded Bundle Prev State: payload_not_started Bundle Curr State: apply_succeeded Peer Plan Id: 6C216817-A3C4-4004-922D-4262C0FBCACD Peer URI: https://10.140.127.113:8089 Peer State: cascade_replication_succeeded Bundle Prev State: payload_not_started Bundle Curr State: apply_succeeded Peer Plan Id: 6C216817-A3C4-4004-922D-4262C0FBCACD Peer URI: https://10.140.127.79:8089 Peer State: cascade_replication_succeeded Bundle Prev State: payload_not_started Bundle Curr State: apply_succeeded Peer Plan Id: 6C216817-A3C4-4004-922D-4262C0FBCACD
Use the REST API to view bundle replication configuration and status
You can use REST API endpoints to view bundle replication configuration and status information.
View bundle replication configuration
To check the bundle replication configuration, use the following endpoint from the search head:
/services/search/distributed/bundle/replication/config
For details, see search/distributed/bundle/replication/config in the REST API Reference Manual.
View bundle replication cycle information
To check the progress of bundle replication, use the following endpoints on the search head:
For all cycles:
/services/search/distributed/bundle/replication/cycles
For the latest cycle:
/services/search/distributed/bundle/replication/cycles?latest=true
For a particular cycle:
/services/search/distributed/bundle/replication/cycles/<cycle_id>
For details, see search/distributed/bundle/replication/cycles in the REST API Reference Manual.
Use metrics.log to access bundle replication data
The metrics.log
file provides bundle replication metrics.
Bundle metadata metrics
Bundle metadata metrics include items such as bundle id, bundle size, timestamp, checksum, and so on. The following examples show the types of metadata metrics available on senders and receivers.
Sender-side metrics:
08-20-2019 23:23:10.060 -0700 INFO Metrics - group=bundle_replication, name=bundle_metadata, bundle_id=bfedc170-0bee-4c20-bb71-8e50890fce32-1566368578, tar_time_msec=767, bundle_bytes=262451200, bundle_type=full_bundle 08-20-2019 23:23:10.060 -0700 INFO Metrics - group=bundle_replication, name=bundle_metadata, bundle_id=bfedc170-0bee-4c20-bb71-8e50890fce32-1566368578, tar_time_msec=161, bundle_bytes=25671680, bundle_type=delta_bundle
Receiver-side metrics:
08-21-2019 11:19:35.100 -0700 INFO Metrics - group=bundle_replication, name=bundle_metadata, bundle_id=bfedc170-0bee-4c20-bb71-8e50890fce32-1566411543, bundle_type=full_bundle, apply_time_msec=852, bundle_bytes=239032320 08-21-2019 11:21:39.100 -0700 INFO Metrics - group=bundle_replication, name=bundle_metadata, bundle_id=bfedc170-0bee-4c20-bb71-8e50890fce32-1566411674, bundle_type=delta_bundle, apply_time_msec=302, bundle_bytes=327680
Cycle dispatch metrics
The following example illustrates the cycle-wide metrics:
08-20-2019 23:23:41.061 -0700 INFO Metrics - group=bundle_replication, name=cycle_dispatch, cycle_id=c9b4fa33-26f8-4aeb-85d7-7c1f79740823, bundle_id=BFEDC170-0BEE-4C20-BB71-8E50890FCE32-1566368578, peer_count=4, peer_success_count=4, replication_time_msec=39844, bundle_bytes=262451200, delta_bundle_bytes=25671680
Per peer metrics
The following example illustrates the bundle replication metrics for each participating peer.
08-20-2019 23:23:41.061 -0700 INFO Metrics - group=bundles_uploads, name=peer_dispatch, peer_name=fool18.sv.splunk.com, bundle_id=BFEDC170-0BEE-4C20-BB71-8E50890FCE32-1566368578, cycle_id=C9B4FA33-26F8-4AEB-85D7-7C1F79740823, replication_time_msec=14854, bundle_type=delta_bundle, status=success
Mounted knowledge bundle replication | Deploy a distributed search environment |
This documentation applies to the following versions of Splunk® Enterprise: 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.3.0, 9.3.1
Feedback submitted, thanks!