Troubleshoot knowledge bundle replication

Several tools are available to view various aspects of the replication process:

• The Settings facility in Splunk Web
• The monitoring console
• CLI commands
• REST API endpoints
• The metrics.log file

Use Splunk Web to view replication status

After you add search peers to the search head, as described in Add search peers to the search head, you can view the replication status of the knowledge bundle:

1. On the search head, click Settings at the top of the Splunk Web page.

2. Click Distributed search in the Distributed Environment area.

3. Click Search peers.

There is a row for each search peer. The column Replication status indicates whether the search head is successfully replicating the knowledge bundle to the search peer.

Note: In the case of a search head cluster, you must view replication status from the search head cluster captain. This is because only the captain replicates the knowledge bundle to the cluster's search peers. The other cluster members do not participate in bundle replication. If you view the search peers' status from a non-captain member, the Replication status column might read "Initial" instead of "Successful."

Use the monitoring console to view current and historical replication activity

You can use the monitoring console to monitor most aspects of your deployment. This section discusses the console dashboards that provide insight into knowledge bundle replication.

The primary documentation for the monitoring console is located in Monitoring Splunk Enterprise.

To view the dashboards, look on the search head, under the monitoring console's Search menu and Knowledge Bundle Replication submenu. There are these dashboards:

• Knowledge Bundle Replication. This dashboard provides information on knowledge bundle replication, including issues such as current configuration and historical bundle replication activity.
• Cascading Replication. This dashboard provides insight into the cascading bundle replication process, both current and historical. As its name suggests, this dashboard is valid only if the replication policy is set to "cascading".

To view the dashboards, ensure that the monitoring console is configured in "Distributed Mode".

Use the CLI to view bundle replication configuration and status

View bundle replication configuration

To check the bundle replication configuration, run the following command from the search head:

splunk show bundle-replication-config

The following examples show results for both classic and cascading replication policies:

     ***Knowledge Bundle Replication Configuration***
 
Replication Policy: classic
Replication Threads: 9
Max Bundle Size: 2147483648 bytes
Status Queue Size: 5
Replication Period: 60s
File Size Warning Threshold: 524288000 bytes
Connection Timeout: 60s
Send Timeout: 60s
Receive Timeout: 60s


     ***Knowledge Bundle Replication Configuration***
 
Replication Policy: cascading
Replication Threads: 9
Max Unchanged Status Periods: 5
Max Bundle Size: 2147483648 bytes
Status Queue Size: 5
Replication Period: 60s
File Size Warning Threshold: 524288000 bytes
Connection Timeout: 60s
Send Timeout: 60s
Receive Timeout: 60s

View bundle replication cycle information

To check the progress of bundle replication, run the following command from the search head:

splunk show bundle-replication-status

The following examples show results for both classic and cascading replication policies:

     ***Knowledge Bundle Replication Cycle Status***
 
Replication Policy: classic
Replication In Progress: 0
Bundle ID: qa-centos7x64-056-1566601708
Cycle ID: F4971121-0C6E-4A88-9B9B-4F273AACF58B
Current Bundle: /root/splunk_install/var/run/qa-centos7x64-056-1566601708.bundle
Current Replication Start Time: 1566601710
Peers:
    Peer URI: https://10.140.126.37:8089
    Peer State: succeeded
 
    Peer URI: https://10.140.127.113:8089
    Peer State: succeeded
 
    Peer URI: https://10.140.127.79:8089
    Peer State: succeeded


     ***Knowledge Bundle Replication Cycle Status***
 
Replication Policy: cascading
Replication In Progress: 0
Bundle ID: qa-centos7x64-056-1566591363
Cycle ID: 6ADFDA1F-5342-4AD8-8277-F8E5F017F39B
Current Bundle: /root/splunk_install/var/run/qa-centos7x64-056-1566591363.bundle
Current Replication Start Time: 1566591365
Status Unchanged Periods: 1
Plans:
    PlanID: 6C216817-A3C4-4004-922D-4262C0FBCACD
Peers:
    Peer URI: https://10.140.126.37:8089
    Peer State: cascade_replication_succeeded
    Bundle Prev State: payload_not_started
    Bundle Curr State: apply_succeeded
    Peer Plan Id: 6C216817-A3C4-4004-922D-4262C0FBCACD
 
    Peer URI: https://10.140.127.113:8089
    Peer State: cascade_replication_succeeded
    Bundle Prev State: payload_not_started
    Bundle Curr State: apply_succeeded
    Peer Plan Id: 6C216817-A3C4-4004-922D-4262C0FBCACD
 
    Peer URI: https://10.140.127.79:8089
    Peer State: cascade_replication_succeeded
    Bundle Prev State: payload_not_started
    Bundle Curr State: apply_succeeded
    Peer Plan Id: 6C216817-A3C4-4004-922D-4262C0FBCACD

Use the REST API to view bundle replication configuration and status

You can use REST API endpoints to view bundle replication configuration and status information.

View bundle replication configuration

To check the bundle replication configuration, use the following endpoint from the search head:

/services/search/distributed/bundle/replication/config

For details, see search/distributed/bundle/replication/config in the REST API Reference Manual.

View bundle replication cycle information

To check the progress of bundle replication, use the following endpoints on the search head:

For all cycles:

 /services/search/distributed/bundle/replication/cycles

For the latest cycle:

/services/search/distributed/bundle/replication/cycles?latest=true

For a particular cycle:

/services/search/distributed/bundle/replication/cycles/<cycle_id>

For details, see search/distributed/bundle/replication/cycles in the REST API Reference Manual.

Use metrics.log to access bundle replication data

The metrics.log file provides bundle replication metrics.

Bundle metadata metrics

Bundle metadata metrics include items such as bundle id, bundle size, timestamp, checksum, and so on. The following examples show the types of metadata metrics available on senders and receivers.

Sender-side metrics:

08-20-2019 23:23:10.060 -0700 INFO  Metrics - group=bundle_replication, name=bundle_metadata, bundle_id=bfedc170-0bee-4c20-bb71-8e50890fce32-1566368578, tar_time_msec=767, bundle_bytes=262451200, bundle_type=full_bundle
 
08-20-2019 23:23:10.060 -0700 INFO  Metrics - group=bundle_replication, name=bundle_metadata, bundle_id=bfedc170-0bee-4c20-bb71-8e50890fce32-1566368578, tar_time_msec=161, bundle_bytes=25671680, bundle_type=delta_bundle

Receiver-side metrics:

08-21-2019 11:19:35.100 -0700 INFO  Metrics - group=bundle_replication, name=bundle_metadata, bundle_id=bfedc170-0bee-4c20-bb71-8e50890fce32-1566411543, bundle_type=full_bundle, apply_time_msec=852, bundle_bytes=239032320
 
08-21-2019 11:21:39.100 -0700 INFO  Metrics - group=bundle_replication, name=bundle_metadata, bundle_id=bfedc170-0bee-4c20-bb71-8e50890fce32-1566411674, bundle_type=delta_bundle, apply_time_msec=302, bundle_bytes=327680

Cycle dispatch metrics

The following example illustrates the cycle-wide metrics:

08-20-2019 23:23:41.061 -0700 INFO  Metrics - group=bundle_replication, name=cycle_dispatch, cycle_id=c9b4fa33-26f8-4aeb-85d7-7c1f79740823, bundle_id=BFEDC170-0BEE-4C20-BB71-8E50890FCE32-1566368578, peer_count=4, peer_success_count=4, replication_time_msec=39844, bundle_bytes=262451200, delta_bundle_bytes=25671680

Per peer metrics

The following example illustrates the bundle replication metrics for each participating peer.

08-20-2019 23:23:41.061 -0700 INFO  Metrics - group=bundles_uploads, name=peer_dispatch, peer_name=fool18.sv.splunk.com, bundle_id=BFEDC170-0BEE-4C20-BB71-8E50890FCE32-1566368578,
cycle_id=C9B4FA33-26F8-4AEB-85D7-7C1F79740823, replication_time_msec=14854, bundle_type=delta_bundle, status=success