Clone data

Clone data

In cloning, the forwarder sends duplicate copies of data to multiple target groups of receivers. Each target group can be either a single receiving server or a load-balanced group of receivers.

Cloning has value for enabling a number of key use cases, such as:

• Providing data redundancy to promote data availability
• Geo-diverse dataset replication, for fast local access
• Migration from one Splunk instance to another (not migration of past data)

In this simple scenario, three forwarders send duplicate copies of data to two Splunk servers:

Enable data cloning

The most direct way to set up cloning is by editing outputs.conf. Simply create multiple target groups. Each target group will automatically receive all the forwarder's data. Here is an example of specifying two target groups in a single outputs.conf file:

[tcpout]
...

[tcpout:indexer1]
server=10.1.1.197:9997

[tcpout:indexer2]
server=10.1.1.200:9999

The forwarder will send duplicate data streams to the servers specified in both the indexer1 and indexer2 target groups.

Provide data redundancy

Data cloning provides a good solution for situations requiring data redundancy. You can use a forwarder to send all data to two or more target groups. If a server in one target group goes down, users can continue to search their data by switching to another target group.

Although the target groups can each consist of single Splunk receivers, the recommended approach is to set up target groups of multiple load-balanced receivers. That way, if a server within a target group goes down while receiving data, the forwarder will automatically start forwarding data to the next server in the group, ensuring that the target group in total still receives all forwarded data. This provides a measure of protection by helping to ensure that two complete sets of the cloned data will exist in your system.

This example outputs.conf file configures a forwarder to clone raw data to two load-balanced target groups of indexers, with the indexing servers specified directly in the target groups. You can also use DNS lists to specify the target group servers, as described in DNS list target in this manual.

[tcpout]
indexAndForward=false

[tcpout:cloned_group1]
sendCookedData=false
autoLB=true
server=10.10.10.1:9997,10.10.10.2:9997,10.10.10.3:9997

[tcpout:cloned_group2]
sendCookedData=false
autoLB=true
server=10.1.1.197:9997, 10.1.1.198:9997, 10.1.1.199:9997, 10.1.1.200:9997

The forwarder will send full data streams to both cloned_group1 and cloned_group2. The data will be load-balanced within each group, rotating among receivers every 30 seconds (the default frequency).

Specify cloning from the CLI

You can also use the CLI to specify cloning. You do this when you start forwarding activity to a set of receivers, using this syntax:

./splunk add forward-server <host>:<port> -method=clone

where <host>:<port> is the host and port number of the receiver.

This example sends cloned data to two receivers:

./splunk add forward-server -method=clone indexer1:9991 
./splunk add forward-server -method=clone indexer2:9991 

• Was this topic useful?
• Post a comment