Customize your forwarder configuration

This topic covers customizations or additional configurations you can use when setting up forwarders for your deployment.

It is possible to configure the universal forwarders for your to use an HTTP load balancer.

You may either use Splunk to Splunk (S2S) service over TCP or HTTP forwarders on your deployment, not both. If you already have a forwarder group using the S2S service over TCP, you will need to remove it before you can configure an HTTP forwarder group.

Before you configure an HTTP forwarder group, you will need to obtain an HEC token from your Splunk Enterprise or Splunk Cloud Platform administrator. See Set up and use HTTP Event Collector in Splunk Web in Getting Data In.

To add an HTTP forwarder group, follow these steps:

1. From the Home menu, select Administration, then Administration Settings. Then select Forwarder Settings.
2. Select +New Group.
3. Select the HTTP forwarder type.

   You cannot create an HTTP forwarder group if you have an existing TCP forwarder group.

4. In the Add a new forwarder group dialog do the following:
   1. In the Name field, type a name for your forwarder group. This name is displayed on the Forwarder Settings page.
   2. Add your HEC token to the HEC token field.
   3. In the Indexer field, add the address for your for your HTTP load balancer. HTTP forwarders can only have a single indexer set.
   4. Select the Data types you want to ingest into Splunk Cloud Platform or Splunk Enterprise.
5. If you do not want to test the connection before the configuration is saved, check the Skip connection check box.
6. Make sure the Enabled slider button is in the on position.
7. Click Save.

After you complete these steps, data will begin to stream from to your Splunk Enterprise deployment through your load balancer.

Adding TLS configuration to your HTTP forwarder group

You can add TLS certificate configuration for a HTTP forwarder group the same way as you would for a TCP forwarder group.

1. On the Forwarder Settings page, click the edit icon on the right-hand end of the forwarder group's entry.
2. Click the Certificate configuration tab.
3. Add your client certificate bundle either by dragging and dropping it onto the box provided, or by clicking the box and navigating to the bundle on your filesystem.
4. (Conditional) If your Client certificate bundle includes an encrypted private key, type your client certificate password in the Client certificate password box.
5. Add your TLS certificate by dragging and dropping the certificate onto the box provided, or by clicking the box and navigating to the certificate on your filesystem.
6. (Optional) Select options as needed:
   • Verify server certificate
   • Verify server name
   • Use client SSL compression
7. (Optional) If you use common names or Subject Alt names for your servers, add them as comma-separated lists to the Allowed common names or Allowed Subject Alt names fields.
8. Click Save.

For more information on configuring universal forwarders to use HTTP, see Configure the universal forwarder to send data over HTTP in the Forwarder Manual.

Define a custom index per instance

This feature is deprecated.
Custom indexes for Splunk SOAR (On-premises) data is deprecated as of release 6.2.0. Existing custom indexes will remain, but no new custom indexes can be created. Although this feature continues to function, it might be removed in a future version.

If you have multiple instances in your environment, index prefixes are applied to the indexes in the Splunk Cloud Platform or Splunk Enterprise created by the Splunk App for SOAR. For more information on indexes, see Set up remote search on a standalone Splunk Cloud Platform or Splunk Enterprise instance in the Install and Configure Splunk App for SOAR manual.

Use the custom prefix to create separate indexes for each instance, which provides data separation and the ability to correlate each index with the appropriate instance.

Define a custom prefix with a standalone external Splunk Cloud Platform or Splunk Enterprise deployment

This feature is deprecated.
Custom indexes for Splunk SOAR (On-premises) data is deprecated as of release 6.2.0. Existing custom indexes will remain, but no new custom indexes can be created. Although this feature continues to function, it might be removed in a future version.

Perform the following tasks on each Splunk SOAR instance to create a custom prefix for each instance with a standalone external Splunk Cloud Platform or Splunk Enterprise deployment for search:

1. Verify that your instance is connected to the Splunk Cloud Platform or Splunk Enterprise by setting up the search settings using a standalone external Splunk instance:
   1. Follow the instructions in Configure the service with Splunk App for SOAR in Install and Configure Splunk App for SOAR.
   2. Make sure to click Test Connection at the end of the procedure and verify that and the Splunk Cloud Platform or Splunk Enterprise are connected.
2. Log in to the instance as the root user. In unprivileged environments, run the script as the specific user configured to run .
3. On each instance, run the set_preference command:

   phenv python -m manage set_preference --splunk-index-prefix="<prefixstring>" --splunk-admin-username <splunkadminusername>

   For example, to set a custom prefix called prefix1 using admin as the admin user for the Splunk Cloud Platform or Splunk Enterprise:

   phenv python -m manage set_preference --splunk-index-prefix="prefix1" --splunk-admin-username admin

   Use an empty prefix string to remove a custom prefix. For example:

   phenv python -m manage set_preference --splunk-index-prefix="" --splunk-admin-username admin

   In Splunk SOAR clusters, the script updates the prefix for all nodes in the cluster.

4. Users on the Splunk Cloud Platform or Splunk Enterprise inherit index permissions from their roles. After creating the new indexes, you can update roles to give all users in the role access to the new indexes, or create new users and new roles to give access to the new indexes. This example shows how to edit the phantomsearch and phantomdelete roles to grant users access to the new indexes.
   1. From Splunk Web, select Settings > Roles.
   2. Click the name of the role you want to edit, such as phantomsearch.
   3. Click the Indexes tab.
   4. Check the boxes next to the names of the new indexes.
   5. Click Save.
   6. Perform this procedure again to grant access to the new indexes for the phantomdelete role.
5. If you need additional custom roles to manage only the new indexes this example shows how to create them.
   1. From Splunk Web, select Settings > Roles.
   2. Click New Role.
   3. Type a name for the role.
   4. On the Inheritance tab, select the existing role you want your new role to inherit from, such as phantomsearch.
   5. Click the Indexes tab.
   6. Check the boxes next to the names of the new indexes.
   7. Uncheck the boxes next to the names of the indexes the new role should not be able to access.
   8. Click Create.
   9. Click the name of the role you want to edit, such as phantomsearch.
   10. Click the Indexes tab.
   11. Uncheck the boxes next to the names of the new indexes. This will prevent items managed by the new role from being repeated in indexes by phantomsearch.
   12. Click Save.
   13. Perform this procedure again to create a new role with access to the new indexes for the phantomdelete role. Custom roles used for deletions must inherit permissions from the phantomdelete role.
6. After the prefix is created, update the Splunk administration for the HEC token to grant access to the new indexes. See Set up the HTTP Event Collector on the standalone Splunk Cloud Platform or Splunk Enterprise instance in the Install and Configure Splunk App for SOAR manual for instructions.
7. Perform this step if you are using a cluster. Run the following commands on each node in your cluster:

   pkill --full add_to_searchindex
   <PHANTOM_HOME>/bin/phsvc restart uwsgi
   

8. Reindex all indexes to search for the data created while using the new prefixes. See Reindex data in the Install and Configure Splunk App for SOAR manual.

Define a custom prefix with a distributed external Splunk Cloud Platform or Splunk Enterprise deployment

This feature is deprecated.
Custom indexes for Splunk SOAR (On-premises) data is deprecated as of release 6.2.0. Existing custom indexes will remain, but no new custom indexes can be created. Although this feature continues to function, it might be removed in a future version.

Perform the following tasks on each instance to create a custom prefix for each instance with a distributed external Splunk Cloud Platform or Splunk Enterprise deployment for search:

The custom prefix script is not supported for use with distributed deployments that are built in the Splunk Cloud Platform.

1. Verify that your instance is connected to the Splunk Cloud Platform or Splunk Enterprise by setting up the search settings using a distributed external Splunk instance:
   1. Follow the instructions in Set up remote search on a distributed Splunk Cloud Platform or Enterprise instance in the Install and Configure Splunk App for SOAR manual.. The Splunk App for SOAR must be installed on all search heads in the cluster.
   2. Make sure to click Test Connection at the end of the procedure and verify that and the Splunk Cloud Platform or Splunk Enterprise are connected.
2. Log in to the instance as the root user. In unprivileged environments, run the script as the specific user configured to run .
3. On each instance, run the set_preference command:

   phenv python -m manage set_preference --splunk-index-prefix="<prefixstring>" --splunk-admin-username <splunkadminusername>

   For example, to set a custom prefix called prefix1 using admin as the admin user for the Splunk Cloud Platform or Splunk Enterprise:

   phenv python -m manage set_preference --splunk-index-prefix="prefix1" --splunk-admin-username admin

   Use an empty prefix string to remove a custom prefix. For example:

   phenv python -m manage set_preference --splunk-index-prefix="" --splunk-admin-username admin

   In Splunk SOAR clusters, the script updates the prefix for all nodes in the cluster.

   Below is sample output from the command run in a unprivileged cluster with a distributed Splunk Enterprise deployment:

   [phanru@phantom ~]$ phenv python -m manage set_preference --splunk-index-prefix prefix1 --splunk-admin-username admin
   Are you sure you wish to apply search index prefix prefix1 for this Phantom instance [yes/no]? yes
   Proceeding ... index configuration stored: /home/phanru/phantomcyber/tmp/indexes.conf
   Done! Next steps:
   - indexes.conf must be updated via splunk cluster manager node.
   - On Splunk Cloud Platform or Splunk Enterprise, edit permissions to allow the current or new HEC token to access new indexes.
   - On Splunk Cloud Platform or Splunk Enterprise, edit permissions to allow the current or new search/delete users to access new indexes.
   - If new HEC token or users are created, update the Phantom search settings.
   Run `pkill --full add_to_searchindex` on each Phantom cluster node
   Run `/home/phanru/phantomcyber/bin/phsvc restart uwsgi` on each Phantom cluster node
   - Rerun Test Connection.
   - All phantom search indexes must now be re-indexed.
   

   Note the location of the new indexes.conf file created by the script. You will need this information in the next step.
4. Edit and save the contents of the new indexes.conf file that was created by the phenv set_preference --splunk-index-prefix command. In our example, we can use cat to view and copy the contents of the <PHANTOM_HOME>/tmp/indexes.conf file.
5. In the manager node of the Splunk indexer cluster, append the contents of the new indexes.conf file to the local indexes.conf file on the manager node, such as /opt/splunk/etc/manager-apps/_cluster/local/indexes.conf.

   As of Splunk Enterprise version 9.0, "master-apps" has been updated to "manager-apps".

6. Run the following commands to push the new indexes.conf to the other indexers in the cluster and verify:

   /opt/splunk/bin/splunk apply cluster-bundle --answer-yes
   /opt/splunk/bin/splunk show cluster-bundle-status
   

7. Users on the Splunk Cloud Platform or Splunk Enterprise inherit index permissions from their roles. After creating the new indexes, you can update roles to give all users in the role access to the new indexes, or create new users and new roles to give access to the new indexes. This example shows how to edit the phantomsearch and phantomdelete roles to grant users access to the new indexes.
   1. From Splunk Web, select Settings > Roles.
   2. Click the name of the role you want to edit, such as phantomsearch.
   3. Click the Indexes tab.
   4. Check the boxes next to the names of the new indexes.
   5. Click Save.
   6. Perform this procedure again to grant access to the new indexes for the phantomdelete role.
8. If you need additional custom roles to manage only the new indexes this example shows how to create them.
   1. From Splunk Web, select Settings > Roles.
   2. Click New Role.
   3. Type a name for the role.
   4. On the Inheritance tab, select the existing role you want your new role to inherit from, such as phantomsearch.
   5. Click the Indexes tab.
   6. Check the boxes next to the names of the new indexes.
   7. Uncheck the boxes next to the names of the indexes the new role should not be able to access.
   8. Click Create.
   9. Click the name of the role you want to edit, such as phantomsearch.
   10. Click the Indexes tab.
   11. Uncheck the boxes next to the names of the new indexes. This will prevent items managed by the new role from being repeated in indexes by phantomsearch.
   12. Click Save.
   13. Perform this procedure again to create a new role with access to the new indexes for the phantomdelete role. Custom roles used for deletions must inherit permissions from the phantomdelete role.
9. After the prefix is created, update the Splunk administration for the HEC token to grant access to the new indexes. See Set up the HTTP Event Collector on the distributed Splunk Cloud Platform or Enterprise instance in the Install and Configure Splunk App for SOAR manual for instructions.
10. Perform this step if you are using a cluster. Run the following commands on each node in your cluster:

    pkill --full add_to_searchindex
    <PHANTOM_HOME>/bin/phsvc restart uwsgi
    

11. Reindex all indexes to search for the data created while using the new prefixes. See Reindex data in the Install and Configure Splunk App for SOAR manual.

Use a custom prefix when you want to change your Splunk Cloud Platform or Splunk Enterprise instance

This feature is deprecated.
Custom indexes for Splunk SOAR (On-premises) data is deprecated as of release 6.2.0. Existing custom indexes will remain, but no new custom indexes can be created. Although this feature continues to function, it might be removed in a future version.

If you have a situation where you want to use the same custom prefix on your instance with a different or new Splunk Cloud Platform or Splunk Enterprise instance, perform the following tasks:

1. Follow the instructions in either Set up remote search on a standalone Splunk Cloud Platform or Enterprise instance or Set up remote search on a distributed Splunk Cloud Platform or Enterprise instance in the Install and Configure Splunk App for SOAR manual to connect your instance with the Splunk Cloud Platform or Splunk Enterprise.
2. Run the set_preference command to create the new prefix.
3. Update the Splunk administration for the HEC token to grant access to the new indexes.
4. Reindex all indexes to search for the data created while using the new prefixes.