Splunk Cloud Platform

Use Edge Processors

Acrobat logo Download manual as PDF


Acrobat logo Download topic as PDF

Set up an Edge Processor

The first step towards incorporating the Edge Processor solution into your existing Splunk ecosystem is to configure and install an Edge Processor. An Edge Processor is a single server instance or a cluster of multiple server instances that provide computing resources for processing and routing data. You install Edge Processors in your own network so that you can reduce and sanitize your data before sending it outside of your local network. See How the Edge Processor solution works for more information about the Edge Processor architecture.

Setting up an Edge Processor involves completing the following tasks:

  1. Adding an Edge Processor in your tenant. See Add an Edge Processor for more information.
  2. Installing an Edge Processor instance on a machine in your network. See Install an Edge Processor instance for more information.
  3. If necessary, adding more instances to the Edge Processor to provide more computing resources. See Add more instances to an Edge Processor for more information.


This is step 2 of 6 for using an Edge Processor to process data and route it to a destination. To see an overview of all of the steps, see Quick start: Process and route data using Edge Processors.

This diagram shows an overview of the steps required to set up and use an Edge Processor.

Prerequisites

Make sure that the environment where you're installing the Edge Processor meets the system and network requirements. See Installation requirements for Edge Processors.

Depending on how you want to configure your Edge Processor, the following requirements might also apply:

  • When installing the Edge Processor instance, you can choose to configure systemd on your host machine to manage the underlying process of the Edge Processor instance as a service. If you want to use this configuration, you must have super user permissions on the host machine.
  • If you want to specify a default destination for storing unprocessed data, you must first create a destination. See Add or manage destinations for more information.
    • If you plan to use a Splunk platform HEC destination as the default destination, make sure that the Default source type setting in the destination specifies an appropriate value. If the events routed to this destination aren't already associated with a source type, then the Default source type value is used.
  • If you want to secure communications between your data source and your Edge Processor with TLS, then you must have the following in Privacy Enhanced Mail (PEM) format:
    • A client certificate, CA certificate, and private key that the data source can use to establish the TLS connection.
    • A server certificate, CA certificate, and private key that the Edge Processor can use to establish the TLS connection.

    These certificates can be self-signed or they can be signed by a third-party. See Obtain TLS certificates for data sources and Edge Processors for information on generating client and server certificates.

Add an Edge Processor

In the Edge Processor service, add an Edge Processor and specify configuration settings that apply to all instances of this Edge Processor.

  1. In the Edge Processor service, navigate to the Edge Processors page and then select New Edge Processor.
  2. Enter a Name and a Description for the Edge Processor.
  3. To specify a default destination that the Edge Processor sends unprocessed data to, select To a default destination. Then, from the Default destination drop-down list, select the destination you want to use.
  4. To turn on receivers that allow your Edge Processor to receive data from specific data inputs, select data inputs as necessary in the Receive data from these inputs section.
  5. If you want to use TLS to secure communications between this Edge Processor and the data sources that are sending data to it, then do the following:
    1. In the Use TLS with these inputs section, select the data inputs for which you want to use TLS encryption.
    2. Upload PEM files containing the appropriate certificates in the Server private key, Server certificate, and CA certificates fields.
    3. The Edge Processor uses the same PEM files for all data sources. For example, if you select both Splunk forwarders and HTTP Event Collector in the Use TLS with these inputs section, then the same server-side PEM files are used for both forwarder communications and HEC communications.

  6. Select Continue.

The Edge Processor service creates an Edge Processor configuration with the settings that you specified. Next, install an instance of this Edge Processor on a machine in your network.

Install an Edge Processor instance

After adding an Edge Processor in your tenant, you can install an instance associated with that Edge Processor on a host machine in your network.

As an optional configuration during this installation procedure, you can configure systemd on your host machine to manage the underlying process of the Edge Processor instance as a service. Configuring systemd to manage the splunk-edge process allows systemd to start the process at boot and automatically restart the process if it is terminated unexpectedly.

Choose the installation procedure that suits your needs:

Install an instance without configuring systemd

Use the installation commands provided in the Edge Processor service to install an Edge Processor instance.

  1. In your cloud tenant, locate and copy the installation commands.
    1. On the Edge Processors page, in the row that lists your Edge Processor, select the Actions icon (Image of the Actions icon) and then select Open.
    2. In the panel that contains your Edge Processor details, select Manage instances.
    3. Select the Install/uninstall tab, and then expand the Step 1: Run commands to install/uninstall instances section.
    4. Select Install to view the commands for downloading and installing an Edge Processor instance on a Linux machine, and then select Copy to clipboard.

      These commands contain sensitive information about your cloud environment. Do not share these commands with anyone except your Splunk representative or trusted members in your organization.

  2. On the machine where you want to install the instance, open a command-line interface in the directory where you want to install the Edge Processor instance and then paste and run the commands.
    The commands create a splunk-edge directory in your chosen installation location.
  3. To verify that the instance was installed successfully, return to your cloud tenant and select the Instances tab in the Manage instances panel. Confirm that your instance is listed and has the Healthy status. It may take up to 1 minute for the status to change to Healthy. See Manage and uninstall Edge Processors for information about instance statuses and what they mean.

You now have a single-instance Edge Processor that you can use to receive and process data. For information about creating and applying pipelines for data processing, see Create pipelines for Edge Processors.

If you want to scale up your Edge Processor to a cluster of multiple instances, see the Add more instances to an Edge Processor section for information.

Install an instance and configure systemd

When configuring systemd to manage the splunk-edge process as a service, you must associate a control group (cgroup) and a user to that service. The user must have read and write permissions for the directory where you want to install the Edge Processor instance.

The following instructions ensure that the user has the necessary permissions by setting the home directory of the user to the installation directory of the Edge Processor instance. However, if desired, you can choose to use an existing cgroup and user or configure the user permissions through another method.

  1. On the machine where you want to install the instance, create a cgroup and a user.
    1. Create a cgroup by running the following command, where <group_name> is the name of the cgroup:
      sudo groupadd <group_name>
    2. Create a user by running the following command, where <install_location> is the directory where you want to install the Edge Processor instance, <group_name> is the name of the cgroup, and <username> is the name of the user.
      sudo useradd -d <install_location>/splunk-edge -g `grep <group_name> /etc/group | awk -F ":" {'print $3'}` -m -s /bin/bash <username>
      
    3. (Optional) To confirm you've successfully created the cgroup and user, run the following commands:
      sudo grep <group_name> /etc/group
      sudo grep <username> /etc/passwd

      These commands return information about the cgroup and the user if you have successfully created them.

  2. In your cloud tenant, navigate to the panel that displays the installation commands.
    1. On the Edge Processors page, in the row that lists your Edge Processor, select the Actions icon (Image of the Actions icon) and then select Open.
    2. In the panel that contains your Edge Processor details, select Manage instances.
    3. Select the Install/uninstall tab, and then expand the Step 1: Run commands to install/uninstall instances section.
    4. Select Install to view the commands for downloading and installing an Edge Processor instance on a Linux machine, and then select Copy to clipboard.

      These commands contain sensitive information about your cloud environment. Do not share these commands with anyone except your Splunk representative or trusted members in your organization.

    5. Open a text editor and paste the commands. Delete the following command, which comes after the # Install the Edge Processor instance comment:
      nohup ./splunk-edge/bin/splunk-edge run >> ./splunk-edge/var/log/install-splunk-edge.out 2>&1 </dev/null &
    6. Copy the commands that remain.
  3. On the machine where you want to install the instance, create and populate the installation directory.
    1. Log in as the user that you created during step 1.
    2. Open a command-line interface in the directory where you want to install the Edge Processor instance.
    3. Paste and run the commands that you copied during step 2f.
      The commands create a splunk-edge directory in your chosen installation location. In the steps that follow, <install_directory> represents the fully qualified path to this splunk-edge directory. For example, if you completed step 3c in the /opt/ directory, then <install_directory> is /opt/splunk-edge.
  4. Open the <install_directory>/etc/splunk-edge.service file and make sure that the User and Group properties are set to the user and cgroup that you created during step 1. Additionally, make sure that the ExecStart property is set as follows:
    ExecStart=<install_directory>/bin/splunk-edge run
  5. To add the splunk-edge process to systemd and then finish installing the Edge Processor instance, run the following commands:
    sudo chown -R splunk: <install_directory>
    sudo cp <install_directory>/etc/splunk-edge.service /etc/systemd/system
    sudo systemctl daemon-reload
    sudo systemctl enable splunk-edge
    sudo systemctl start splunk-edge
    

    When the installation is complete, the following message is returned:

    splunk-edge.service - Splunk edge starter
        Loaded: loaded (/etc/systemd/system/splunk-edge.service, enabled)
        Active: active (running)
    
  6. To confirm that you've successfully added the splunk-edge process to systemd, run the following command:
    sudo systemctl status splunk-edge.service

    Review the status information that is returned and confirm that there are no errors.

  7. To verify that the instance is healthy, return to your cloud tenant and select the Instances tab in the Manage instances panel. Confirm that your instance is listed and has the Healthy status. It may take up to 1 minute for the status to change to Healthy. See Manage and uninstall Edge Processors for information about instance statuses and what they mean.

You now have a single-instance Edge Processor that you can use to receive and process data. For information about creating and applying pipelines for data processing, see Create pipelines for Edge Processors.

If you want to scale up your Edge Processor to a cluster of multiple instances, see the Add more instances to an Edge Processor section for information.

Add more instances to an Edge Processor

To ensure that your Edge Processor has sufficient computing resources for your data processing workload, you can scale up your Edge Processor into a cluster and add more instances as needed.

Be aware that there is a soft limit on the maximum number of Edge Processor clusters and instances that can be supported. See Tested and recommended service limits (soft limits) in the Splunk Cloud Platform Service Details for more information.

To scale up your Edge Processor by adding more instances, do the following:

  1. Install an instance on another machine in your environment. See the Install an Edge Processor instance section on this page.
  2. If you have already configured data sources to send data to this Edge Processor, then you must update their configurations to account for the added Edge Processor instance:
    Type of data source Configuration instructions
    Splunk forwarders In the outputs.conf file, update the server property to include the host and port information of your new instance. You can get an outputs.conf stanza with the settings relevant to your Edge Processor by selecting the Configure data sources action for your Edge Processor and then selecting Splunk forwarder from the drop-down list.


    As a best practice, if you have many forwarders configured to send data to the same multi-instance Edge Processor, use a DNS record to keep your outputs.conf settings up to date. Map all the Edge Processor instance hosts to a DNS record, and then set the server property in your outputs.conf files to the IP address of that DNS record. When you add or remove instances to your Edge Processor, you only need to update the DNS record instead of updating multiple outputs.conf files. For more information about using a DNS to manage forwarder outputs, see Options for configuring receiving targets for load balancing in the Splunk Cloud Platform Forwarding Data manual.

    HTTP clients or logging agents using HTTP Event Collector (HEC) If you want the HTTP client or logging agent to send data to multiple Edge Processor instances, you must set up a load balancer to pass the HTTP request to all of the instances. Then, update the URI of the HTTP request so that the request is directed to the load balancer.


    Otherwise, if you want the HTTP client or logging agent to send data to the new Edge Processor instance only, update the URI of the HTTP request so that the request is directed to the new instance. You can get HTTP request examples with hostname and port values relevant to your instance by selecting the Configure data sources action for your Edge Processor and then selecting HTTP Event Collector from the drop-down list.

You now have a multi-instance Edge Processor cluster that you can use to receive and process data. For information about creating and applying pipelines for data processing, see Create pipelines for Edge Processors.

Last modified on 22 July, 2023
PREVIOUS
Installation requirements for Edge Processors
  NEXT
Manage and uninstall Edge Processors

This documentation applies to the following versions of Splunk Cloud Platform: 9.0.2209, 9.0.2303, 9.0.2305 (latest FedRAMP release)


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters