Docs » SignalFx Smart Agent » Install and configure the SignalFx Smart Agent

Install and configure the SignalFx Smart Agent 🔗

Note

The SignalFx Smart Agent has reached End of Support. While the agent can capture and export telemetry to Splunk Observability Cloud, Splunk no longer provides any support, feature updates, security, or bug fixes. Such requests are not bound by any SLAs.

The SignalFx Smart Agent gathers host performance, application, and service-level metrics from both containerized and non-container environments. This page provides a complete list of Smart Agent resources. Use your browser’s search function to quickly find what you’re looking for.

See SignalFx Smart Agent commands reference for a list of the most commonly used commands for the SignalFx Smart Agent.

Components 🔗

The agent has three main components:

  • Observers that discover applications and services running on the host. For a list of supported observers and their configurations, see observer configuration .

  • Monitors that collect metrics, events, and dimension properties from the host and applications. For a list of supported monitors and their configurations, see monitor configuration .

  • The writer that sends the metrics, events, and dimension updates collected by monitors to Splunk Observability Cloud. The writer collects metrics emitted by configured monitors and sends them to Splunk Observability Cloud on a regular basis. You can configure writer settings in the configuration schema.

Use cases 🔗

The Smart Agent gathers metrics using monitors, including Python-based plugins such as Mongo, Redis, and Docker. See Supported integrations in Splunk Observability Cloud for a list of data source integrations.

Use the Smart Agent to integrate with cloud services, including Amazon Web Services, Microsoft Azure, Google Cloud Platform, and Kubernetes environments. See Connect to your cloud service provider. Next, log in to Splunk Observability Cloud to view the incoming metrics in Dashboards in Splunk Observability Cloud and Charts in Splunk Observability Cloud.

Check out the health of your network and nodes using navigators. Some features of Splunk Observability Cloud such as related content do not work with the Smart Agent. To learn more, see Use navigators in Splunk Infrastructure Monitoring.

The Smart Agent also supports receiving and sending trace data. See Download traces.

Check exposed ports 🔗

Before installing the Smart Agent, check exposed ports to make sure your environment doesn’t have conflicts. You can change the ports in the Smart Agent configuration.

  • Port 8095 is the default port that the internal status server listens on. Configure port 8095 using the internalStatusPort option.

  • Port 9080 is the default port that the server listens on. Configure port 9080 using the listenAddress option. The listenAddress option is a configurable option for the trace-forwarder monitor.

Install the Smart Agent 🔗

There are several options available to install the Smart Agent. Select the option that matches your situation or preference. Each of the following links includes prerequisites, configuration instructions, installation instructions, and instructions for verifying your installation.

The Smart Agent is incompatible on Linux systems with SELinux activated. Check the documentation for your distribution to learn how to deactivate SELinux.

Note

To uninstall the Smart Agent, see Uninstall the Smart Agent.

Configure the Smart Agent 🔗

You can configure the Smart Agent by editing the agent.yaml file. By default, the configuration is installed at and looked for at /etc/signalfx/agent.yaml on Linux and \ProgramData\SignalFxAgent\agent.yaml on Windows. You can override default locations using the -config command line flag.

The configuration schema includes the options that you can use in the agent.yaml file to control the behavior of your integrations. Example.yaml provides an autogenerated example of a YAML configuration file, with default values where applicable. Remote configuration describes how to configure the Smart Agent from remote sources, such as other files on the file system, or from key-value stores such as etcd.

Capture logs using the Smart Agent 🔗

The default log level is info, which logs Smart Agent events without spamming the logs. Most of the info level logs are created upon startup and capture service discovery changes to record messages about routine operations. The debug log level creates verbose log output and should only be used when troubleshooting.

You can change the log level using the logging: {level: info} YAML configuration option. Other valid log levels include warn and error.

The Smart Agent sends logs either as unstructured text (default) or JSON format. You can configure the Smart Agent to send JSON logs using the logging: {format: json} YAML configuration option.

Linux 🔗

The Smart Agent supports logging to stdout/stderr, which is generally redirected by the init scripts provided to either a file at /var/log/signalfx-agent.log or to the systemd journal on newer distros.

Windows 🔗

On Windows, the Smart Agent logs to the console when executed directly in a shell. If the Smart Agent is configured as a Windows service, log events are logged to the Windows Event Log. To read logs, select Start. Next, select Administrative Tools, then Event Viewer.

Activate proxy support in the Smart Agent 🔗

To use an HTTP or HTTPS proxy, set the environment variable HTTP_PROXY or HTTPS_PROXY in the container configuration to proxy either protocol. The SignalFx ingest and API servers both use HTTPS. If the NO_PROXY environment variable exists, the Smart Agent automatically appends the local services to the environment variable to not use the proxy.

If the Smart Agent is running as a local service on the host, refer to the host documentation for information on passing environment variables to the Smart Agent service to activate proxy support when the service is started. For example, if the host services are managed by systemd, create the /etc/systemd/system/signalfx-agent.service.d/myproxy.conf file and add the following to the file:

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:8080/"
Environment="HTTPS_PROXY=https://proxy.example.com:8081/"

After updating the file, run systemctl daemon-reload and systemctl restart signalfx-agent.service to restart the service with proxy support.

Sys-V based init.d systems: Debian and RHEL 🔗

Create the /etc/default/signalfx-agent.yaml file with the following contents:

HTTP_PROXY="http://proxy.example.com:8080/"
HTTPS_PROXY="https://proxy.example.com:8081/"

Smart Agent diagnostics 🔗

The Smart Agent serves diagnostic information on an HTTP server at the address configured by the internalStatusHost and internalStatusPort options. Use the signalfx-agent command status to read the server and extract its contents. Use the content to identify and resolve issues with the Smart Agent. The signalfx-agent command also explains how to get further diagnostic information.

Service discovery using the Smart Agent 🔗

The Smart Agent includes a comprehensive service discovery feature. This feature allows the Smart Agent to identify each of the services within your environment and automatically configure the appropriate integration plugins from within its bundle. This is particularly valuable in large ephemeral container environments that experience high-churn and dynamic service creation, as new services are automatically discovered, installed, and configured within minutes. However, this capability is extended to non-containerized environments as well. See Service Discovery for more information and configuration options.

Filtering data using the Smart Agent 🔗

Filter out certain data points or properties to prevent them from ever leaving the Smart Agent. Filtering can be useful to reduce clutter in charts without having to resort to filtering in the UI.

If possible, it is preferable to prevent the data points and properties you want to omit from being generated by a monitor in the first place, as this reduces CPU and memory usage of the Smart Agent, but sometimes this is not feasible.

See Filtering for more information and configuration options for Smart Agent 4.7.0+. See Legacy Filtering for more information on the old style of filtering, which is deprecated and removed in Smart Agent 5.0+.

Uninstall the Smart Agent 🔗

To remove the Smart Agent from your system, follow the steps for each installation method.

Linux (package manager) 🔗

  • For Debian-based distributions, including Ubuntu, run the following command:

    sudo dpkg --remove signalfx-agent

  • For Red Hat, CentOS, and other RPM-based installs, run the following command:

    sudo rpm -e signalfx-agent

Note

Configuration files might persist in /etc/signalfx.

Linux (tar file) 🔗

To uninstall the Smart Agent, stop the signalfx-agent process and delete the signalfx-agent directory.

Windows (installer) 🔗

Uninstall the Smart Agent from Programs and Features in the Control Panel.

Note

Configuration files might persist in \ProgramData\SignalFxAgent.

Windows (ZIP file) 🔗

Run the following PowerShell commands to stop and uninstall the signalfx-agent service:

SignalFxAgent\bin\signalfx-agent.exe -service "stop"
SignalFxAgent\bin\signalfx-agent.exe -service "uninstall"

Then, delete the SignalFxAgent folder.

Deployed using kubectl 🔗

To delete all Smart Agent resources, run the following command in the directory containing the YAML configuration files:

cat *.yaml | kubectl delete -f -

For more details, see the Kubectl Reference Documentation.

Deployed using Helm 🔗

To uninstall the Helm release of the Smart Agent, follow these steps:

Deployed in AWS ECS 🔗

To deregister the signalfx-agent task definitions, see the official AWS documentation at https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deregister-task-definition.html.

To delete the signalfx-agent service, see Deleting a service at https://docs.aws.amazon.com/AmazonECS/latest/developerguide/delete-service.html.

Frequently asked questions 🔗

See Frequently Asked Questions to troubleshoot issues with the Smart Agent.

This page was last updated on Dec 12, 2023.