Docs » Install and configure the SignalFx Smart Agent

Install and configure the SignalFx Smart Agent 🔗

Note

The SignalFx Smart Agent is deprecated. For details, see the Deprecation Notice.

The SignalFx Smart Agent gathers host performance, application, and service-level metrics from both containerized and non-container environments. This page provides a complete list of Smart Agent resources. Use your browser’s search function to quickly find what you’re looking for.

Note about the SignalFx Smart Agent

If you want to use Splunk Observability Cloud integration for any of the following services, you still need to deploy the SignalFx Smart Agent instead of Splunk Distribution of OpenTelemetry Collector:

Follow the links above for integration instructions for each of these services.

Note also that any metrics ingested via Splunk Distribution of OpenTelemetry Collector will not appear in the classic SignalFx UI. Smart Agent and cloud metrics will continue to appear in either experience, but you can only view OpenTelemetry-sourced metrics in the Splunk Observability Cloud UI.

Components 🔗

The agent has three main components:

  • Observers that discover applications and services running on the host. For a list of supported observers and their configurations, see observer configuration.

  • Monitors that collect metrics, events, and dimension properties from the host and applications. For a list of supported monitors and their configurations, see monitor configuration.

  • The writer that sends the metrics, events, and dimension updates collected by monitors to Splunk Observability Cloud. The writer collects metrics emitted by configured monitors and sends them to Observability Cloud on a regular basis. You can configure writer settings in the configuration schema.

Use cases 🔗

The Smart Agent gathers metrics using monitors, including Python-based plugins such as Mongo, Redis, and Docker. See Supported data sources for a list of data source integrations.

Use the Smart Agent to integrate with cloud services, including Amazon Web Services, Microsoft Azure, Google Cloud Platform, and Kubernetes environments. See Connect to your cloud service provider. Next, log in to Observability Cloud to view the incoming metrics in Dashboards in Splunk Observability Cloud and Charts in Splunk Observability Cloud.

Check out the health of your network and nodes using Splunk Infrastructure Monitoring navigators. Some features of Observability Cloud such as related content do not work with the Smart Agent.

The Smart Agent also supports receiving and sending trace data. See Download traces.

Check exposed ports 🔗

Before installing the Smart Agent, check exposed ports to make sure your environment doesn’t have conflicts. You can change the ports in the Smart Agent configuration.

  • Port 8095 is the default port that the internal status server listens on. Configure port 8095 using the internalStatusPort option.

  • Port 9080 is the default port that the server listens on. Configure port 9080 using the listenAddress option. The listenAddress option is a configurable option for the trace-forwarder and signalfx-forwarder monitors.

Install 🔗

There are several options available to install the Smart Agent. Select the option that matches your situation or preference. Each of the following links includes prerequisites, configuration instructions, installation instructions, and instructions for verifying your installation.

Note

The Smart Agent is incompatible on Linux systems with SELinux enabled. Check the documentation for your distribution to learn how to disable SELinux.

Configure 🔗

You can configure the Smart Agent by editing the agent.yaml file. By default, the configuration is installed at and looked for at /etc/signalfx/agent.yaml on Linux and \ProgramData\SignalFxAgent\agent.yaml on Windows. You can override default locations using the -config command line flag.

The configuration schema includes the options that you can use in the agent.yaml file to control the behavior of your integrations. Example.yaml provides an autogenerated example of a YAML configuration file, with default values where applicable. Remote configuration describes how to configure the Smart Agent from remote sources, such as other files on the file system, or from key-value stores such as etcd.

Capture logs 🔗

The default log level is info, which logs Smart Agent events without spamming the logs. Most of the info level logs are created upon startup and service discovery changes to record messages about routine operation. The debug log level creates verbose log output and should only be used when troubleshooting.

You can change the log level using the logging: {level: info} YAML configuration option. Other valid log levels include warn and error.

The Smart Agent sends logs either as unstructured text (default) or JSON format. You can configure the Smart Agent to send JSON logs using the logging: {format: json} YAML configuration option.

Linux 🔗

The Smart Agent supports logging to stdout/stderr, which is generally redirected by the init scripts provided to either a file at /var/log/signalfx-agent.log or to the systemd journal on newer distros.

Windows 🔗

On Windows, the Smart Agent logs to the console when executed directly in a shell. If the Smart Agent is configured as a Windows service, log events are logged to the Windows Event Log. Select Start > Administrative Tools > Event Viewer to read logs. Select Windows Logs > Application to see logged events from the Smart Agent service.

Use proxy support 🔗

To use an HTTP or HTTPS proxy, set the environment variable HTTP_PROXY and/or HTTPS_PROXY in the container configuration to proxy either protocol. The SignalFx ingest and API servers both use HTTPS. If the NO_PROXY environment variable exists, the Smart Agent automatically appends the local services to the environment variable to not use the proxy.

If the Smart Agent is running as a local service on the host, refer to the host documentation for information on passing environment variables to the Smart Agent service to enable proxy support when the service is started. For example, if the host services are managed by systemd, create the /etc/systemd/system/signalfx-agent.service.d/myproxy.conf file and add the following to the file:

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:8080/"
Environment="HTTPS_PROXY=https://proxy.example.com:8081/"

After updating the file, run systemctl daemon-reload and systemctl restart signalfx-agent.service to restart the service with proxy support.

Sys-V based init.d systems: Debian and RHEL 🔗

Create the /etc/default/signalfx-agent.yaml file with the following contents:

HTTP_PROXY="http://proxy.example.com:8080/"
HTTPS_PROXY="https://proxy.example.com:8081/"

Diagnostics 🔗

The Smart Agent serves diagnostic information on an HTTP server at the address configured by the internalStatusHost and internalStatusPort options. Use the signalfx-agent command status to read the server and extract its contents. Use the content to identify and resolve issues with the Smart Agent. The signalfx-agent command also explains how to get further diagnostic information.

Service discovery 🔗

The Smart Agent includes a comprehensive service discovery feature. This feature allows the Smart Agent to identify each of the services within your environment and automatically configure the appropriate integration plugins from within its bundle. This is particularly valuable in large ephemeral container environments that experience high-churn and dynamic service creation, as new services are automatically discovered, installed, and configured within minutes. However, this capability is extended to non-containerized environments as well. See Service Discovery for more information and configuration options.

Filtering 🔗

Filter out certain data points or properties to prevent them from ever leaving the Smart Agent. Filtering can be useful to reduce clutter in charts without having to resort to filtering in the UI.

If possible, it is preferable to prevent the data points and properties you want to omit from being generated by a monitor in the first place, as this reduces CPU and memory usage of the Smart Agent, but sometimes this is not feasible.

See Filtering for more information and configuration options for Smart Agent 4.7.0+. See Legacy Filtering for more information on the old style of filtering, which is deprecated and removed in Smart Agent 5.0+.

Frequently asked questions 🔗

See Frequently Asked Questions to troubleshoot issues with the Smart Agent.

Migrating from the Smart Agent to Splunk Distribution of OpenTelemetry Collector 🔗

Splunk Distribution of OpenTelemetry Collector is the next-generation agent and gateway for Splunk Observability products. See the following topics for information on migrating to Splunk Distribution of OpenTelemetry Collector:

  • Migrating from the SignalFx Smart Agent, which describes how to transition to Splunk Distribution of OpenTelemetry Collector without functionality loss.

  • How mapping makes upgrades easier, which describes the Mapping Service. The Mapping Service is a transition tool that defines equivalencies between legacy collectd (Smart Agent) metric naming and semantic conventions to the OpenTelemetry names and formats for metrics and metric metadata.

  • About the mapping service transition impact report, which describes how to migrate your data and metadata from dashboards, charts, and detectors from the Smart Agent to Splunk Distribution of OpenTelemetry Collector.