Docs » Org reference information » Per product system limits in Splunk Observability Cloud » System limits details for Splunk Infrastructure Monitoring

Edit this page
Learn how

---

System limits details for Splunk Infrastructure Monitoring

Splunk Infrastructure Monitoring has system limits that help ensure good performance, stability, and reliability. These limits also protect the Infrastructure Monitoring multitenant environment. Exceeding these limits might degrade your Infrastructure Monitoring experience.

This section provides details about each Infrastructure Monitoring system limit.

Email address invitations per minute

• Default limit value: 1

• Notes: You can only invite an email address to an organization once within one minute. If you try to invite the same email address multiple times in one minute, the UI returns an error, and you must wait at least one minute before sending another invitation.

Organization invitations per day

• Default limit value: 5,000

• Notes: You can only send 5,000 invitations for an organization within 24 hours. If you exceed this limit, the UI returns an error, and you must wait at least one day before sending another invitation.

Maximum number of dashboards you can retrieve

• Default limit value: 20,000

• Notes: Maximum number of dashboards you can retrieve per query using either the UI or the API. If you reach this limit, you receive an error.

• Customer impact: When you exceed this limit, the user interface displays the error message “Unexpected error has occurred”. After you exceed the limit, the dashboards page stops displaying dashboards.

Number of input MTS per job

• Default limit value: 250,000

• Notes: Maximum number of input MTS per job. When you use the same MTS multiple times in a job, each use counts towards the maximum.

• Customer impact: If the job is for a chart, the chart doesn’t load and you receive an error message. If the job is for a detector, the system aborts the job. You can monitor aborted detector SignalFlow programs using a built-in metric. Your organization also receives an event with information about the detector that aborted. Your job might reach this limit after it starts. A chart might initially load, but fail when its SignalFlow job aborts.

Maximum number of derived MTS per SignalFlow program

• Default limit value: 500,000

• Notes: Maximum number of derived MTS per SignalFlow program, where derived MTS are temporary MTS that a SignalFlow function or method has to maintain in memory. For example, if there are 20,000 MTS for the metric jvm.load, and each MTS comes from a unique host, then "data('jvm.load').sum(by=['host']).publish()" tracks 40,000 derived MTS. The data() SignalFlow function or method uses 20,000, and the sum() uses another 20,000. The number of input MTS is still 20,000.

• Customer impact: If the SignalFlow program is for a chart, the chart doesn’t load and you receive an error message. If the SignalFlow program is for a detector, the system aborts the program. You can monitor aborted detector SignalFlow programs using a built-in metric. Your organization also receives an event with information about the detector that aborted. Your SignalFlow program might reach this limit after it starts. A chart might initially load, but fail when its SignalFlow program aborts.

Maximum number of MTS allowed per chart data() function

• Default limit value:

  • 10,000 for standard subscriptions

  • 30,000 for enterprise subscriptions

• Notes: If you’re using Enterprise Edition, this limit is 30,000. You can have the limit set higher depending on your subscription. To have your limit changed, contact sales or customer support.

• Customer impact: If you exceed the limit, the system only keeps the most recently created MTS, based on the MTS creation timestamps. This might result in inaccurate computations.

Note

For a chart that is unavailable for autosharding, this limit is 10,000. A chart becomes unavailable for autosharding when:

• It has been manually sharded using the partition_filter() function.

• It uses one of the following functions: percentile(), mean_plus_stddev(), median(), stddev(), variance(), sample_stddev(), sample_variance(), ewma(), double_ewma(), kpss(), union().

Maximum number of MTS per detector data() function

• Default limit value:

  • 10,000 for standard subscriptions

  • 30,000 for enterprise subscriptions

• Notes: If you’re using Enterprise Edition, this limit is 30,000. You can have the limit set higher depending on your subscription. To have the limit changed, contact sales or customer support.

• Customer impact: If you exceed the limit, the system only keeps the most recently created MTS, based on the MTS creation timestamps. Detectors might not trigger, or they might trigger incorrectly.

Note

For a detector that is unavailable for autosharding, this limit is 10,000. A detector becomes unavailable for autosharding when:

• It has been manually sharded using the partition_filter() function.

• It uses one of the following functions: percentile(), mean_plus_stddev(), median(), stddev(), variance(), sample_stddev(), sample_variance(), ewma(), double_ewma(), kpss(), union().

Maximum number of active alerts per detector

• Default limit value: 200,000

• Notes: Maximum number of active alerts you can have for a detector.

• Customer impact: Once you reach this limit, Splunk Infrastructure Monitoring aborts the detector and deletes all active alerts. To avoid hitting this limit, configure autoclear on your detectors to clear active alerts based on defined thresholds. To learn more, see Auto-clear alerts.

Note

When you update or delete a detector, Splunk Observability Cloud stops the SignalFlow program associated with the detector and sends a stop notification to all the recipients currently configured for the detector. If the detector has a large number of recipients or a large number of alerts, sending the notification causes a flood of notifications. Your first reaction might be to delete the detector, but that might cause additional problems.

If your detector has a large number of recipients or a large number of alerts, do the following:

• To update the detector, first mute its alerts. You can unmute them when you’re finished editing. To learn more about muting alerts, see Mute specific detectors or alerts.

• To delete the detector, first delete all its recipients. To learn more about deleting recipients, see Remove notification recipients from a detector.

Maximum alert rate per detector

• Default limit value:

  • 10,000 alerts/minute for a detector with resolution smaller or equal to 1 minute

  • 20,000 or (job resolution/1m)*10,000)) for a detector with resolution larger than 1 minute, whichever is smaller

• Notes: Maximum alert rate limits the maximun amount of alerts a detector can fire within the job resolution.

• Customer impact: When the detector exceeds this limit, it’s aborted. For example:

  • If a detector runs at a 30-second resolution, it can fire at most 10,000 alerts within a minute.

  • If a detector runs at a 2-minute resolution, it can fire at most 20,000 alerts within 2 minutes.

  • If a detector runs at 5-minute resolution, it can fire at most 20,000 alerts within 5 minutes.

Maximum number of allocated data points per SignalFlow program

• Default limit value: 60,000,000

• Notes: Total number of data points a SignalFlow program must buffer to satisfy time window transformations. This is at least the number of input MTS, but if the SignalFlow has a time window calculation, the actual value might be much more. For example, a sum over 1m at 1s resolution requires 60 data points per MTS. If the SignalFlow has 10,000 MTS and only one window transform, the SignalFlow needs 10,000*60=600,000 data points.

• Customer impact: If the SignalFlow program is for a chart, the chart doesn’t load and you receive an error message. If the SignalFlow program is for a detector, the system aborts the SignalFlow program. You can monitor aborted detector SignalFlow programs using a built-in metric. Your organization also receives an event with information about the detector that aborted. Your SignalFlow program might reach this limit after it starts. A chart might initially load, but fail when its SignalFlow exceeds reaches the limit.

Maximum number of functions and methods per SignalFlow program

• Default limit value: 1,000

• Notes: The SignalFlow program "A = data().sum(by="az").sum().publish()" has 4 functions and methods (data, sum, sum, publish).

• Customer impact: SignalFlow programs which are violating the limit can’t start. You immediately get an error message.

Maximum number of queries per SignalFlow program

• Default limit value: 200

• Notes: Maximum number of queries you can have in a SignalFlow program used in a chart or detector. Queries that count toward this limit include data(), graphite(), events(), and alerts(). Using a timeshift() function on a stream causes all the queries for that stream to run again and increases the total number of queries in the program. For example, in the following program, queries A and B run again to retrieve data for D.

A = data('jvm.a').publish('A')
B = data('jvm.b').publish('B')
C = data('jvm.c').publish('C')
D = union(A, B).timeshift('1h').publish('D')

• Customer impact: SignalFlow programs which violate the limit can’t start. You immediately get an error message. This limit puts limit on how many detect() calls you can use if you use different data() or graphite() calls in the detect().

Maximum SignalFlow program stack size

• Default limit value: 64

• Notes: A SignalFlow function can’t recursively call itself more than this limit.

Maximum number of MTS analyzed across all SignalFlow programs

• Default limit value: The larger of 10,000,000 AMTS or 20% of your total AMTS.

• Notes: Maximum number of MTS that can concurrently use SignalFlow programs in your organization, including detector chart SignalFlow. For example, suppose you open 10 charts and keep them open. If each chart uses on average 5,000 MTS, you’re using 50,000 MTS, even if each chart looks at the same 5,000 MTS. If you close the charts, your usage goes to zero. Detector SignalFlow programs are always running, so they always use a portion of your MTS usage limit. This limit only applies to streaming SignalFlow programs, not ones that look at historical data.

• Customer impact: If the SignalFlow program is for a chart, the chart doesn’t load and you receive an error message. If the SignalFlow program is for a detector, the system aborts the program. You can monitor aborted detector SignalFlow programs using a built-in metric. Your organization also receives an event with information about the detector that aborted. Your SignalFlow program might reach this limit after it starts. A chart might initially load, but fail when the program reaches the limit.

Maximum max delay setting for SignalFlow programs

• Default limit value: 15 min

• Notes: The maximum allowed max delay value that you can set for a SignalFlow program. Higher values aren’t allowed, because they cause SignalFlow programs to use too much memory when data is slow to arrive.

• Customer impact: SignalFlow programs which are violating the limit can’t start. You immediately get an error message.

Maximum min delay setting for SignalFlow programs

• Default limit value: 15 min

• Notes: The maximum allowed min delay value that you can set for a SignalFlow program. Higher values aren’t allowed, because they cause SignalFlow programs to use too much memory when data is slow to arrive.

• Customer impact: SignalFlow programs which are violating the limit can’t start. You immediately get an error message.

Maximum number of wildcards per filter() function

• Default limit value: 35

• Notes: "data('jvm.load', filter=filter('host', 'kafka*east'))" counts as 1 wildcard filter

• Customer impact: SignalFlow programs which are violating the limit can’t start. You immediately get an error message.

Maximum number of prefix wildcards per filter() function

• Default limit value: 150

• Notes: "data('jvm.load', filter=filter('host', 'kafka*'))" counts as 1 prefix filter

• Customer impact: SignalFlow programs which are violating the limit can’t start. You immediately get an error message.

Maximum SignalFlow program text size

• Default limit value: 50,000

• Notes: Maximum character length of a SignalFlow program allowed in charts and detectors.

• Customer impact: You can’t save a SignalFlow program that exceeds the limit; instead, an error message appears.

Maximum SignalFlow programs per minute

• Default limit value: 1,000 SignalFlow programs per minute

• Notes: Maximum number of SignalFlow programs started per minute. The following actions start SignalFlow programs:

  • Creating or updating charts

  • Creating or updating detectors

  • Running a SignalFlow job using the API

  • Opening an alert from the list displayed the Alerts UI page. This action displays an alert dialog box and runs a SignalFlow program that provides charts and information to the page.

  You don’t get a notification when Splunk Observability Cloud starts a SignalFlow program for an alert dialog box, but the program counts against your SignalFlow programs per minute limit.

• Related metrics:

  • sf.org.numComputationsStarted

  • sf.org.numComputationsThrottled

• Customer impact: SignalFlow programs which are violating the limit can’t start. You immediately get an error message.

Maximum number of query arguments in a filter() function

• Default limit value: 256

• Notes: Limit to the number of query arguments in a SignalFlow filter

• Customer impact: Maximum number of derived MTS per SignalFlow program, where derived MTS are temporary MTS that a SignalFlow function or method has to maintain in memory. For example, if there are 20,000 MTS for the metric jvm.load, and each MTS comes from a unique host , then "data('jvm.load').sum(by=['host']).publish()" tracks 40,000 derived MTS. The data() function uses 20,000, and the sum() uses another 20,000. The number of input MTS is still 20,000.

Maximum number of detectors per organization

• Default limit value: 1,000

• Notes: The maximum number of detectors that you can create in a single organization.

• Related metrics:

  • sf.org.limit.detector

  • sf.org.num.detector

• Customer impact: The user interface displays an error reporting that you’ve exceeded the limit.

Maximum number of SignalFlow jobs per organization

• Default limit value: 5,000 per minute

• Notes: The maximum number of SignalFlow jobs you can run for your organization. Each token in the organization shares the same limit. For example, you can run 5,000 jobs per minute with one token, but you can’t run more jobs with any other token in the same organization.

• Related metrics:

  • sf.org.numComputationsStarted

  • sf.org.numComputationsThrottled

  • sf.org.numComputationsStartedByToken

  • sf.org.numComputationsThrottledByToken

• Customer impact: You reach this limit when the total number of jobs across all tokens for an organization exceeds 5,000 per minute. A single token, or a combination of different tokens in an organization, can use up the capacity.

  To check whether a single token hits the limit, use the related metrics. For example, if you see that the sf.org.numComputationsThrottledByToken metric increases for one token, but the sf.org.numComputationsThrottled metric doesn’t increase for the organization, then only a single token has used up the capacity.

Maximum number of SignalFlow jobs per websocket connection

• Default limit value: 300

• Notes: The maximum number of SignalFlow jobs you can run for each of your websocket connection.

• Customer impact: When you reach this limit, you get an error message saying “Too many channels in this connection”.

  You might reach this limit when you have too many charts open on the same page. For example, you open a dashboard with more than 300 charts. In this case, the charts outside the 300 limit don’t display. To avoid hitting this limit, you can reduce the number of charts by putting them into another dashboard or removing them.

New dimension or property key name limit

• Default limit value: 40 per week

• Notes: The maximum number of new custom fields (property or dimension keys) you can create, per organization per week. This limit applies to MTS and ETS. For example, host: 1 and host: 2 have the same key, which is host. hostname: host1 and hosttype: QA have different keys, which are hostname and hosttype.

• Related metrics:

  • sf.org.numPropertyLimitedMetricTimeSeriesCreateCalls

  • sf.org.numPropertyLimitedMetricTimeSeriesCreateCallsByToken

• Customer impact: The system rejects MTS creations that exceed the limit are rejected, and no error message appears.

Events per minute

• Default limit value: Determined by your subscription

• Notes: Maximum number of custom events you’re allowed to ingest per minute

• Customer impact: If you have this limit set for an org token, you receive a HTTP 429 error from Data Ingestion APIs when you exceed the limit.

MTS creations per minute limit

• Default limit value: 6,000 or determined by your subscription.

• Notes: Maximum number of MTS you can create per minute.

• Related metrics:

  • sf.org.numMetricTimeSeriesCreated

  • sf.org.limit.metricTimeSeriesCreatedPerMinute

• Customer impact: Infrastructure Monitoring drops new MTS that exceed the limit without returning an error, but accepts data points for existing MTS.

MTS creations per hour limit

• Default limit value: 60 times your MTS per minute limit

• Notes: Maximum number of MTS you can create per hour.

• Customer impact: Infrastructure Monitoring drops new MTS that exceed the limit without returning an error, but accepts data points for existing MTS.

MTS creations bursting per minute limit

• Default limit value: 10 times your MTS per minute limit, with a maximum of 20 minutes worth of bursting capacity per hour.

• Notes: Splunk Observability Cloud allows bursting for metric time series (MTS) creation to better support bursty or spiky patterns in MTS creation traffic. This limit is the maximum bursting capacity for MTS creations. Bursting is not guaranteed and is available only when there is enough additional capacity.

• Customer impact: The default MTS creations per minute limit is enforced once you have used up the 20 minutes maximum bursting capacity per hour.

  For example, your default MTS creations per minute limit is 6,000. You can max out the MTS creations burst limit in the following ways:

  1. Create 60,000 MTS per minute for consecutive or nonconsecutive 20 minutes, then go back to creating 6,000 MTS per minute for the rest of the hour.

  2. Spread the bursting capacity utilization over the entire hour by creating MTS at the rate of less than or equal to 24,000 per minute.

Number of dimensions per MTS

• Default limit value: 36

• Notes: Maximum number of dimensions per MTS. Infrastructure Monitoring drops invalid data points without returning an error, but keeps valid data points in the same request.

• Customer impact: Infrastructure Monitoring accepts valid data points but drops invalid data points. For invalid data points, Infrastructure Monitoring doesn’t send an error message.

Dimension/Metric value length

• Default limit value: UTF-8 strings with a maximum length of 256 characters (1024 bytes).

• Notes: Maximum length of a metric value or dimension value

• Customer impact: While ingesting data, the system drops data points with invalid dimension or metric values and doesn’t return an error. Ingest continues for valid data points.

Maximum dimension name length

• Default limit value: 128 characters (512 bytes)

• Notes: Maximum length of a dimension name

• Customer impact: While ingesting data, the system drops data points with invalid dimension names and doesn’t return an error. Ingest continues for valid data points.

Maximum number of API calls per minute

• Default limit value: 100,000

• Notes: Maximum number of REST API calls you can make per endpoint per minute. The limit for GET calls is 10 times the rate for other calls. The limit protects the system from gross misuse or attacks. This applies to metadata API to api.signalfx.com

• Related metrics: sf.org.numRestCalls

• Customer impact: The API returns an HTTP error code 429 that indicates that you’ve reached your API call limit.

Number of tags per dimension

• Default limit value: 50

• Notes: Maximum number of tags per dimension. Infrastructure Monitoring drops excess tags without returning an error.

• Customer impact: Infrastructure Monitoring drops tags that exceed the limit but doesn’t issue an error message.

Number of properties per dimension

• Default limit value: 75

• Notes: Maximum number of custom properties per dimension. Infrastructure Monitoring drops excess properties, but it doesn’t return an error.

• Customer impact: Infrastructure Monitoring drops properties that exceed the limit, but it doesn’t issue an error message.

timeserieswindow API data point limit

• Default limit value: 1,000,000

• Notes: The maximum number of data points you can retrieve in a single call to GET /v2/timeserieswindow.

• Customer impact: The request fails and returns an HTTP error code 400

Custom MTS entitlement

• Default limit value: Set by your contract entitlement

• Notes: Number of custom MTS entitled, as determined by your contract.

• Related metrics: sf.org.numCustomMetrics

• Customer impact: Splunk charges an overage of 1.5 times the normal price for usage that exceeds your contractual entitlement.

Custom MTS burst/overage limit

• Default limit value: Multiples of entitlement

• Notes: Maximum number of active custom MTS, within a moving window of the previous 60 minutes, that you’re allowed to have in your organization.

• Related metrics:

  • sf.org.numCustomMetrics

  • sf.org.limit.customMetricTimeSeries

• Customer impact: If you exceed this limit, Infrastructure Monitoring stops accepting data points for new custom MTS, but it continues to accept data points for custom MTS that already existed.

Host entitlement

• Default limit value: Contract entitlement

• Notes: Number of hosts in your contract, if applicable.

• Related metrics: sf.org.numResourcesMonitored

• Customer impact: Splunk charges an overage of 1.5 times the normal price for usage that exceeds your contractual entitlement.

Host burst/overage limit

• Default limit value: Multiples of entitlement

• Notes: For host-based pricing contracts, the maximum number of hosts that can send data to your organization. This limit is higher than your contractual limit to allow for burst and overage usage.

• Related metrics:

  • sf.org.numResourcesMonitored (filter for the dimension resourceType:hosts)

  • sf.org.limit.hosts

• Customer impact: If you exceed this limit, Infrastructure Monitoring drops data points from new hosts but keeps accepting data points for existing hosts.

Container entitlement

• Default limit value: Set by your contract entitlement

• Notes: Number of containers in your contract, if applicable

• Related metrics: sf.org.numResourcesMonitored

• Customer impact: Splunk charges an overage of 1.5 times the normal price for usage above contractual entitlement.

Container burst/overage limit

• Default limit value: Multiples of entitlement

• Notes: For host-based pricing contracts, maximum number of containers that can send data to your organization. This limit is higher than your contractual limit to allow for burst and overage usage.

• Related metrics:

  • sf.org.numResourcesMonitored (filter for the dimension resourceType:containers)

  • sf.org.limit.containers

• Customer impact: If you exceed this limit, Infrastructure Monitoring drops data points from new containers but keeps accepting data points for existing containers.

Bundled MTS limit

• Default limit value: Determined by your subscription

• Notes: This limit is scaled to subscription, and is no longer a pricing measure.

IMTS Limit

• Default limit value: Determined by your subscription

• Notes: Maximum number of inactive MTS, as allowed by your contract.

• Related metrics: sf.org.numInactiveTimeSeries

• Customer impact: When you reach this limit, the system deletes the MTS with the longest period of inactivity.

AMTS limit

• Default limit value: Determined by your subscription

• Notes: Maximum number of active MTS in a 25 hour period. If you’re using Kubernetes, the period is 1 hour.

• Related metrics:

  • sf.org.numActiveTimeSeries

  • sf.org.limit.activeTimeSeries

• Customer impact: When you exceed this limit, Infrastructure Monitoring refuses new MTS without issuing an error message. Infrastructure Monitoring continues to ingest data points for existing MTS.

Data points per minute (DPM) limit

• Default limit value: Set by your contract entitlement

• Notes: Limit on the number of data points you can send to Infrastructure Monitoring per minute. If you exceed the limit, Infrastructure Monitoring stops creating new MTS and rejects the data points.

• Customer impact: Infrastructure Monitoring drops new data points and MTS above the limit without returning an error.

Burst DPM limit

• Default limit value: Multiples of entitlement

• Notes: Limit on the number of data points you can send to Infrastructure Monitoring each minute. If you have this limit set on the org token you use, the data ingest API returns HTTP response code 429 when you exceed the limit.

• Customer impact: If you have this limit set on the org token you use, you receive a HTTP 429 error from Data Ingestion APIs when you exceed the limit.

Maximum rendered MTS for line, histogram, or heatmap visualizations

• Default limit value: 1,000

• Notes: When a visualization exceeds the limit, the UI arbitrarily selects the MTS it renders.

Maximum rendered MTS for area or stacked column visualizations

• Default limit value: 500

• Notes: When a visualization exceeds the limit, the UI arbitrarily selects the MTS it renders.

Maximum rendered MTS for column chart visualizations

• Default limit value: 20

• Notes: When a visualization exceeds the limit, the UI arbitrarily selects the MTS it renders.

This page was last updated on Sep 24, 2024.

---

❮

Previous

System limits for Splunk Infrastructure Monitoring

Next

Splunk APM system limits

❯

---

• API docs
• Blog
• Training
• Free Trial