Docs » Splunk Infrastructure Monitoring reference » System limits for Splunk Infrastructure Monitoring

System limits for Splunk Infrastructure Monitoring 🔗

Splunk Infrastructure Monitoring has system limits that help ensure good performance, stability, and reliability. These limits also protect the Infrastructure Monitoring multi-tenant environment. Exceeding these limits might degrade your Infrastructure Monitoring experience.

To help you optimize your product experience, this topic describes the following:

  • The name and value of each system limit

  • If available, the organization metrics associated with the limit

  • The impact you observe when you exceed the limit.

The first section in this topic contains tables of limit names and values, organized by product area. Each entry in the table shows the name of the limit and its default value. The limit name is also a link to more information about the limit.

The first table in the section lists the most important limits to pay attention to as you scale your deployment.

Abbreviations 🔗

This documentation uses the following abbreviations:

  • MTS: Metric time series

  • AMTS: Active MTS

  • IMTS: Inactive MTS

  • ETS: Event time series

Limit summaries 🔗

The following tables summarize limits for these product areas:

Important limits 🔗

Limit name

Default limit value

AMTS limit

Determined by your subscription

Maximum number of detectors per organization

1,000

Maximum number of MTS allowed per chart data() function

  • 5,000 for standard subscriptions

  • 10,000 for enterprise subscriptions

Maximum number of MTS per detector data() function

  • 5,000 for standard subscriptions

  • 10,000 for enterprise subscriptions

MTS creations per minute limit

6,000 or determined by your subscription

Number of input MTS per job

250,000

Data ingestion limits 🔗

Limit name

Default limit value

New dimension or property key name limit

40 per week

Dimension/Metric value length

256

Events per minute

Determined by your subscription

Maximum dimension name length

128

MTS creations per hour limit

500,000 MTS per hour or 50 times your MTS per minute limit.

MTS creations per minute limit

6,000 or determined by your subscription

Number of dimensions per MTS

36

Subscription limits 🔗

Limit name

Default limit value

AMTS limit

Determined by your subscription

Burst DPM limit

Multiples of entitlement

Bundled MTS limit

500,000

Container burst/overage limit

Multiples of entitlement

Container entitlement

Set by your contract entitlement

DPM limit

Set by your contract entitlement

Custom MTS burst/overage limit

Multiples of entitlement

Custom MTS entitlement

Set by your contract entitlement

High resolution custom metrics burst/overage limit

Multiples of entitlement

High resolution custom metrics entitlement

Set by your contract entitlement

Host burst/overage limit

Multiples of entitlement

Host entitlement

Contract entitlement

IMTS Limit

Determined by your subscription

Other limits 🔗

Limit name

Default limit value

timeserieswindow API data point limit

1,000,000

Limit details 🔗

This section provides details about each Infrastructure Monitoring system limit.

Maximum number of dashboards you can retrieve 🔗

  • Default limit value: 20,000

  • Notes: Maximum number of dashboards you can retrieve per query using either the UI or the API. If you reach this limit, you receive an error.

  • Customer impact: When you exceed this limit, the user interface displays the error message “Unexpected error has occurred”. After you exceed the limit, the dashboards page stops displaying dashboards.

Number of input MTS per job 🔗

  • Default limit value: 250,000

  • Notes: Maximum number of input MTS per job. When you use the same MTS multiple times in a job, each use counts towards the maximum.

  • Customer impact: If the job is for a chart, the chart doesn’t load and you receive an error message. If the job is for a detector, the job is aborted. You can monitor aborted detector SignalFlow programs using a built-in metric. Your organization also receives an event with information about the detector that aborted. Your job might reach this limit after it starts. A chart might initially load, but fail when the limit is reached.

Maximum number of derived MTS per SignalFlow program 🔗

  • Default limit value: 500,000

  • Notes: Maximum number of derived MTS per SignalFlow program, where derived MTS are temporary MTS that a SignalFlow SignalFlow function or method has to maintain in memory. For example, if there are 20,000 MTS for the metric jvm.load, and each MTS comes from a unique host , then "data('jvm.load').sum(by=['host']).publish()" tracks 40,000 derived MTS. The data() SignalFlow function or method uses 20,000, and the sum() uses another 20,000. The number of input MTS is still 20,000.

  • Customer impact: If the SignalFlow program is for a chart, the chart doesn’t load and you receive an error message. If the SignalFlow program is for a detector, the SignalFlow program is aborted. You can monitor aborted detector SignalFlow programs using a built-in metric. Your organization also receives an event with information about the detector that aborted. Your SignalFlow program might reach this limit after it starts. A chart might initially load, but fail when the limit is reached.

Maximum number of MTS allowed per chart data() function 🔗

  • Default limit value:

    • 5,000 for standard subscriptions

    • 10,000 for enterprise subscriptions

  • Notes: If you’re using Enterprise Edition, this limit is 10,000. If you have auto-sharding enabled, you can have the limit set higher depending on your subscription. Please contact sales or customer support.

  • Customer impact: If you exceed the limit, only the most recently created MTS are kept, based on the creation timestamp maintained for each MTS. This might result in inaccurate computations.

Maximum number of MTS per detector data() function 🔗

  • Default limit value:

    • 5,000 for standard subscriptions

    • 10,000 for enterprise subscriptions

  • Notes: If you’re using Enterprise Edition, this limit is 10,000. If you have auto-sharding enabled, you can have the limit set higher depending on your subscription. Please contact sales or customer support.

  • Customer impact: If you exceed the limit, only the most recently created MTS are kept, based on the creation timestamp maintained for each MTS. Detectors might not trigger, or they might trigger incorrectly.

Maximum number of allocated data points per SignalFlow program 🔗

  • Default limit value: 60,000,000

  • Notes: Total number of data points a SignalFlow program must buffer to satisfy time window transformations. This is at least the number of input MTS, but if the SignalFlow has a time window calculation, the actual value might be much more. For example, a sum over 1m at 1s resolution requires 60 data points per MTS. If the SignalFlow has 10,000 MTS and only one window transform, the SignalFlow needs 10,000*60=600,000 data points.

  • Customer impact: If the SignalFlow program is for a chart, the chart doesn’t load and you receive an error message. If the SignalFlow program is for a detector, the SignalFlow program is aborted. You can monitor aborted detector SignalFlow programs using a built-in metric. Your organization also receives an event with information about the detector that aborted. Your SignalFlow program might reach this limit after it starts. A chart might initially load, but fail when the limit is reached.

Maximum number of functions and methods per SignalFlow program 🔗

  • Default limit value: 1,000

  • Notes: The SignalFlow program "A = data().sum(by="az").sum().publish()" has 4 functions and methods (data, sum, sum, publish).

  • Customer impact: SignalFlow programs which are violating the limit can’t start. You immediately get an error message.

Maximum number of queries per SignalFlow program 🔗

  • Default limit value: 200

  • Notes: Maximum number of queries you can have in a SignalFlow program used in a chart or detector. Queries that count toward this limit include data(), graphite(), newrelic(), events(), and alerts(). Using a timeshift() function on a stream causes all the queries for that stream to run again and increases the total number of queries in the program. For example, in the following program, queries A and B run again to retrieve data for D.

A = data('jvm.a').publish('A')
B = data('jvm.b').publish('B')
C = data('jvm.c').publish('C')
D = union(A, B).timeshift('1h').publish('D')
  • Customer impact: SignalFlow programs which violate the limit can’t start. You immediately get an error message. Note that this puts a limit on how many detect() you can use if you use different data(), graphite(), or newrelic() methods in the detect().

Maximum SignalFlow program stack size 🔗

  • Default limit value: 64

  • Notes: A SignalFlow function can’t recursively call itself more than this limit.

Maximum number of MTS analyzed across all SignalFlow programs 🔗

  • Default limit value: The larger of 10,000,000 AMTS or 20% of your total AMTS.

  • Notes: Maximum number of MTS that can concurrently use SignalFlow programs in your organization, including detector chart SignalFlow. For example, suppose you open 10 charts and keep them open. If each chart uses on average 5,000 MTS, you’re using 50,000 MTS, even if each chart looks at the same 5,000 MTS. If you close the charts, your usage goes to zero. Detector SignalFlow programs are always running, so they always use a portion of your MTS usage limit. This limit only applies to streaming SignalFlow programs, not ones that look at historical data.

  • Customer impact: If the SignalFlow program is for a chart, the chart doesn’t load and you receive an error message. If the SignalFlow program is for a detector, the SignalFlow program is aborted. You can monitor aborted detector SignalFlow programs using a built-in metric. Your organization also receives an event with information about the detector that aborted. Your SignalFlow program might reach this limit after it starts. A chart might initially load, but fail when the limit is reached.

Maximum max delay setting for SignalFlow programs 🔗

  • Default limit value: 15 min

  • Notes: The maximum allowed max delay value that you can set for a SignalFlow program. Higher values aren’t allowed, because they cause SignalFlow programs to use too much memory when data is slow to arrive.

  • Customer impact: SignalFlow programs which are violating the limit can’t start. You immediately get an error message.

Maximum min delay setting for SignalFlow programs 🔗

  • Default limit value: 15 min

  • Notes: The maximum allowed min delay value that you can set for a SignalFlow program. Higher values aren’t allowed, because they cause SignalFlow programs to use too much memory when data is slow to arrive.

  • Customer impact: SignalFlow programs which are violating the limit can’t start. You immediately get an error message.

Maximum number of wildcards per filter() function 🔗

  • Default limit value: 35

  • Notes: "data('jvm.load', filter=filter('host', 'kafka*east'))" counts as 1 wildcard filter

  • Customer impact: SignalFlow programs which are violating the limit can’t start. You immediately get an error message.

Maximum number of prefix wildcards per filter() function 🔗

  • Default limit value: 150

  • Notes: "data('jvm.load', filter=filter('host', 'kafka*'))" counts as 1 prefix filter

  • Customer impact: SignalFlow programs which are violating the limit can’t start. You immediately get an error message.

Maximum SignalFlow program text size 🔗

  • Default limit value: 50,000

  • Notes: Maximum character length of a SignalFlow program allowed in charts and detectors.

  • Customer impact: SignalFlow programs which are violating the limit can’t be saved. You immediately get an error message.

Maximum SignalFlow programs per minute 🔗

  • Default limit value: 1,000 SignalFlow programs per minute

  • Notes: Maximum number of SignalFlow programs started per minute. The following actions start SignalFlow programs:

    • Creating or updating charts

    • Creating or updating detectors

    • Running a SignalFlow job using the API

    • Opening an alert from the list displayed the Alerts UI page. This action displays an alert modal page and runs a SignalFlow program that provides charts and information to the page.

    You don’t get a notification when Observability Cloud starts a SignalFlow program for an alert modal page, but the program counts against your SignalFlow programs per minute limit.

  • Related metrics:

    • sf.org.computations.started

    • sf.org.computations.throttled

  • Customer impact: SignalFlow programs which are violating the limit can’t start. You immediately get an error message.

Maximum number of query arguments in a filter() function 🔗

  • Default limit value: 256

  • Notes: Limit to the number of query arguments in a SignalFlow filter

  • Customer impact: Maximum number of derived MTS per SignalFlow program, where derived MTS are temporary MTS that a SignalFlow function or method has to maintain in memory. For example, if there are 20,000 MTS for the metric jvm.load, and each MTS comes from a unique host , then "data('jvm.load').sum(by=['host']).publish()" tracks 40,000 derived MTS. The data() function uses 20,000, and the sum() uses another 20,000. The number of input MTS is still 20,000.

Maximum number of detectors per organization 🔗

  • Default limit value: 1,000

  • Notes: The maximum number of detectors that you can create in a single organization.

  • Related metrics:

    • sf.org.limit.detector

    • sf.org.num.detector

  • Customer impact: The user interface displays an error reporting that the limit has been exceeded.

New dimension or property key name limit 🔗

  • Default limit value: 40 per week

  • Notes: The maximum number of new custom fields (property or dimension keys) you can create, per organization per week. This limit applies to MTS and ETS. For example, host:1 and host:2 are considered to have 1 key which is host. foo:1 and bar:1 are considered to have two keys: foo and bar.

  • Related metrics:

    • sf.org.numPropertyLimitedMetricTimeSeriesCreateCalls

    • sf.org.numPropertyLimitedMetricTimeSeriesCreateCallsByToken

  • Customer impact: MTS creations above the limit are rejected, and no error message appears.

Events per minute 🔗

  • Default limit value: Determined by your subscription

  • Notes: Maximum number of custom events you’re allowed ingest per minute

  • Customer impact: If you have this limit set for an org token, you will receive a HTTP 429 error from Data Ingestion APIs when you exceed the limit.

MTS creations per minute limit 🔗

  • Default limit value: 6,000 or determined by your subscription

  • Notes: Maximum number of MTS you can create per minute

  • Related metrics:

    • sf.org.numMetricTimeSeriesCreated

    • sf.org.limit.metricTimeSeriesCreatedPerMinute

  • Customer impact: Infrastructure Monitoring drops new MTS above the limit without returning an error.

MTS creations per hour limit 🔗

  • Default limit value: 500,000 MTS per hour or 50 times your MTS per minute limit.

  • Notes: If you exceed your MTS limit per minute for more than 30 minutes of any hour, you can’t create any more MTS for the remainder of the hour.

  • Customer impact: Infrastructure Monitoring drops new MTS above the limit without returning an error. Data points for existing MTS are still accepted.

Number of dimensions per MTS 🔗

  • Default limit value: 36

  • Notes: Maximum number of dimensions per MTS. Infrastructure Monitoring silently drops invalid data points, but valid data points in the same request are kept.

  • Customer impact: Infrastructure Monitoring accepts valid data points but drops invalid data points. For invalid data points, Infrastructure Monitoring doesn’t send an error message.

Dimension/Metric value length 🔗

  • Default limit value: 256

  • Notes: No error is returned. Invalid data points are silently dropped. Valid data points in the same request are let in.

  • Customer impact: No error is returned. Invalid data points are silently dropped. Valid data points in the same request are let in.

Maximum dimension name length 🔗

  • Default limit value: 128

  • Notes: Maximum number of dimensions per MTS. Infrastructure Monitoring silently drops invalid data points, but valid data points in the same request are kept.

  • Customer impact: Infrastructure Monitoring accepts valid data points but drops invalid data points. For invalid data points, Infrastructure Monitoring doesn’t send an error message.

Maximum number of API calls per minute 🔗

  • Default limit value: 100,000

  • Notes: Maximum number of REST API calls you can make per endpoint per minute. The limit for GET calls is 10 times the rate for other calls. The limit protects the system from gross misuse or attacks. This applies to metadata API to api.signalfx.com

  • Related metrics: sf.org.numRestCalls

  • Customer impact: The API returns an HTTP error code 429 that indicates that you’ve reached your API call limit.

Number of tags per dimension 🔗

  • Default limit value: 50

  • Notes: Maximum number of tags per dimension. Infrastructure Monitoring silently drops excess tags.

  • Customer impact: Infrastructure Monitoring drops tags that exceed the limit but doesn’t issue an error message.

Number of properties per dimension 🔗

  • Default limit value: 75

  • Notes: Maximum number of custom properties per dimension. Infrastructure Monitoring silently drops excess properties.

  • Customer impact: Infrastructure Monitoring drops properties that exceed the limit but doesn’t issue an error message.

timeserieswindow API data point limit 🔗

  • Default limit value: 1,000,000

  • Notes: The maximum number of data points you can retrieve in a single call to GET /v2/timeserieswindow.

  • Customer impact: The request fails and returns an HTTP error code 400

Custom MTS entitlement 🔗

  • Default limit value: Set by your contract entitlement

  • Notes: Number of custom MTS entitled, as determined by your contract.

  • Related metrics: sf.org.numCustomMetrics

  • Customer impact: Splunk charges an overage of 1.5 times the normal price for usage above contractual entitlement.

Custom MTS burst/overage limit 🔗

  • Default limit value: Multiples of entitlement

  • Notes: Maximum number of active custom MTS, within a moving window of the previous 60 minutes, that you’re allowed to have in your organization.

  • Related metrics:

    • sf.org.numCustomMetrics

    • sf.org.limit.customMetricTimeSeries

  • Customer impact: If you exceed this limit, Infrastructure Monitoring stops accepting data points for new custom MTS, but it continues to accept data points for custom MTS that already existed.

Host entitlement 🔗

  • Default limit value: Contract entitlement

  • Notes: Number of hosts in your contract, if applicable.

  • Related metrics: sf.org.numResourcesMonitored

  • Customer impact: Splunk charges an overage of 1.5 times the normal price for usage above contractual entitlement.

Host burst/overage limit 🔗

  • Default limit value: Multiples of entitlement

  • Notes: For host-based pricing contracts, the maximum number of hosts that can send data to your organization. This limit is higher than your contractual limit to allow for burst and overage usage.

  • Related metrics:

    • sf.org.numResourcesMonitored (filter for the dimension resourceType:hosts)

    • sf.org.limit.hosts

  • Customer impact: If you exceed this limit, Infrastructure Monitoring drops data points from new hosts but keeps accepting data points for existing hosts.

Container entitlement 🔗

  • Default limit value: Set by your contract entitlement

  • Notes: Number of containers in your contract, if applicable

  • Related metrics: sf.org.numResourcesMonitored

  • Customer impact: Splunk charges an overage of 1.5 times the normal price for usage above contractual entitlement.

Container burst/overage limit 🔗

  • Default limit value: Multiples of entitlement

  • Notes: For host-based pricing contracts, maximum number of containers that can send data to your organization. This limit is higher than your contractual limit to allow for burst and overage usage.

  • Related metrics:

    • sf.org.numResourcesMonitored (filter for the dimension resourceType:containers)

    • sf.org.limit.containers

  • Customer impact: If you exceed this limit, Infrastructure Monitoring drops data points from new containers but keeps accepting data points for existing containers.

High resolution custom metrics entitlement 🔗

  • Default limit value: Set by your contract entitlement

  • Notes: Number of high resolution metrics allowed in your contract

  • Customer impact: Splunk charges an overage of 1.5 times the normal price for usage above contractual entitlement.

High resolution custom metrics burst/overage limit 🔗

  • Default limit value: Multiples of entitlement

  • Notes: This limit is to protect the SaaS platform. It’s typically a multiple of your contractual limit. For example, if you purchase 500 hosts, Infrastructure Monitoring might set limit to 800. The multiple decreases as your contractual limit increases.

  • Customer impact: MTS creations for high resolution metrics above the limit are rejected.

Bundled MTS limit 🔗

  • Default limit value: 500,000

  • Notes: This limit applies to host based contracts only. It is the total number of bundled MTS a customer could has, beyond the standard host based or container based MTS. MTS for SQS queues is an example.

  • Customer impact: MTS creations above the limit are rejected, and no error message appears.

IMTS Limit 🔗

  • Default limit value: Determined by your subscription

  • Notes: Maximum number of inactive MTS, as allowed by your contract.

  • Related metrics: sf.org.numInactiveTimeSeries

  • Customer impact: When you reach this limit, the system deletes the inactive MTS that have been inactive the longest.

AMTS limit 🔗

  • Default limit value: Determined by your subscription

  • Notes: Maximum number of active MTS in a 25 hour period. If you’re using Kubernetes, the period is 1 hour.

  • Related metrics:

    • sf.org.numActiveTimeSeries

    • sf.org.limit.activeTimeSeries

  • Customer impact: When you exceed this limit, Infrastructure Monitoring refuses new MTS without issuing an error message. Infrastructure Monitoring continues to ingest data points for existing MTS.

DPM limit 🔗

  • Default limit value: Set by your contract entitlement

  • Notes: Limit on the number of data points you can send to Infrastructure Monitoring per minute. If you exceed the limit, Infrastructure Monitoring stops creating new MTS and rejects the data points.

  • Customer impact: Infrastructure Monitoring drops new data points and MTS above the limit without returning an error.

Burst DPM limit 🔗

  • Default limit value: Multiples of entitlement

  • Notes: Limit on the number of data points you can send to Infrastructure Monitoring each minute. If you have this limit set on the organization token, the data ingest API returns HTTP response code 429 when you exceed the limit.

  • Customer impact: If you have this limit set for an org token, you will receive a HTTP 429 error from Data Ingestion APIs when you exceed the limit.

Maximum rendered MTS for line, histogram, or heatmap visualizations 🔗

  • Default limit value: 1,000

  • Notes: When a visualization exceeds the limit, the UI arbitrarily selects the MTS it renders.

Maximum rendered MTS for area or stacked column visualizations 🔗

  • Default limit value: 500

  • Notes: When a visualization exceeds the limit, the UI arbitrarily selects the MTS it renders.

Maximum rendered MTS for column chart visualizations 🔗

  • Default limit value: 20

  • Notes: When a visualization exceeds the limit, the UI arbitrarily selects the MTS it renders.