Splunk Cloud

Splunk Cloud Admin Manual

Acrobat logo Download manual as PDF

Acrobat logo Download topic as PDF

Introduction to Getting Data In

Splunk Cloud administrators can add data to their Splunk Cloud deployment using a variety of methods. This topic provides an overview of those methods.

Fundamental Splunk and Splunk Cloud concepts

Before attempting to get data into your Splunk Cloud deployment, you should have a solid understanding of certain Splunk and Splunk Cloud concepts. The table lists these concepts. You should also review the Splunk Cloud information in the Getting Data In manual.

Concept Description
deployment server A deployment server is a Splunk Enterprise instance that acts as a centralized configuration manager for any number of forwarders, called "deployment clients". The deployment server is hosted on your premises or your Cloud environment (such as AWS or Azure). For a more detailed description of the components of a deployment server, see Deployment Server Architecture.
indexes The index is the repository for your data. When the Splunk platform indexes raw data, it transforms the data into searchable events. For more information about indexes, see Manage Splunk Cloud Indexes.
Inputs Data Manager The Inputs Data Manager (IDM) is a component of your Splunk Cloud environment optimized for data ingestion. It is intended for use with cloud data sources or when using add-ons that require inputs on the Search tier.

Customers on the Splunk Cloud Victoria Experience don't need to use an IDM. For more information, see Determine your Splunk Cloud Experience.

search head

search head cluster

For more information, see search head and search head cluster in the Splexicon.
source types A source type is one of the critical default fields that Splunk software assigns to all incoming data. It tells Splunk software what kind of data you have, so that it can format the data intelligently during indexing. For more information, see Why source types matter.
Splunk applications and add-ons A Splunk app is an application that runs on the Splunk platform and typically addresses several use cases. Add-ons support and extend the functionality of the Splunk platform and the apps that run on it, usually by providing inputs for a specific technology or vendor. For more information about add-ons, see About Splunk add-ons.
universal forwarder To forward data to Splunk Cloud, you typically use the Splunk universal forwarder. The universal forwarder is a dedicated, streamlined version of Splunk Enterprise that contains only the essential components needed to forward data. The universal forwarder does not support Python and does not expose a UI. In most situations, the universal forwarder is the best way to forward data to indexers. Its main limitation is that it forwards unparsed data, except in certain cases, such as structured data. For more information, see Work with forwarders.

Types of data that Splunk Cloud accepts

Splunk Cloud accepts a wide variety of data, and can also monitor relational databases and third-party infrastructures. For more information, see the following sections in the Getting Data In manual:

Tools to get data into Splunk Cloud

This section is designed to help you make decisions about the best way to get data into your Splunk Cloud instance. There are a few different ways to get data into Splunk Cloud: forwarders, HTTP Event Collector (HEC), apps and add-ons, or the Inputs Data Manager (IDM). The best way to get data in depends on the source of the data and what you intend to do with it.

Work with forwarders

Usually, to get data from your customer site to Splunk Cloud, you use a forwarder.

A forwarder is a version of Splunk Enterprise optimized to send data. A universal forwarder is a purpose-built data collection mechanism with very minimal resource requirements, whereas a heavy forwarder is full Splunk Enterprise deployment configured to act as a forwarder with indexing disabled.

Splunk forwarders send data from a datasource to your Splunk Cloud deployment for indexing, which makes the data searchable. Forwarders are lightweight processes, so they can usually run on the machines where the data originates. For more information, see the following topics:

Work with HTTP Event Collector

The HTTP Event Collector (HEC) uses a token-based authentication model so you can securely send data and application events to a Splunk deployment over the HTTP and Secure HTTP (HTTPS) protocols.

For more information, see the following sections in Set up and use HTTP Event Collector in Splunk Web in the Getting Data In manual:

Work with Apps and Add-ons

Splunk apps and add-ons extend the capability and simplify the process of getting data into your Splunk platform deployment.

For more information, see Use apps and add-ons to get data in in the Getting Data In manual.

Splunk Cloud considerations

Apps and add-ons that contain a data collection component should be installed on forwarders or IDMs and on your Splunk Cloud instance for their data collection functions (modular or scripted inputs). For more information, see Work with forwarders and Work with Inputs Data Manager (IDM).

The following graphic shows a common topology with add-ons installed on forwarders and on the Splunk Cloud instance to extend the functionality for getting data in.

This graphic shows an example of add-ons installed on multiple forwarders.

Work with Inputs Data Manager

The Inputs Data Manager (IDM) is a hosted solution for Splunk Cloud to support scripted and modular inputs and cloud-based inputs that you want to send directly to Splunk Cloud. In a majority of cases, an IDM eliminates the need for customer-managed infrastructure.

Customers on the Splunk Cloud Victoria Experience don't need to use an IDM. For more information, see Determine your Splunk Cloud Experience.

As a best practice, use an IDM in the following cases:

  • You have scripted or modular inputs that you want to send to Splunk Cloud. For example, you can poll a cloud-based database, web service, or API for specific data and process the results.
  • You have cloud-based inputs such as Microsoft Azure or AWS that you want to send directly to Splunk Cloud without the intermediary step of sending data to an on-premise forwarder. You can send these inputs directly to an IDM rather than routing them through a forwarder to get the data into Splunk Cloud.

The following graphic shows the typical architecture of IDM. Note that the search tier and index tier are not hosted on the IDM. The IDM is not intended to store data or perform searches.

This graphic shows an example of add-ons installed on multiple forwarders.

IDM is not supported on the Splunk Cloud Free Trial.

Ports opened for IDM

The following port access applies to inbound and outbound IDM ports:

  • Inbound access to ports 443 and 8089 are controlled by an access list. Contact Support if you need to modify the access list.
  • Outbound access to port 443 is open by default. Contact Support if you need to open additional outbound ports.

When you contact Support, provide a list of public IP addresses and subnets with this request. For example, you might want to open port 8089, the port for the REST API. Note that opening a specific outbound port opens the same port for all tiers in your Splunk Cloud environment.

Apps supported on IDM

If the app contains modular inputs and is Splunk Cloud certified, it is compatible with Splunk Cloud IDM. Generally, apps that are cloud-based are well-suited to IDM. Many cloud-based apps are supported.

To verify if your app is supported on IDM, check Splunkbase.

Limitations when working with IDM

The IDM is intended to function specifically as a forwarder for modular and scripted inputs, or to obviate the need to route cloud-based inputs through an on-premise forwarder. The following functions are not intended to be performed on the IDM:

  • Search capabilities are capped for users on IDM. The IDM is not intended to function as a search head.
  • IDM does not currently support Self-Service App Installations. To get modular and scripted input onto the IDM, you need to create a private app and request that Support upload it.
  • If an add-on is tightly integrated with an Enterprise Security search head, do not use IDM.
  • HEC inputs are not supported with IDM.
  • IDM isn't a syslog sink, nor can it receive unencrypted TCP streams.
  • IDM isn't a one-to-one replacement for a heavy forwarder. You must still use a heavy forwarder if you need to perform parsing or activities other than standard scripted and modular data inputs.

Use IDM with scripted and modular inputs

To use scripted or modular inputs, you must package them in a private app. To do this, complete the following high-level steps:

  1. Create your modular or scripted inputs. For instructions on creating these inputs, see Get data from APIs and other remote data interfaces through scripted inputs in the Getting Data In manual.
  2. Package the script or modular input in a private app. For instructions on building a private app for Splunk Cloud, see Overview of developing a private Splunk Cloud app.
  3. Submit the private app for Splunk Cloud vetting.
  4. Request that Support upload the app to your IDM.

Use IDM with cloud-based add-ons

When you work with IDM and Cloud-based add-ons, complete the following high-level steps to get data in:

  1. Create a support request to install the Add-on.
  2. Configure an index on your Splunk Cloud instance. This index is going to be associated with your cloud input.
  3. Perform any configurations needed on the cloud-based source that enables you to get data in.
  4. Configure the Splunk Add-on on your Inputs Data Manager (IDM).
  5. You will also need to configure inputs on the IDM. The IDM is responsible for data ingestion.
  6. Verify that data is flowing to your Splunk Cloud environment.

As a best practice, install cloud-based add-ons on an IDM, and install on-premises-based add-ons on a universal forwarder or heavy forwarder.

See also

For more information about See
Getting AWS data in using IDM Get Amazon Web Services (AWS) data into Splunk Cloud
Getting Microsoft Azure data in using IDM Get Microsoft Azure data into Splunk Cloud
Last modified on 23 July, 2021
Determine your Splunk Cloud Experience
Get Amazon Web Services (AWS) data into Splunk Cloud

This documentation applies to the following versions of Splunk Cloud: 8.0.2006, 8.0.2007, 8.1.2008, 8.1.2009, 8.1.2011, 8.1.2012 (latest FedRAMP release), 8.1.2101, 8.1.2103, 8.2.2104, 8.2.2105

Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters