Assign the correct source types to your data

The source type is one of the default fields that the Splunk platform assigns to all incoming data, and determines how the Splunk platform formats the data during indexing. By assigning the correct source type to your data, the indexed version of the data appears the way you want it to with correct timestamps and event breaks.

You can confirm that the Splunk platform indexes your data as you want it to appear using the Set Source Type page in Splunk Web.

Assigning source types to your data

comes with many predefined source types and attempts to assign the correct source type to your data based on its format. In some cases, you might need to manually select a different predefined source type to the data. In other cases, you might need to create a new source type with customized event processing settings.

On the Set Source Type page, you can see how will index the data based on the application of a predefined source type. You can modify the settings interactively and save those modifications as a new source type.

Ensure that you're assigning the right source type to your data by following these steps on the Set Source Type page:

1. See what your data will look like without any changes using the default event-processing configuration.
2. Apply a different source type to see whether it offers more preferable results.
3. Modify settings for timestamps and event breaks to improve the quality of the indexed data and save the modifications as a new source type.
4. Create a new source type.

If you use Splunk Enterprise, you can save any new source types to a props.conf configuration file that you can later distribute across the indexers in your deployment so that the source types are available globally. See Distribute source type configurations in Splunk Enterprise.

Some source types, such as those in the Log to Metrics category, cannot be previewed. See "About the Log to Metrics source type category" later in this topic for details.

For information on source types and why they are so important, see Why source types matter.

About the Log to Metrics source type category

Source types in the Log to Metrics category are special source types. The Splunk platform uses these source types for the ingest-time conversion of log events to metric data points. If you select a source type from this category, a set of Metrics controls will appear on the left side of the Set Source Type page. For more information about log-to-metrics conversion and the Metrics settings, see Set up ingest-time log-to-metrics conversion in Splunk Web in the Metrics manual.

When you apply a Log to Metrics source type to an input, you can't preview the data for that input.

Assign source types to your data

When the Set Source Type page opens, chooses a source type based on the data you specified. You can accept that source type or change it by following these steps.

1. Check the preview pane to see how will index the data. Review event breaks and timestamps.
2. (Optional) View the event summary by clicking View event summary.
   Splunk Web displays the event summary in a new window. See View event summary.
3. If the data appears the way that you want, then click Next to proceed to the Inputs Settings page. Otherwise, choose from one of the following options:
   
   • Choose an existing source type to change the data formatting. See Choose an existing source type.
   • Adjust timestamps, delimiters, and line breaking manually, then save the changes as a new source type. See Adjust timestamps and event line breaks.

Choose an existing source type

If the data does not appear in the way that you want, see whether or not an existing source type fixes the problem.

If the Splunk platform can detect a source type, it displays the source type in the Source type: <sourcetype> drop-down list. If it can't determine a source type, it displays Sourcetype: System Defaults.

1. Click the Source type: <sourcetype> drop-down list to see a list of source type categories. Each category contains a list of source types within that category.
2. Hover over the category that best represents your data.
   As you do, the source types under that category appear in a drop-down list.
3. Select the source type that best represents your data.
   Splunk Web updates the data preview pane to show how the data looks under the new source type. You might need to scroll to see all source types in a category.
4. Review your data again in the preview pane. If the existing source types don't work for your data, you must manually adjust the timestamps, delimiters, and event breaking. See Adjust timestamps and event line breaks.
5. If you're satisfied with the results, click Next to proceed to the Inputs Settings page.

View event summary

You can see a summary of the events within the data sample by clicking View Event Summary. This summary shows the following information:

• The size of the sample data, in bytes.
• The number of events that were present in the sample.
• The chart that represents the distribution of the events over time. the Splunk platform uses date stamps within the file to determine how to display this chart.
• A breakdown of the number of lines each event in the sample took up.

Adjust timestamps and event breaks

If you choose an existing source type without success, then you can manually adjust how processes timestamps and event line breaks for the incoming data.

To manually adjust timestamp and event line breaking parameters, use the Event Breaks, Timestamp, Delimited Settings, and Advanced drop-down lists on the left pane of the Set Source Type page. The preview pane updates as you make changes to the settings.

The Event breaks tab appears only when the Splunk platform can't determine how to line-break the file, or if you select a source type that doesn't define line breaking. The Delimited settings tab appears only when the Splunk platform detects that you want to import a structured data file, or you select a source type for structured data such as CSV.

If you need more information about how to adjust timestamps and event breaks, see Modify event processing.

To manually adjust timestamps and event breaks, follow these steps:

1. Click Event breaks. The list displays the Break type options, which controls how the Splunk platform line-breaks the file into events. You can choose from the following options:
   
   • Auto: Detect event breaks based on the location of the time stamp.
   • By Line: Breaks every line into a single event.
   • Regex…: Uses the specified regular expression to determine line breaking.
2. Click Timestamps. The list expands to show extraction options. Select from one of the following options:
   
   • Auto: Extract timestamps automatically by looking in the file for timestamp events.
   • Current time: Apply the current time to all events detected.
   • Advanced: Specify the time zone, timestamp format in strptime(), and any fields that comprise the timestamp. For more information about strptime(), see Date and Time functions in the Search Reference.
3. Click Delimited settings to display delimiting options.
   

   Field	Description
   Field delimiter	The delimiting character for structured data files, such as comma-separated value (CSV) files.
   Quote character	The character that uses to determine when something is in quotes.
   File preamble	A regular expression that tells to ignore one or more preamble lines, or lines that don't contain any actual data, in the structured data file.
   Field names	Determines field names automatically, based on line number, based on a comma-separated list, or through a regular expression.

4. After the results look the way you want, save your changes as a new source type, which you can then apply to the data when it is indexed.
5. If you want to make configuration changes to props.conf, click the Advanced tab to display fields that let you enter attribute/value pairs that get committed directly to the props.conf configuration file. See Make configuration changes in the Advanced tab for more instructions.
   

   Making configuration changes in the Advanced tab requires advanced knowledge of Splunk software features. Changes made here might negatively affect the indexing of your data. Consider consulting Splunk Professional Services before configuring these options.

6. If you're satisfied with the results, click Next to proceed to the Inputs Settings page.

Make configuration changes in the Advanced tab

Making configuration changes in the Advanced tab requires advanced knowledge of Splunk software features. Changes made here might negatively affect the indexing of your data. Consider consulting Splunk Professional Services before configuring these options.

1. Click a field to edit props.conf entries that generates based on your previous choices.
2. Click the X to the right of an attribute/value field pair to delete that pair.
3. Click New setting to create a new attribute/value field pair and specify a valid attribute and value for props.conf.
4. Click Apply settings to commit the changes to the props.conf file.

Next step

Once you assign the correct source types to your data, see Modify input settings.