Splunk Cloud

Getting Data In

Download manual as PDF

Download topic as PDF

Troubleshoot HTTP Event Collector

Logging

HTTP Event Collector saves usage data about itself to log files. You can query these usage metrics using Splunk Cloud to explore usage trends system-wide, per token, per sourcetype, and more, as well as to evaluate HTTP Event Collector performance. Metrics are logged whenever HTTP Event Collector is enabled. HTTP Event Collector is disabled by default, so it will not log data until you enable it.

You can also view HTTP Event Collector error logs in splunkd.log. See enable debug logging in the Splunk Enterprise Troubleshooting Manual for how to enable debugging on your Splunk instance.

Log file location and management

HTTP Event Collector metrics are written to the http_event_collector_metrics.log file located at the following path:

$SPLUNK_HOME/var/log/introspection/splunk/

A new http_event_collector_metrics.log file is created when you start your Splunk Cloud instance. Any existing file with that name is renamed by giving it next higher available numeric extension. For example, if you restart Splunk Cloud and there exists http_event_collector_metrics.log, http_event_collector_metrics.1, and http_event_collector_metrics.2 files, the http_event_collector_metrics.log file will be renamed http_event_collector_metrics.3 and HTTP Event Collector will begin logging to a new http_event_collector_metrics.log file.

You configure the logging frequency of HTTP Event Collector metrics in the limits.conf file. 60 seconds is the default frequency. HTTP Event Collector continues logging system-level metrics even when there is no data input activity. When there is no activity, you can expect about 200 kilobytes (KB) of metrics log data to be produced every 24 hours. The maximum size of a metrics log file is 25 megabytes (MB). If a log file reaches that limit, the log file is renamed as described in the previous paragraph and a new one is created. Up to five metrics log files can be stored at a time. The props.conf file defines parameters for reading and indexing the metrics log file.

Querying HTTP Event Collector metrics data

HTTP Event Collector metrics data is indexed to the "_introspection" index. To query the accumulated HTTP Event Collector metrics using Splunk Cloud, use the following command:

index="_introspection" token

Metrics log data format

HTTP Event Collector metrics data is recorded to the log in JSON format. This means that the log is both easily human-readable and consistent with other Splunk Cloud log formats. A single entry consists of both input summary metrics (series = http_event_collector) and per-token metrics (series = http_event_collector_token), as shown in the following example:

{  
   "datetime":"09-01-2016 19:21:19.014 -0700",
   "log_level":"INFO",
   "component":"HttpEventCollector",
   "data":{  
      "series":"http_event_collector",
      "transport":"http",
      "format":"json",
      "total_bytes_received":0,
      "total_bytes_indexed":0,
      "num_of_requests":0,
      "num_of_events":0,
      "num_of_errors":0,
      "num_of_parser_errors":0,
      "num_of_auth_failures":0,
      "num_of_requests_to_disabled_token":0,
      "num_of_requests_to_incorrect_url":0,
      "num_of_requests_in_mint_format":0,
      "num_of_ack_requests":0,
      "num_of_requests_acked":0,
      "num_of_requests_waiting_ack":0
   }
}

{  
   "datetime":"08-22-2016 12:38:04.854 -0700",
   "log_level":"INFO",
   "component":"HttpEventCollector",
   "data":{  
      "token_name":"test",
      "series":"http_event_collector_token",
      "transport":"http",
      "format":"json",
      "total_bytes_received":57000,
      "total_bytes_indexed":44000,
      "num_of_requests":1000,
      "num_of_events":1000,
      "num_of_errors":0,
      "num_of_parser_errors":0,
      "num_of_requests_to_disabled_token":0,
      "num_of_requests_in_mint_format":0
   }
}

HEC summary metrics

System-wide summary metrics are always accumulated even if there is no input activity. These metrics are identified by "series":"http_event_collector".

Field Description Value
component HTTP Event Collector metrics data identifier HttpEventCollector
data:format HTTP Event Collector data format json
data:num_of_auth_failures Total number of authentication failures due to invalid token unsigned integer
data:num_of_errors Total number of per-token errors, which include:
  1. bad data format
  2. no authorization
  3. bad authorization
  4. connectivity problems
unsigned integer
data:num_of_events Total number of per-token events received by the HTTP Event Collector endpoint unsigned integer
data:num_of_parser_errors Total number of per-token parser errors due to incorrectly formatted event data unsigned integer
data:num_of_requests Total number of valid per-token individual HTTP(S) requests received by an HTTP Event Collector endpoint. Each request can have one or more data events unsigned integer
data:num_of_requests_to_incorrect_url Total number of requests to an incorrect URL unsigned integer
data:num_of_requests_in_mint_format Total number of requests from Splunk MINT unsigned integer
data:num_of_requests_to_disabled_token Total number of per-token requests to disable token unsigned integer
data:series Metrics data type http_event_collector
data:total_bytes_indexed Total amount of per-token data sent to the indexer unsigned integer
data:total_bytes_received Total amount of per-token data received by calling the /receive/token endpoint unsigned integer
data:transport Data transport protocol for HTTP Event Collector data http
datetime Date and time associated with the data. Format:

MM-DD-YYYY HH:MM:SS.SSS +/-GMTDELTA

string
log_level Log severity level INFO

Per-token metrics

In contrast to the system-wide summary metrics, per-token metrics are accumulated only when HTTP Event Collector is enabled. These metrics are identified by "series":"http_event_collector_token".

The [http_input] stanza in the limits.conf file defines the logging interval and the maximum number of tokens logged for these metrics. To change any of these defaults, contact Splunk support.

Field Description Value
component HTTP Event Collector metrics data identifier HttpEventCollector
data:format HTTP Event Collector data format json
data:num_of_errors Total number of per-token errors, which include:
  1. bad data format
  2. no authorization
  3. bad authorization
  4. connectivity problems
unsigned integer
data:num_of_events Number of events received by the HTTP Event Collector endpoint unsigned integer
data:num_of_parser_errors Number of parser errors due to incorrectly formatted event data unsigned integer
data:num_of_requests Number of valid individual HTTP(S) requests received by an HTTP Event Collector endpoint. Each request can have one or more data events unsigned integer
data:num_of_requests_in_mint_format Total number of requests from Splunk MINT unsigned integer
data:num_of_requests_to_disabled_token Total number of per-token requests to disable token unsigned integer
data:series Metrics data type http_event_collector
data:token_name token name string
data:total_bytes_indexed Total amount of per-token data sent to the indexer unsigned integer
data:total_bytes_received Total amount of per-token data received by calling the /receive/token endpoint unsigned integer
data:transport Data transport protocol for HTTP Event Collector data http
datetime Date and time associated with the data. Format:

MM-DD-YYYY HH:MM:SS.SSS +/-GMTDELTA

string
log_level Log severity level INFO

Possible error codes

The following status codes have particular meaning for all HTTP Event Collector endpoints:

Status Code HTTP status code ID HTTP status code Status message
0 200 OK Success
1 403 Forbidden Token disabled
2 401 Unauthorized Token is required
3 401 Unauthorized Invalid authorization
4 403 Forbidden Invalid token
5 400 Bad Request No data
6 400 Bad Request Invalid data format
7 400 Bad Request Incorrect index
8 500 Internal Error Internal server error
9 503 Service Unavailable Server is busy
10 400 Bad Request Data channel is missing
11 400 Bad Request Invalid data channel
12 400 Bad Request Event field is required
13 400 Bad Request Event field cannot be blank
14 400 Bad Request ACK is disabled
15 400 Bad Request Error in handling indexed fields
16 400 Bad Request Query string authorization is not enabled

Cloud Monitoring Console

The Cloud Monitoring Console provides prebuilt dashboards for HTTP event collector that you can use to investigate your instance's performance.

Detecting scaling problems

If you are experiencing performance slowdowns, or want to speed up your HTTP Event Collector deployment, the following factors can affect performance:

  • HTTP vs. HTTPS: There is a significant performance improvement when sending data over HTTP versus sending data over HTTPS.
  • Batching: If you batch multiple events into single requests, it can speed up data transmission. Because a request's metadata applies to all events in the request, less data is sent overall. For more information about how event data is packaged, see format events for HTTP Event Collector.
  • HTTP Keep-alive: Setting keepalive on your connection can increase performance. As long as the client sending the data supports HTTP 1.1 and is set up to support a connection, you're taking advantage of keep-alive.
PREVIOUS
Automate indexed field extractions with HTTP Event Collector
  NEXT
Monitor First In, First Out (FIFO) queues

This documentation applies to the following versions of Splunk Cloud: 7.1.3, 7.1.6, 7.2.3, 7.2.4, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 8.0.0


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters