Upgrade to a new version of Splunk App for Infrastructure that's monitoring Windows systems

What changed in version 2.2.0

Splunk App for Infrastructure (SAI) version 2.2.0 requires that you convert existing props stanzas to a new stanza format in props.conf if you add a custom perfmon object to SAI to improve performance.

When you upgrade to version 2.2.0 or higher from a pre-2.2.0 version and add a custom perfmon metrics object to inputs.conf, you also need to update the stanzas in props.conf.

Note: This is a new props stanza format that is different from the previous props stanza format added for perfmon metrics objects.

Old props stanza for custom perfmon objects

New props stanza for custom perfmon objects

[PerfmonMetrics:<object name>] TRANSFORMS-_value_for_perfmon_metrics_store_sai = value_for_perfmon_metrics_store_sai TRANSFORMS-metric_name_for_perfmon_metrics_store_sai = metric_name_for_perfmon_metrics_store_sai TRANSFORMS-object_for_perfmon_metrics_store_sai = object_for_perfmon_metrics_store_sai TRANSFORMS-instance_for_perfmon_metrics_store_sai = instance_for_perfmon_metrics_store_sai TRANSFORMS-collection_for_perfmon_metrics_store_sai = collection_for_perfmon_metrics_store_sai EVAL-metric_type = "gauge" SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

[PerfmonMetrics:<object name>] TRANSFORMS-_fields_for_perfmon_metrics_store_sai = fields_for_perfmon_metrics_store_sai TRANSFORMS-_value_for_perfmon_metrics_store_sai = value_for_perfmon_metrics_store_sai EVAL-metric_type = "gauge" SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

Example updated custom perfmon stanzaS

[PerfmonMetrics:<custom object 1>]
TRANSFORMS-_fields_for_perfmon_metrics_store_sai = fields_for_perfmon_metrics_store_sai
TRANSFORMS-_value_for_perfmon_metrics_store_sai = value_for_perfmon_metrics_store_sai
EVAL-metric_type = "gauge"
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

[PerfmonMetrics:<custom object 2>]
TRANSFORMS-_fields_for_perfmon_metrics_store_sai = fields_for_perfmon_metrics_store_sai
TRANSFORMS-_value_for_perfmon_metrics_store_sai = value_for_perfmon_metrics_store_sai
EVAL-metric_type = "gauge"
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

What changed in version 1.4.0

Splunk App for Infrastructure (SAI) version 1.4.0 changed Windows Performance (perfmon) inputs in inputs.conf and props in props.conf to migrate perfmon inputs from a log-based format to a metrics-based format in Splunk Enterprise. SAI stores metrics in a metrics index rather than an events index. In inputs.conf, Perfmon stanza names have changed, and each perfmon stanza now specifies a sourcetype. In props.conf: Perfmon prop names have changed.

When you upgrade to version 1.4.0 or higher from a pre-1.4.0 version, you must complete these actions:

• Update perfmon inputs in inputs.conf on every universal forwarder that's handling perfmon inputs.
• Update props in props.conf for the Splunk Add-on for Infrastructure.

Until you update Windows Performance Monitor (perfmon) inputs and props, you won't collect Windows perfmon data from Windows systems you're monitoring.

The following changes are for the default perfmon inputs you can configure from the Add Data pages in the UI. If you are collecting other perfmon inputs, you have to update the inputs and props for those as well.

These are the new perfmon stanza names and source types in inputs.conf:

Old perfmon stanza name	New perfmon stanza name	Source type
[perfmon://CPU Load]	[perfmon://CPU]	PerfmonMetrics:CPU
[perfmon://Physical Disk]	[perfmon://PhysicalDisk]	PerfmonMetrics:PhysicalDisk
[perfmon://Network Interface]	[perfmon://Network]	PerfmonMetrics:Network
[perfmon://Available Memory]	[perfmon://Memory]	PerfmonMetrics:Memory
[perfmon://System]	[perfmon://System]	PerfmonMetrics:System
[perfmon://Process]	[perfmon://Process]	PerfmonMetrics:Process
[perfmon://Free Disk Space]	[perfmon://LogicalDisk]	PerfmonMetrics:LogicalDisk

These are the new perfmon prop names in props.conf:

Old prop stanza name	New prop stanza name
[Perfmon:CPU]	[PerfmonMetrics:CPU]
[Perfmon:Memory]	[PerfmonMetrics:Memory]
[Perfmon:PhysicalDisk]	[PerfmonMetrics:PhysicalDisk]
[Perfmon:LogicallDisk]	[PerfmonMetrics:LogicallDisk]
[Perfmon:Network]	[PerfmonMetrics:Network]
[Perfmon:System]	[PerfmonMetrics:System]
[Perfmon:Process]	[PerfmonMetrics:Process]

Example updated perfmon stanzas

These perfmon stanzas are the default stanzas SAI adds to inputs.conf when you enable all default metrics on the Add Data page.

[perfmon://CPU]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 30
object = Processor
index = em_metrics
_meta =  os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:CPU

[perfmon://PhysicalDisk]
counters = % Disk Read Time;% Disk Write Time
instances = *
interval = 30
object = PhysicalDisk
index = em_metrics
_meta =  os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:PhysicalDisk

[perfmon://Network]
counters = Bytes Received/sec;Bytes Sent/sec;Packets Received/sec;Packets Sent/sec;Packets Received Errors;Packets Outbound Errors
instances = *
interval = 30
object = Network Interface
index = em_metrics
_meta =  os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:Network

[perfmon://Memory]
counters = Cache Bytes;% Committed Bytes In Use;Page Reads/sec;Pages Input/sec;Pages Output/sec;Committed Bytes;Available Bytes
interval = 30
object = Memory
index = em_metrics
_meta =  os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:Memory

[perfmon://System]
counters = Processor Queue Length;Threads
instances = *
interval = 30
object = System
index = em_metrics
_meta =  os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:System

[perfmon://Process]
counters = % Processor Time;% User Time;% Privileged Time
instances = *
interval = 30
object = Process
index = em_metrics
_meta =  os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:Process

[perfmon://LogicalDisk]
counters = Free Megabytes;% Free Space
instances = *
interval = 30
object = LogicalDisk
index = em_metrics
_meta =  os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:LogicalDisk

Example updated perfmon prop stanzas

These props stanzas are the default stanzas SAI adds to props.conf when you enable all default metrics on the Add Data page.

[PerfmonMetrics:CPU]
TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
EVAL-metric_type = "gauge"
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

[PerfmonMetrics:Memory]
TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
EVAL-metric_type = "gauge"
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

[PerfmonMetrics:PhysicalDisk]
TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
EVAL-metric_type = "gauge"
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

[PerfmonMetrics:LogicalDisk]
TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
EVAL-metric_type = "gauge"
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

[PerfmonMetrics:Network]
TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
EVAL-metric_type = "gauge"
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

[PerfmonMetrics:System]
TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
EVAL-metric_type = "gauge"
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

[PerfmonMetrics:Process]
TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
EVAL-metric_type = "gauge"
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

Steps

Follow these steps to update perfmon props for the Splunk Add-on for Infrastructure and perfmon inputs for universal forwarders on Windows systems when you upgrade to version 1.4.0. If you're using an indexer cluster manager node to manage a distributed indexer deployment, use it to upgrade the Splunk Add-on for Infrastructure on each indexer.

For information about upgrading apps and add-ons, see Manage app and add-on objects in the Splunk Enterprise Admin Manual. If you're running a distributed deployment, see Update common peer configurations and apps in the Splunk Enterprise Managing Indexers and Clusters of Indexers guide.

1. Upgrade SAI

Download SAI version 1.4.0 from Splunkbase and install the app package directly over existing app package as normal. You can do this manually, or from Splunk Web. There are no special steps to upgrade SAI if you're monitoring Windows systems. All of the work happens when you update the add-on and universal forwarders.

If you're running a search head cluster, see Install the Splunk App for Infrastructure in a distributed deployment.

2. Upgrade the Splunk Add-on for Infrastructure

When you upgrade the Splunk Add-on for Infrastructure, remove the old perfmon props and transforms from $SPLUNK_HOME/etc/apps/Splunk_TA_Infrastructure/default/props.conf and $SPLUNK_HOME/etc/apps/Splunk_TA_Infrastructure/default/transforms.conf.

If you're running a distributed deployment, see Install the Splunk App for Infrastructure in a distributed deployment for more information about installing the add-on in a distributed deployment.

1. On every instance running the Splunk Add-on for Infrastructure, move existing perfmon props from $SPLUNK_HOME/etc/apps/Splunk_TA_Infrastructure/default/props.conf to $SPLUNK_HOME/etc/apps/Splunk_TA_Infrastructure/local/props.conf. Doing this addresses a conf file precedence issue. These are the default stanzas SAI creates:

   [Perfmon:CPU]
   TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
   TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
   TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
   TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
   TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
   EVAL-metric_type = "gauge"
   SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g
   
   [Perfmon:Memory]
   TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
   TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
   TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
   TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
   TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
   EVAL-metric_type = "gauge"
   SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g
   
   [Perfmon:PhysicalDisk]
   TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
   TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
   TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
   TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
   TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
   EVAL-metric_type = "gauge"
   SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g
   
   [Perfmon:LogicalDisk]
   TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
   TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
   TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
   TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
   TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
   EVAL-metric_type = "gauge"
   SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g
   
   [Perfmon:Network]
   TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
   TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
   TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
   TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
   TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
   EVAL-metric_type = "gauge"
   SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g
   
   [Perfmon:System]
   TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
   TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
   TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
   TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
   TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
   EVAL-metric_type = "gauge"
   SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g
   
   [Perfmon:Process]
   TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
   TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
   TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
   TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
   TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
   EVAL-metric_type = "gauge"
   SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g
   

2. Upgrade the Splunk Add-on for Infrastructure to version 1.4.0.
3. Delete the props and transforms in $SPLUNK_HOME/etc/apps/Splunk_TA_Infrastructure/local/inputs.conf.

3. Update each Windows system you're monitoring

You must update perfmon stanzas in inputs.conf on the universal forwarder on each Windows systems you're monitoring. There are a few ways to do this:

• Run the updated easy install script from SAI version 1.4.0. For more information, see Collect Windows metrics and logs with Splunk App for Infrastructure in the Administer Splunk App for Infrastructure guide.
• Manually update perfmon stanzas on each universal forwarder. You must add the sourcetype field-value pair to each perfmon stanza in inputs.conf. For steps to do this, see Manually configure metrics and log collection for a Windows host for Splunk App for Infrastructure in the Administer Splunk App for Infrastructure guide.
• Use a third-party deployment server to update perfmon stanzas on each universal forwarder.