Configure inputs for the Splunk Add-on for Microsoft SCOM

Configure inputs on the node responsible for data collection. The Splunk Add-on for Microsoft SCOM supports three different methods for configuring inputs:

1. Use the Splunk Add-on for Microsoft SCOM configuration UI.
2. Use the PowerShell scripted input UI.
3. Use the configuration files.

Regardless of the method you use, switching to a different method to update data inputs later may cause inconsistencies in data collection. Splunk recommends that you configure inputs through the Splunk Add-on for Microsoft SCOM configuration UI.

Splunk recommends that you do not override the source types in this add-on. In all cases where the input collects data using a group metric (a collection of individual SCOM commands), the modular input code sets the source type based on the group from which the event originated, regardless of any custom source type you attempt to set in the configuration files or the UI configuration screens. You can override the source type for any input configured to collect metrics consisting of individual SCOM commands, but this can cause inconsistencies and errors in CIM mapping, dashboard displays, and workflow actions.

The first time when you configure the inputs to collect performance data from Microsoft SCOM, it may take a long time (about 20 minutes) to get data in.

Configure inputs through the Splunk Add-on for Microsoft SCOM configuration UI

1. Access Splunk Web on the node responsible for data collection.
2. Go to the Splunk Add-on for Microsoft SCOM configuration page by clicking on the Microsoft SCOM add-on in the left navigation banner on the Splunk platform home page. You can also go to Manage Apps, then click Launch App in the row for Splunk Add-on for Microsoft SCOM.
3. Review the configured inputs. The templates that are assigned to each input provide the metrics to be collected.
4. Click the Configuration tab to review the configured templates on the Template tab and see the metrics each template collects.
   1. To add or remove metrics from a template, click Edit under the Actions menu next to a template.
   2. You can select a metric group to automatically run all the related commands in that group, or choose individual SCOM commands.
   3. To create a new template, click Add. Give your template a name and select a set of metrics. You can select a metric group to automatically run all the related commands in that group, or choose individual SCOM commands.
5. If you have management servers for which you want to collect SCOM data installed on separate machines from your Operations console, click the Server tab to provide information about the remote management servers.
   1. Go to Microsoft SCOM Server and Click Add.
   2. Provide a name, host (for example, scom-management) and a username and password that the add-on can use to connect to this server.
   3. For Host, you can provide the hostname, IP address, or fully qualified domain name.
   4. For Username and Password, use an Admin account or another account that has read permissions for the commands you are using to collect SCOM data. If you are using a domain user account, use the format <domain>\<username> in the Username field.
   5. Do not edit an existing server to add a new remote server. The checkpoint is correlated to the server name so editing an existing remote server's host, username, and password will reuse the checkpoint file which can cause event data loss.
6. Click the Logging tab to set the logging level to use. You can select WARN, DEBUG, or ERROR. The default is WARN.
7. Review or add inputs by clicking Inputs tab at the top.
   1. Click Edit under the Action menu to adjust a configured input. You can add or remove templates, change the server to a remote server if necessary, set a different interval by entering a different cron expression, set the start time or select a custom index.
   2. If the input settings match your use case, you can leave them unchanged and click the Status toggle button to enable it as a data input.
8. If you prefer to create a new input, click Create New Input.
   1. Give your input a name and select one or more collection templates.
   2. If you are collecting data from a remote server, select the server name from the list of remote servers you previously added under the Server tab or select localhost if the server is local.
   3. Provide a schedule by entering a cron expression in the Interval field and the Start Time.
9. If a template is selected which has performance command (Metrics as cmd=Get-SCOMAllPerfData), then a field filter parameter will be visible:
   1. By default the value will be "CounterName IS NOT NULL", this will fetch all the performance data. This is also applicable when this field is kept empty.
   2. Users can also add filter parameters as per their requirements to fetch specific data.
   3. The filter parameter should be in SCOM defined syntax of criteria expression.
   4. The property name which can be used in criteria expression can be found on https://docs.microsoft.com/en-us/dotnet/api/microsoft.enterprisemanagement.monitoring.monitoringperformancedatacriteria?view=sc-om-dotnet-2019#remarks page and the expression should be formed as per criteria expression syntax defined on https://docs.microsoft.com/en-us/previous-versions/system-center/developer/bb437603%28v=msdn.10%29?redirectedfrom=MSDN page.
   5. Some of the examples of the valid expressions are:
      1. ObjectName MATCHES 'Health Service'
      2. CounterName = 'NumberAgents'
      3. MonitoringClassId = 'ab4c891f-3359-3fb6-0704-075fbfe36710'

Default value for Start Time field is one day before current UTC time.

1. 1. You can also select a custom index for the data.
2. You can also copy an existing input by selecting Clone from the Action link of the input to start with the settings of the input.
3. Enable the inputs to start collecting data. No restart is required.
4. Click the Search tab at the top and perform the following search to verify the Splunk platform is indexing the data you expect.

   sourcetype=microsoft:scom*

5. If you encounter problems or do not see the data you expect, see the Troubleshooting suggestions.

To prevent the Splunk platform from indexing duplicate data, do not enable more than one input that collects the same metric more than once. For example, if you enable one input configured to use a template that collects the Alerts group and another input configured to use a template that collects data using individual alert commands, the Splunk platform indexes the alert command data twice. Verify that each input that you enable invokes templates that do not have overlapping metrics. To see how the groups and commands are related, see the source types table.

Configure inputs through the PowerShell scripted input UI

Prerequisite: If you are collecting data from a remote management server with this script, you must first add the remote management server using the Splunk Add-on for MS SCOM configuration user interface or in the microsoft_scom_servers.conf file before creating the PowerShell scripted input.

1. Access Splunk Web on the node responsible for data collection. Click Splunk in the upper left to start from the home screen.
2. Go to Settings > Data inputs, then select PowerShell v3 Modular Input.
3. Click New.
4. Enter a name for your input.
5. In the Command or Script Path field, enter & "$SplunkHome\etc\apps\Splunk_TA_microsoft-scom\bin\scom_command_loader.ps1" followed by one or more metrics expressed as groups or commands. For example: & "$SplunkHome\etc\apps\Splunk_TA_microsoft-scom\bin\scom_command_loader.ps1" -groups override or & "$SplunkHome\etc\apps\Splunk_TA_microsoft-scom\bin\scom_command_loader.ps1" -commands Get-SCOMAlert, Get-SCOMEvent
6. If you are collecting data from a remote management server with this script, you need to specify the -server parameter and the stanza name for the remote server from the microsoft_scom_servers.conf file. For example: & "$SplunkHome\etc\apps\Splunk_TA_microsoft-scom\bin\scom_command_loader.ps1" -groups task -server remote_management_server
7. You can also specify the -starttime and -loglevel parameter. For example: & "$SplunkHome\etc\apps\Splunk_TA_microsoft-scom\bin\scom_command_loader.ps1" -groups task -server remote_management_server -loglevel DEBUG -starttime "2021-01-02T00:00:00Z"
8. You can also specify the -performancefilter. For example: &"$SplunkHome\etc\apps\Splunk_TA_microsoft-scom\bin\scom_command_loader.ps1" -commands "get-scomallperfdata" -server "abcd" -loglevel WARN -starttime "2022-02-21T07:21:02Z" -performancefilter "CounterName IS NOT NULL"
9. For more information about the group and individual command metrics, see the source types table.
10. Enter a cron expression in the Cron Schedule field to specify how often the data should be collected.
11. Optionally, check More settings to configure a custom source type, host or index value.
12. Click Next to save and enable the input.
13. Go to the Search & Reporting app and search for sourcetype=microsoft:scom* to verify the Splunk platform is indexing the data you expect.
14. If you encounter problems or do not see the data you expect, see the Troubleshooting suggestions.

To prevent the Splunk platform from indexing duplicate data, do not create multiple inputs using the same metrics. For example, if you enable one input using the Alerts group and another input using individual alert commands, the Splunk platform indexes the alert command data twice. To see how the groups and commands are related, see the source types table.

Configure inputs through the configuration files

If you want to collect SCOM data from management servers that are installed on separate machines from your Operations console, you need to provide information about the remote management servers in the microsoft_scom_servers.conf file before you configure your inputs in the inputs.conf file.

Configure remote servers in microsoft_scom_servers.conf

If the management server and the Operations console are installed on the same machine, you do not need to perform the steps in this section.

1. Open %SPLUNK_HOME%\etc\apps\Splunk_TA_microsoft-scom\default\microsoft_scom_servers.conf.
2. Copy the contents to %SPLUNK_HOME%\etc\apps\Splunk_TA_microsoft-scom\local\microsoft_scom_servers.conf. The contents look like this:

   [localhost]
   host = localhost
   password = ********
   username =
   

3. Change the [localhost] stanza name to a name that describes the remote server, then type the name of the host and a username and password for an account on that server. Do not use / or \ characters in the stanza name. For Host, you can provide the hostname, IP address, or fully qualified domain name.
4. For username and password, use an Admin account or another account that has read permissions for the commands you are using to collect SCOM data. If you are using a domain user account, use the format <domain>\<username> for username. For example:

   [remote_management_server]
   host = <your SCOM host>
   password = <your SCOM server password>
   username = <your SCOM server username>
   

   The password will be encrypted upon reloading the configuration page.
5. Create a stanza for each remote management server you want to collect metrics for, then save the file.

If you need to add a new remote server, do not edit an existing server's host, username and password. The checkpoint is strongly correlated to the server's stanza name. Editing an existing stanza will reuse the checkpoint file which can cause event data loss.

Configure local inputs.conf

1. Open %SPLUNK_HOME%\etc\apps\Splunk_TA_microsoft-scom\default\inputs.conf.template.
2. Copy the contents to %SPLUNK_HOME%\etc\apps\Splunk_TA_microsoft-scom\local\inputs.conf.
3. Enable the inputs for one or more of the predefined stanzas by changing disabled = 1 to disabled = 0.
4. If you would like to customize your own input, customize the final stanza [powershell://scom_commands] with your desired settings and enable it. Follow the instructions in the file.
5. Go to the Search & Reporting app and search for sourcetype=microsoft:scom* to verify the Splunk platform is indexing the data you expect.
6. If you encounter problems or do not see the data you expect, see the Troubleshooting suggestions.

To prevent the Splunk platform from indexing duplicate data, do not create more than one input using the same metrics. For example, if you enable one input using the Alerts group and another input using individual alert commands, the Splunk platform indexes the alert command data twice. To see how the groups and commands are related, see the source types table.