Configuration monitoring

Configuration monitoring

You are probably familiar with configuration files, but in the application management world, there are additional ways to think about config that might not be as familiar.

Note: This topic currently provides an outline, which will be expanded over time. The outline and contents may change.

Outline of configuration monitoring

• What is "configuration"?
  • Code/versions deployed, for example, verify .jar files against MD5 sum
  • Traditional application or device configuration
  • Data / Metadata (e.g., database table schemas & indexing)
• Where do you get configuration info?
  • *nix: configuration files
  • Network devices: (give examples)
  • Windows: varies more. May be registry, WMI, files, etc.
  • Other sources including configuration database/CMDB
• Configuration monitoring with Splunk
• Configuration data input options
  • Track changes only (no contents) vs. index file contents
  • OS auditing vs. fschange vs. scripted inputs (when to use each?)
• Comparing against known-good configuration
  • Alerting on changes
  • Tying alerts to ticketing system
  • Correlating with performance/availability issues
• Integration with other configuration management tools

Basic use case: monitoring files handles on a secure box.

Example: monitor changes to a WSDL

This section is based on the Splunk blog post Change Management for SOA Configuration.

A typical Service-oriented Architecture (SOA) deployment has a number of configuration files stored directly on the file system of a production machine. At a minimum these include:

• XML Schema (XSD) files
• Web Services Definition Language (WSDL) files
• XML files

This section shows how to monitor a WSDL directory and its subdirectories for file changes and then use the diff command on a changed file to determine precisely what changes were made.

Set up fschange

fschange is a scripted input that uses the underlying file system's change monitoring mechanism to detect when changes are made to the status of a file. This includes not only when a file's content changes, but also changes to permissions and modes.

Note: fschange is a polling-based input that can be used to monitor and alert on configuration changes that affect application availability and performance. To monitor file changes for security purposes, you should use the file auditing facilities native to your operating system.

To set up a complex file input like this, you often want to skip Splunk's Manager interface and go directly to the configuration files. File inputs can be set in the $SPLUNK_HOME/etc/system/local/ directory.

Note: Configuration files are stored in a number of locations, including various default directories, such as $SPLUNK_HOME/etc/system/default/. These are used by Splunk for internal configurations and will be overwritten whenever you upgrade Splunk. To ensure your configurations are retained on upgrade, make sure to always save your custom configurations in the local directory.

To set up change monitoring for your WSDL configuration files:

1. Open the inputs.conf file in $SPLUNK_HOME/etc/system/local/ in a text editor. If the file does not yet exist, create it.

2. Enter the following stanza to monitor the WSDL configuration directory:

[fschange:/AppServer/configuration/wsdl]
sourcetype = wsdl_monitor
index = testing
disabled = false
filters = tilda,files
recurse = true
pollPeriod = 600
fullEvent = true
sendEventMaxSize = -1

[filter:blacklist:tilda] 
regex1 = [~]$

[filter:whitelist:files] 
regex1 = \.wsdl$

Things to know:

• [fschange:/AppServer/configuration/wsdl] is the beginning of an input stanza that sets the type of input (fschange) and the location to be monitored.
• sourcetype = wsdl_monitor sets the source type to the string wsdl_monitor. If you do not set the sourcetype, Splunk uses fschange.
• index = test sets the target index for the fschange output. If you do not set this, Splunk sends fschange events to the default index, unless you set signedaudit = true.
• disabled = false enables fschange.
• filter:blacklist:tilda = [~]$ tells Splunk not to monitor any files ending with a tilde (~). These are WSDL backup files.
• filter:whitelist:files = \.wsdl$ tells Splunk to monitor all files ending in .wsdl.
• recurse = true sets fschange to monitor files in all subdirectories of the parent /AppServer/configuration/wsdl directory.
• pollPeriod = 600 sets how frequently (in seconds) fschange polls the underlying file system for changes.
• fullEvent = true tells Splunk to index the full contents of a file whenever a change is detected. This allows you to compare different versions of the file and determine what has changed. This is the only way you should monitor the contents of a file that is being watched by fschange.
• sendEventMaxSize = -1 tells Splunk that the entire file should be indexed after a change regardless of size. The sendEventMaxSize parameter allows you to set an upper limit (in bytes) on the file size to be indexed in full. -1 (the default) means there is no limit.

3. Restart Splunk.

Splunk will now monitor all changes to these configuration files.

Search for changes

If you use a unique source type like wsdl_monitor, you can search for changes by searching on the source type. If you use a _monitor or _fschange suffix for all your fschange inputs, you can extend your searches to things like sourcetype=*_monitor.

Note: Be careful, however, not to use multiple underscores. A bug in some versions of Splunk means that a search like sourcetype=*_monitor only finds matches with a single underscore. It will not find my_wsdl_monitor.

To search for changes:

1. Enter the following from the Search bar:

Here are some typical events returned by fschange:

Mon Mar 16 18:22:06 2009 action=update, path="/AppServer/configuration/wsdl/iptocountry.wsdl", isdir=0, size=9614, gid=20, uid=503, modtime="Mon Mar 16 18:21:58 2009", mode="rw-r--r--", hash=, chgs="modtime "
source=fschangemonitor | host=macbookpro.local | sourcetype=wsdl_monitor

Mon Mar 16 18:19:03 2009 action=add, path="/AppServer/configuration/wsdl/iptocountry.wsdl", isdir=0, size=9599, gid=20, uid=503, modtime="Mon Mar 16 18:18:02 2009", mode="rw-r--r--", hash=
source=fschangemonitor | host=macbookpro.local | sourcetype=wsdl_monitor

Mon Mar 16 18:09:49 2009 action=delete, path="/AppServer/configuration/wsdl/WeatherService.wsdl"
source=fschangemonitor | host=macbookpro.local | sourcetype=wsdl_monitor

You can now tell which files or directories changed and when.

The fschange output contains a number of fields you can use to search on or analyze your events. For example, the action field shows the nature of the action (add, update or delete) that resulted in the file modification.

Alert on changes

You can easily use Splunk to monitor changes to your application configuration and set up proactive alerts that fire when changes happen. In addition, if a scheduled change was supposed to happen, and a search yields no results within the last 24 hours, the absence of a change may be worthy of an alert.

Compare versions of a monitored file

You can use the diff command in Splunk to compare different versions of a changed file. The diff command compares two events in your search results and shows the difference. Because you set fullEvent = true in your input.conf file, the contents of the configuration file are indexed as an "event" each time the file changes. By default diff compares the first and second events returned, that is the most recent is compared to the next most recent. You can specify the events to compare. See diff in the Search Reference manual for more information.

To compare a changed version of the file with a previous version:

1. Enter the following search in the Search bar:

2. You will get results such as the following:

   </wsdl:message>
   <wsdl:portType name="iptocountrySoap">
     <wsdl:operation name="FindCountryAsString">
-      <wsdl:documentation xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/">This returns a string value for Country.</wsdl:documentation>
+      <wsdl:documentation xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/">This returns a string value</wsdl:documentation>

Things to know:

• - indicates a line present in the latest version but missing in the previous version
• + indicates a line present in the previous version but missing in the latest version
•  ! (not shown here) indicates a line that exists in both versions, but contains different information

In this case, the change is for the documentation, so it is relatively harmless.

If you craft your search properly, you can also use diff to compare a configuration in the wild against a master file to see if they match.

• Was this topic useful?
• Post a comment