Logging for Splunk

Logging for Splunk

This topic describes logging practices that help you use the power of Splunk. Splunk can reveal a tremendous amount of actionable information from within your logs, and these guidelines can help. The information in this section is taken from Logging Best Practices at Splunk Labs. Also see Why think about logging? at the same site.

Note: This page is not intended to replace existing logging best practices documents already on the web. It focuses solely on techniques to improve the performance and effectiveness of your logs in Splunk.

Formatting for Splunk

The following best practices make it easier for Splunk to handle your logs.

Use text

Don't include binary data, such as JPGs or Word files, in logs you are feeding to Splunk -- it cannot be meaningfully searched and analyzed. If you have binary data in your logs, include textual metadata in each event so that you can search on this data. For example, don't feed Splunk the binary data of a JPG file, but do log the properties, such as the image size, creation tool, username, camera, GPS location, etc. If you do have binary fields in your data, you can have Splunk strip them out during indexing. Remember, information you throw away before it ever reaches the Splunk index doesn't count towards your license.

XML logs can impact indexing performance because they require more work for Splunk to parse. Expect Splunk to index a given volume of XML logs more slowly than most other logs. You may also be able to strip out some of the XML structure during indexing to reduce overall volume in the index.

Timestamp every event

Most logging systems automatically timestamp your events. The correct time is critical to understanding the proper sequence of events and for debugging, analytics, and deriving transactions.

• Make your timestamps as explicit as possible (e.g., 2010 instead of 10). Splunk stores timestamps in UTC format.
• Use whatever time-granularity makes sense to your application.

Mark key-value pairs

One of the most powerful features of Splunk is its ability to extract fields from events at search time. This is how Splunk creates structure out of unstructured event data. To format your logs to guarantee that the field extractions will work as intended, create key/value pairs using the equals sign (=). You can delimit these fields with spaces or commas.

key=value,key2=value2,key3=value . . .

If your field values contain spaces, wrap your fields in quotes (e.g. username="bob smith") to ensure the fields are extracted correctly.

Adding value to your logs

Where appropriate, put semantic meaning in events so you can get more out of your data. Log audit trails, what users are doing, transactions, timing information -- anything that can add value when aggregated, charted, or analyzed. Specific examples follow.

Log unique identifiers

Things like transaction IDs and user IDs are tremendously helpful when debugging, and even more helpful when troubleshooting or gathering analytics. Unique IDs can point you to the exact transaction. Without them, many times all you have to go on is a time range.

If possible, make the format and content of your IDs consistent between modules. In some cases, adding a single well-chosen identifier across all relevant modules can make an enormous difference in your ability to effectively troubleshoot and monitor your application.

IDs need to be consistent and unique within a small window of time to be used successfully with the transaction command.

Application versions

Most application servers (WebLogic, JBoss, etc.) create a log entry when a server is installed and/or started. For platform servers, this already includes version information, which you can use for configuration management. For your own application containers, logging application versions along with your application name gives you a way to verify which versions of your applications are currently deployed in your environment. It also helps you compare application versions and configurations across environments, to verify normalization between dev, test, UAT, and prod.

Avoiding contention on Windows

The information in this section is taken from Patterns of writing logfiles on Windows on the Splunk community wiki.

There are specific issues that can arise when using Splunk on Windows. These problems are artifacts of the Windows file system and apply to any tool that attempts to read logs. The following issues can affect Splunk and logging on a Windows system:

1. File renaming -- for example, for file rolling -- can be problematic on Windows, especially when any application is actively reading from or writing to the file.

2. Exclusive access to files on Windows is enabled by default and specific action is required to enable concurrent access.

Renaming

On a Windows system, renaming is impossible so long as the file is open for either reading or writing. If your application is writing to the log file and Splunk is reading from the same file, essentially that file can never be renamed. To avoid problems, have your application write to new files instead of rolling -- for example, write to a new file each day, or simply write to a new file including the timestamp. If you are using log4j for your Java logs on Windows you can use the third-party dated file appender class, which contains no renames.

One known issue is that log4j 1.2.x prior to release 1.2.15 does not recover from the failure to rename the logfile in its default configurations (such as RollingLogAppender or DailyRollingAppender), and will overwrite its own datafiles if the rename attempts fail. Be sure to update to 1.2.15 or later if you are monitoring your log files via any method on Windows.

File-access exclusion

Whenever files are opened (via CreateFile() and equivalent calls), the default is exclusive access. In order to enable concurrent access by both a writer (a logging application) and a reader (such as Splunk), both programs must enable sharing flags. Splunk does enable the maximum amount of sharing possible in its attempts to open files for reading. However, if a logging application does not enable sharing for read-only access, a denial of access will occur when Splunk attempts to open the log file.

In order to mitigate this problem, you must actively enable sharing in your log file-writing routines. Alternatively a non-file data routing method (such as network transmission) may be possible. As a last resort, monitoring the next-to-most-recent log file may enable functional operation with a data lag.

• Was this topic useful?
• Post a comment