Documentation > Splunk > Admin Manual > Configure character set encoding

Admin Manual

 


Welcome to Splunk administration
What to do first
Use Splunk Web
Meet Splunk apps
Configure Splunk
Indexing with Splunk
Add data and configure inputs
Set up forwarding and receiving
Configure event processing
Configure event timestamping
Configure indexed field extraction
Manage users
Manage indexes
Define alerts
Set up backups and retention policies
Configure data security
Deploy to other Splunk instances
Set up distributed search
Manage search jobs
Use Splunk's command line interface (CLI)
Configuration file reference
Troubleshooting

Configure character set encoding

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.

Contents

Configure character set encoding

Splunk allows you to configure character set encoding for your data sources. Splunk has built-in character set specifications to support internationalization of your Splunk deployment. Splunk supports 71 languages (including 20 that aren't UTF-8 encoded). You can retrieve a list of Splunk's valid character encoding specifications by using the iconv -l command on most *nix systems.

Splunk attempts to apply UTF-8 encoding to your sources by default. If a source doesn't use UTF-8 encoding or is a non-ASCII file, Splunk will try to convert data from the source to UTF-8 encoding unless you specify a character set to use by setting the CHARSET key in props.conf.

Note: If a source's character set encoding is valid, but some characters from the specification are not valid in the encoding you specify, Splunk escapes the invalid characters as hex values (for example: "\xF3").

Supported character sets

Splunk supports an extremely wide range of character sets, including such key ones as UTF-8, UTF-16LE, Latin-1, BIG5, and SHIFT-JIS. See "Comprehensive list of supported character sets" at the end of this topic for the exhaustive list.

Here's a short list of some of the main character sets that Splunk supports, along with the languages they correspond to.

Language Code
Arabic CP1256
Arabic ISO-8859-6
Armenian ARMSCII-8
Belarus CP1251
Bulgarian ISO-8859-5
Czech ISO-8859-2
Georgian Georgian-Academy
Greek ISO-8859-7
Hebrew ISO-8859-8
Japanese EUC-JP
Japanese SHIFT-JIS
Korean EUC-KR
Russian CP1251
Russian ISO-8859-5
Russian KOI8-R
Slovak CP1250
Slovenian ISO-8859-2
Thai TIS-620
Ukrainian KOI8-U
Vietnamese VISCII

Manually specify a character set

Manually specify a character set to apply to an input by setting the CHARSET key in props.conf: 

[spec]
CHARSET=<string>

For example, if you have a host that is generating data in Greek (called "GreekSource" in this example) and that uses ISO-8859-7 encoding, set CHARSET=ISO-8859-7 for that host in props.conf: 

[host::GreekSource]
CHARSET=ISO-8859-7

Note: Splunk will only parse character encodings that have UTF-8 mappings. Some EUC-JP characters do not have a mapped UTF-8 encoding.

Automatically specify a character set

Splunk can automatically detect languages and proper character sets using its sophisticated character set encoding algorithm.

Configure Splunk to automatically detect the proper language and character set encoding for a particular input by setting CHARSET=AUTO for the input in props.conf. For example, if you want Splunk to automatically detect character set encoding for the host "my-foreign-docs", set CHARSET=AUTO for that host in props.conf: 

[host::my-foreign-docs]
CHARSET=AUTO

If Splunk doesn't recognize a character set

If you want to use an encoding that Splunk doesn't recognize, train Splunk to recognize the character set by adding a sample file to the following directory: 

$SPLUNK_HOME/etc/ngram-models/_<language>-<encoding>.txt

Once you add the character set specification file, you must restart Splunk. After you restart, Splunk can recognize sources that use the new character set, and will automatically convert them to UTF-8 format at index time.

For example, if you want to use the "vulcan-ISO-12345" character set, copy the specification file to the following path: 

/SPLUNK_HOME/etc/ngram-models/_vulcan-ISO-12345.txt

Comprehensive list of supported character sets

The common character sets described earlier are just the tip of Splunk's CHARSET iceberg. Splunk actually supports a long list of character sets and aliases, identical to the list supported by the *nix iconv utility. Here's the full list, with aliases indicated in parantheses:

Note: Splunk ignores punctuation and case when matching CHARSET, so, for example, "utf-8", "UTF-8", and "utf8" are all considered identical.

Retrieved from "http://docs.splunk.com/Documentation/Splunk/4.1/Admin/Configurecharactersetencoding"

This documentation applies to the following versions of Splunk: 4.1 , 4.1.1 , 4.1.2 , 4.1.3 , 4.1.4 , 4.1.5 , 4.1.6 , 4.1.7 , 4.1.8 View the Article History for its revisions.


Comments

"Splunk escapes the invalid characters as hex values (for example: "\xF3")."

How do we tell splunk to just ignore them?

Oreoshake
July 24, 2011

You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!