Troubleshoot Hunk

This section describes some of the issues you may have with various components of your configuration and possible ways to resolve those issues.

For more troubleshooting questions and answers, and to post questions yourself, search for Hunk issues in Splunk Answers.

Issue: Splunk throws a "failed search" message.

For example:

[APACHE] External result provider name=APACHE asked to finalize the search
[APACHE] MapReduce job id=job_201303081521_0020 failed, state=FAILED, message=# of failed Map Tasks exceeded allowed limit. FailedCount: 1.   
LastFailedTask: task_201303081521_0020_m_000000 

This sort of error appears because of java child processes that are also running. Check the MapReduce logs, where you should see something like the following:

TaskTree [pid=7535,tipID=attempt_201303061716_0093_m_000000_0] is running beyond memory-limits. 
Current usage : 2467721216bytes. Limit : 2147483648bytes. Killing task.

To resolve this, edit indexes.conf as follows:

vix.mapred.child.java.opts = -server -Xmx1024m

Issue: A reporting search throws an error.    

If a reporting search throws the following error:

INFO mapred.JobClient: Cleaning up the staging area hdfs://qa-centos-amd64-26.sv.splunk.com:8020/user/apatil/.staging/job_201303061716_0033 
ERROR security.UserGroupInformation: PriviledgedActionException as:apatil cause:org.apache.hadoop.ipc.RemoteException: java.io.IOException:
job_201303061716_0033(-1 memForMapTasks -1 memForReduceTasks): Invalid job requirements.
at org.apache.hadoop.mapred.JobTracker.checkMemoryRequirements(JobTracker.java:5019)

Try adding the following parameters to indexes.conf

vix.mapred.job.map.memory.mb = 2048
vix.mapred.job.reduce.memory.mb = 256

Issue: Hadoop fails to start

Make sure that the user account has proper permission to the needed Hadoop directories.