A job using distCp fails in an Okera-enabled cluster.

Problem:

A distCp job fails with this below error:

Container killed by the ApplicationMaster. 
Container killed on request. Exit code is 143 
Container exited with a non-zero exit code 143

18/08/28 16:23:27 INFO mapreduce.Job: Task Id : attempt_1535443630554_0019_r_000227_2, Status : FAILED 
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#1 
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) 
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377) 
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:422) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) 
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 
Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out. 
at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:366) 
at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:288) 
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:282) 
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323) 
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)

Container killed by the ApplicationMaster. 
Container killed on request. Exit code is 143 
Container exited with a non-zero exit code 143

18/08/28 16:23:28 INFO mapreduce.Job: Task Id : attempt_1535443630554_0019_r_000231_2, Status : FAILED 
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#1 
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) 
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377) 
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:422) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) 
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 
Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out. 
at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:366) 
at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:288) 
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:282) 
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323) 
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)

Container killed by the ApplicationMaster. 
Container killed on request. Exit code is 143 
Container exited with a non-zero exit code 143

18/08/28 16:23:28 INFO mapreduce.Job: Task Id : attempt_1535443630554_0019_r_000230_2, Status : FAILED 
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#1 
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) 
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377) 
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:422) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) 
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 
Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out. 
at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:366) 
at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:288) 
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:
282) 
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323) 
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)

Answer: 

distCp is a utility commonly used to move HDFS directories and files from one Hadoop cluster to another, or from one location in a single HDFS filesystem to another. It executes as a MapReduce job without regard to a file's content.

The error message shows one task (attempt_1535443630554_0019_r_000230_2) failing repeatedly as it tries to read data written by a prior task: 

Shuffle$ShuffleError: error in shuffle in fetcher#1

When this happens, the job's ApplicationMaster kills the task container and tries again. After four attempts (default), the ApplicationMaster will bail on the job:

java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out

The job owner may not have read permission for the prior task's output location, or possibly is not listed as a user on the node (UserGroupInformation.doAs(UserGroupInformation.java:1698)). Check the directory permissions and the user database on the system in question.