Read timeout on large tables on EMR

Question

User is reporting a read timeout on large tables. What is the effect of altering "recordservice.worker.rpc.timeoutMs"?

Symptom

Application logs are reporting an error similar to the one below. Okera cerebro-worker logs show a corresponding timeout error as well.

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Could not get table: Could not reach service because of java.net.SocketTimeoutException: Read timed out.)

Solution

Ensure the RPC settings are appropriate for the cluster. The default values are 30 minutes for Okera workers and 30 seconds for the Okera planner.

Since generating a plan or getting partitions for very large tables might take more than 30s, increase the timeout as follows. On your EMR master node, open "yarn-site.xml" and look for the property "recordservice.planner.rpc.timeoutMs"

1. If it exists, increase the value.

2. If it does not exist, add the following to "yarn-site.xml"

recordservice.planner.rpc.timeoutMs=120000

3. Restart the EMR application.