Increase Planner timeout for large tables

Problem

Observing a timeout while waiting for the Okera planner to scan a large table. This is an issue where metadata operations in S3 take longer compared to other sources. What should be the timeout setting?

Solution

The time taken by the ODAS Planner to plan a given query is directly proportional to the number of files being scanned. A good way to find an acceptable timeout value is: 

1. Set a very high timeout value, for example 24h
2. Run the query that was failing
3. Look at the audit logs, and see how much time the planner took to plan this query
4. Increase the timeout to some reasonable number greater than that

General Notes

Instead of setting very high timeouts, Okera recommends to limit the scope of queries for slower storage systems. For example, the use of LAST PARTITION with a dedicated VIEW is more efficient and leads to fewer (unintentional) full table scans.

For Hive and Spark on EMR, the timeouts can be set in the following way

1. Edit "/etc/hadoop/conf/core-site.xml"
2. Set or add the key "recordservice.planner.rpc.timeoutMs"
3. Set the value as desired
4. Make sure this value is also not redundantly set in "/etc/hadoop/conf/yarn-site.xml"
5. Restart EMR services using "/usr/lib/cerebro/restart-emr.sh"

For Presto on EMR

1. Go to and edit "/etc/presto/conf/catalog/recordservice.properties"
2. Add the "recordservice.planner.rpc.timeoutMs" key and value
3. Restart EMR services using "/usr/lib/cerebro/restart-emr.sh"

Example

The current timeout is set to five minutes and a specific table with 10 months of data (with daily partitioning) can be planned reliably. The goal is to query (plan) two years worth of data, requiring a new timeout set to about 15 minutes (that is, the increase in metadata operations is roughly linear with the planning time needed). A timeout value of 30 minutes is chosen to allow for some extra leeway.