Problem
Alter/Msck repair queries have created lock on partitions table on HMS.
We have restarted catalog service and tried deleting the locks from HMS directly, however the locks are coming back again.
We have restarted catalog service and tried deleting the locks from HMS directly, however the locks are coming back again.
Solution
Note: The cluster deployment has changed and the DM/env.sh mechanism no longer exists. To add environment variables, edit the kubernetes configmap and restart the Okera catalog pod.
This issue occurred due to RDS setting a write lock caused by MSCK repair and auto-recovery of Partitions being set at the same time. The solution is to disable auto-recovery of partitions on the clusters these jobs are running. The steps to do this are as follows
1) Edit the env.sh
2) Add to the property:
SERVICE_ENVIRONMENT_CONFIGS
The following property:
'OKERA_DISABLE_AUTO_RECOVER_PARTITIONS=true'
OR
To make sure you don't remove any presets, you can simply add
OKERA_SERVICE_ENVIRONMENT_CONFIGS='$OKERA_SERVICE_ENVIRONMENT_CONFIGS;OKERA_DISABLE_AUTO_RECOVER_PARTITIONS=true;'
3) Restart the DM and start your cluster.
Comments
0 comments
Please sign in to leave a comment.