Troubleshooting Overloaded Worker

Problem

A cerebro-worker is overloaded

Solution

To understand where the connections are coming from on an overloaded worker,  collect the output from a few of the stats endpoints on the affected worker:

1. Use the "System" tab in the Okera UI to identify the host and IP address for the "cerebro_worker:webui" service of the overloaded worker
2. Option 1 (if the port is exposed):
   1. Paste the <host:port> into a browser.
   2. Click on the "/queries" link to look at the list of queries.
   3. Click on the "/sessions" link to look at the list of sessions and related information.
   4. Click on the "/logs" link to look for errors.
3. Option 2:
   1. Use kubernetes to exec onto that host. 
   2. Find the kubernetes pod running the worker:

kubectl exec -it <container id> curl localhost:11050/queries > /tmp/queries.txt

kubectl exec -it 5c9e6d107e3c curl localhost:11050/queries > /tmp/queries.txt

kubectl exec -it 5c9e6d107e3c curl localhost:11050/sessions > /tmp/sessions.txt

Look at the output of sessions.txt and queries.txt for the query that is causing the bottleneck.