Increasing Logging Verbosity of an ODAS cluster

Problem

The default logging level for an ODAS cluster does not always provide enough detail to troubleshoot performance issues.

Answer

NOTES

• Be thoughtful of the logging level that you set. Not only can you severely degrade performance, you may also log user data.
• The record service uses glog and that employs the -v log specification. Larger values log more information and level "4" is already very detailed. You would not want to scan a large dataset with logging dialed up that high.
• This change alters how the updated service logs its data, and is reflected both in the local log output (`sudo docker logs <docker id>`) as well as the output uploaded to S3.
  • The default value is 1;
  • Increasing verbosity to 2 or 3 is generally okay;
  • Using level 4 logs the rows returned from queries and will contain user data (that potentially contains PII).

The Kubernetes deployment specification controls the logging level for the ODAS services. The example below assumes you are changing the logging level for the worker service. To change other services, replace the deployment name in step 2.

1. Log into a host that has access to the Kubernetes master.

2. Find the deployment name.

sudo kubectl [ -n <namespace> ] get deployments

3. Edit the current configuration for the deployment. NOTE: This will drop you into a text editor

sudo kubectl [ -n <namespace> ] edit deployments <deployment_name>

4. Update the deployment specification to add the following to the environments variable section:

LOGGING_LEVEL 
"5"

5. Save and close the file. Kubernetes will restart the pods that correspond to that deployment using the updated settings.