Show Full Spark Query in Okera Audit Logs

Problem

When running a query in Databricks, the full Spark query does not appear in the Okera audit logs.

Answer

There are two configuration parameters that are required.

On the server side, set the audit_request_query parameter as part of the RS_ARGS definition.

For example,

RS_ARGS: ... current configs ... --audit_request_query=true

On the client side, add the following to the environment variable section in the in Databricks client configuration:

AUTO_POPULATE_REQUEST_NAME=true