Users facing token issue while spark-submit in cluster mode

Problem

A user is receiving the below error for the spark job:

pyspark.sql.utils.AnalysisException:u”org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table <table name>. Could not connect to planner due to IOException: Could not get version from ‘planner’ at ‘10.180.45.59:50075’. This likely indicates an authentication issue:\nClient did not provide any authentication credentials. Ensure the server is also running unauthenticated.;”

Solution

If users facing token issue while spark-submit in cluster mode, user needs to pass the spark property spark.recordservice.delegation-token.token as part of the spark-submit. For example:

Usage spark-submit ... --conf spark.recordservice.delegation-token.token=<Your-Token-text-here>