Delete Table with Large Number of Partitions

Problem

A production table has many partitions many files. The data producer has changed the Avro schema and we need to apply the new schema to the table.

Solution

There are two workarounds which will enable you to drop the table.

Workaround 1

alter table <table_name> rename to <table_name_old> ;
create table <table_name> ... ;
msck repair table <table_name> ;

Once this operation is complete, you can go back and clean up table_name_old.

Workaround 2

This solution is executed in a script where the partitions are iterated through in a loop and deleted.

alter table <table_name> drop partition <partition_name>