Charlie Grooms, the fired chairman of the Miss Madison nonprofit, said they had no choice.