Who Do You Vote to Punish?