Prince Andrew is afraid that King Charles will kick him out of the house