Queen Elizabeth wanted Prince Andrew to stay in the Royal Lodge