Crowded Conservative leadership race could shrink