Crumbling schools: Education secretary says most people won't be affected