Roger Federer finished his U.S. Open match in front of an almost empty stadium