The Complexity of Simple Solutions