To Memorize or Not to Memorize?