The problem with any past situation or relationship is how we decide to remember it. Either we aggrandize and embellish, we overlook, or just plain forget chunks of how it really was. The human tendency is to take a fresh mental sheet of paper and carefully sketch the past to be how we wish it had been. At that point, it no longer is our past, but rather a cleverly constructed self-deception.
Imagine the commercial fisherman who lowers his nets in the waters, returns the next day to pull in his catch. He throws some fish back, some he keeps. That's what our minds do. Sort, keep, and throw back. The ones we decide to keep become our new recollection, and that's what we hold onto.