The (also probably simplified) answer seems to me to be that there is a difference between perception and production.
Everyone, including adults, can adjust quickly to perceive new accents. They might even, in an experiment, get tricked into perceiving a word the wrong way (think "die" vs. "day" in an Australian accent) if they have heard the new accent for long enough (where an American saying "die" in the experiment is now perceived like the Australian saying "day" so they research participant hears "day"). But after getting new input like this (and storing the exemplars) they won't start producing speech that way after the experiment.
So it isn't an issue of what exemplars exist but how they are used in perception and production (which might differ).
I'm not sure how specific versions of the theory would explain the details. But I imagine that's a starting point.
An alternative explanation would just be that older exemplars don't fade very quickly in adults and habits are retained more easily. Too much experience, too little change/learning.