I wouldn't jump to a causal connection immediately, but totally possible.
Why don't we get an 'extra have' with other modals? "I wanted've ..." or "I can've..." etc.
The only similarity I can see is that "have" and "would" (and "should", "could") both reduce to
'd in certain circumstances. "have" and these modals aren't otherwise in the same class. There must be some analogy going on, and the limit to a word that happens to have that ending is interesting. I have no direct evidence, though.
I'm not sure I follow what you mean. Can you elaborate?
I'm not sure that "of" is a
mistake. It might be how it's actually represented (and acquired) in spoken English. "Of" thus joins "to" ("I want to...") and "from" ("I refrain from...") as a verbal connecting particle.