In a language like Latin, the answer for "yes" is to repeat the verb in the question:
A: Did you read the book?
B: I read. ['Yes.']
Therefore, I believe that the word yes or no is substituting for a whole sentence. Consider these other proforms:
John walked home then he went to sleep.
John walked home quickly, but Mary less so.
John went home by bus, but Mary did so by train.
Now consider (with the right intonation!):
John won the game, but Mary, no.
John didn't win the game, but Mary, yes.
A simple way to think about this is what part of speech these words are. I can only think that they substitute for full sentences.