The name of familiar from Popper, but I will address the idea as you present it. The claim, I take it, is that "a society as a whole behaves as if every individual were rational". This is plainly false. Society as a whole does not act as if every individual were rational. The claim depends on a number of un-inspected concepts: "society", "as a whole", "behaves", and "rational" (perhaps also "individual"). Let's take these in order of occurrence.
"Society" in the singular is a false abstraction: there are thousands of overlapping societies. There are a couple hundred countries (defining one class of societies) and many of them have sub-societies (region, district, province, state, canton etc). The Roman Catholic church is another organizing principle, idem the Ethiopian Orthodox Church, Islam (with subdivisions) as well as the Pastafarians; there are professional societies, literary societies, groupings that racially-based, education-based, philosophy-based and so on. There is nothing that even vaguely resembles a listing of "the societies", and a person is a member of myriad societies. Moral of the story, some societies see the world one way, others see the world another way.
"As a whole" is meaningless, since society is an abstraction over individuals and society cannot behave at all. Perhaps you mean "the majority of individuals in the society", but maybe you mean "the important members of the society", which gets us into a whole 'nother endless digression (consider "the society of anarchists" vs. "North Korean society"). Similarly, society cannot "behave", only individuals can. In addition, "behave as if" is really an idiom, meaning "the actions taken can only be explain if it is assumed that...", which results in a rationality paradox. Plainly, not all societies are fully rational. Some even make it a point to reject rationality, but most pretend to follow some
sort of reasoning w.r.t. why "they" act the way they do.
And finally, what does it mean to be rational? Disagreeing with me is irrational because I am entirely rational. So there. Bear in mind that I don't dispute that there is such a thing as rationality, I claim that there is massive disagreement over what it is: and everybody else is wrong.
Now as far as the "cancel each other out" part is concerned, the irrational forces are indeed trying to extinguish the rational forces and may well succeed. But to integrate the first query and answer it, societies behave as though there is a substantial irrational population and attempts to protect the rational portion, by any means necessary.
Prisons exist only because there are irrational members of society (who may or may not be politicians) and because there are rational politicians. The alternative is for every man to defend his interests by shooting (or otherwise destroying) threats. Every man then has to define what constitutes a threat to his interests. Actually, even wild animals have something resembling "societies" that regulate behavior.
There is no necessary relation between this "principle" and the humaneness of prisons. Rocks are naturally occurring objects, prisons are man-made. Rocks just happen, prisons are created (or not) by conscious human action. Excluding certain neural shortcut reactions (the Babinsky response, pupil dilation etc) humans chose their actions, and humans have free will. To answer the question, you have to first develop a theory of law, crime and punishment.
