Could a voiceless obstruent (like "t") function as a vowel? Hypothetically, sure. For a number of well-understood reasons, however, it's not very likely.
I'm having a very hard time following your interpretations of my statements (i.e., no, this is definitely not just a convenient example of how weird languages are in general), and likewise I'm at a bit of a loss for how to proceed. I suspect that we are operating under some very different understandings about what phonology is.
Specifically, the idea of a "universal deterministic system" is deeply antithetical to virtually all work done in phonology over the last century. That's an extremely important point. In terms of pure phonology, categories construct a closed system. There is no equivalence across languages. Asking what an English /r/ has in common with a Hindi /r/ is not meaningful except as shorthand for a very different question about perceptual typology. If we're talking about perceptual typology, of course the category is not deterministic. How on earth could it be?
What the term "rhotics" describes is a cross-linguistic pattern in the relationship between a cluster of phonetic natural classes (roughly, non-lateral approximants) and a bunch of language-specific phonological categories. What's interesting about rhotics is that in many languages the phonological category extends (sometimes significantly) beyond the boundaries of the phonetic natural class. That fact of extension is itself motivated by the articulatory and perceptual properties of approximants. The typological generalization is that non-lateral approximants and other sounds treated as equivalent to them by speakers of a particular language frequently have distinctive distributions within syllables.
What's "fuzzy", then, is not the border in any sense, but rather the degree of isomorphism between language-specific phonological categories, on the one hand, and natural phonetic classes, on the other. In those terms, hunting for the exact border that separates a rhotic from a non-rhotic is missing the point entirely. It's like arguing that the term "vowel" is just handwaving because we don't know for sure whether or not "l" counts as one. That's just not how it works.
In that regard, rhotics are pretty middle-of-the-pack conventional. They're more typologically irregular than some things, but less typologically irregular than others. Far from being unusual, this is what the bulk of phonology looks like once you get beyond the trivial stuff that shows up in problem sets. The notion of rhotics is no more and no less coherent than five dozen other different words we use in typological comparisons: "tone", "sonorance", "vowel", "syllable", "stress", etc. These things are all generalizations with significantly different organizations in different languages.