Let me just try once more, since this is crucial to understanding whatever it is linguistics is.
Yes, until this last sentence. There is no inventory of sounds in the phonemic layer. The phonemic layer is composed entirely of mental categories. Sounds are the other side of the "coin", and categories and sounds associate to produce "signs", but there is a firewall of sorts between them.
What Saussure is accomplishing here is very significant in the history of thought. He is arguing that material things ("sounds") and ideational things ("categories") can travel together without being somehow of the same sort. He's demonstrating that language can be understood as formal system.
FROM A SEMANTIC PERSPECTIVE:
0. The "sign" is a unit with two sides:
1. signified is the material thing in the world that (2) points to
2. signifier is the ideational thing: the language used to point to (1)
FROM A SPEECH SOUND PERSPECTIVE:
2a. The signifier itself is a kind of "sign" with two sides of its own:
2b. signified is the material thing: phonetic sequence [sound]
2c. signifier is the ideational thing: phonemes & distinctive features [category]
Sorry if I am pushing it, but I just want to make sure I'm not "mixing levels" of description here. What do you think of this understanding?