I'd like a little context for those sentences-- they don't immediately have exactly one meaning to me and might be ambiguous. But I'd need to imagine some contexts to know whether that really works out. They're low frequency, certainly!
My intuitions:
Lily whined to Marlene for Nedra to keep quiet
Lily complained to Marlene in order to make Nedra quiet.
Either Marlene would ask Nedra to be quiet (1), or Nedra would decide to be quiet because Lily was complaining (2).
"Lily complained to Marlene to get the volume turned down on the TV" (1)
"The student complained to the teacher in order to get her peer to stop cheating." (an indirect threat, 2)
Lily whined for Nedra to keep quiet
Lily expressed her desire, via whining, for Nedra to be quiet.
Possibly this is a complaint directed at Nedra, but I don't think it
must be.
Lily whined to Marlene to keep quiet
"Marlene, be quiet", Lily whined.
Or
"Lily whined to Marlene so that she [Lily] was distracted enough from speaking to the other group" ("The angry politician complained to his friend in order to avoid yelling in front of the voters.")
Lily whined to keep quiet
Awkward.
Maybe "Keep quiet", Lily whined. But awkward/choppy. Something seems to be missing, maybe in the right context-- 'They arrived at the haunted house, and Lily was scared. Turning to her friend, Lily whined to keep quiet... "I'm scared!"'
But this is more likely just:
Lily whined in order to stay quiet-- whining instead of speaking loudly. (Odd meaning, acceptable syntax.)
As for your readings:
1. Lily ordered Marlene to keep Nedra quiet
Possibly. Awkward. Probably what the original sentence was intended to convey.
2. Lily ordered someone to keep Nedra quiet
Maybe. But including Nedra herself. "I hope Nedra is quiet!!!", etc.
3. Lily ordered Marlene to keep people quiet
Unlikely. Marlene should be quiet here. "to keep quiet" isn't a general order for people in general. It's usually specific.
4. Lily ordered people to keep themselves quiet
Maybe. As I said above, this is awkward.
I hope that's somewhat helpful. These are weird sentences.