That's a huge generalization (that I could have qualified a bit more). Certainly it's for some groups and not others, and I think/hope it is becoming less of a problem these days with better interaction.
I know little about this topic, but I've heard it brought up a few times as a problem for fieldworkers: the fact that you want to work with a group doesn't mean that would would trust you or allow you to work with them, and this is sometimes due to historical mistreatment.
I don't really need to outline the mistreatment of Native Americans by settlers and early (political) Americans going westward. It's just like any case of invasion/conquest: one group stealing from, slaughtering, displacing and giving disease to another. At the time of the arrival of the first settlers, there were around 300 different nations/peoples in the North American region. Not all were treated equally and not all responded equally. A small minority were fierce warriors who did in fact kill settlers and early Americans, certainly out of defense, and (being completely objective) probably also in a very small number of cases out of intended violence. (Before the Europeans arrived, the tribes were sometimes violent with each other. Europeans didn't introduce violence in the first place, but they did bring more powerful weapons, with many more people, and they were clearly the problem in any general sense.)
This all extended stereotypes and fears, resulting in the intended slaughter of more Native Americans because they were considered to be violent savages. (Of course other things were going on as well like the fact that they didn't adopt Jesus quickly enough! Right...)
[Edit: also look up "Manifest Destiny". It's one of those things from history that makes your head hurt because it's so absurd and evil yet was actually what people believed at the time.]
The problem with linguists is that they were often hired by the US government and part of the problem or even supporting it. They might have the relatively innocent goal of preserving the culture before it was destroyed (in which case the alternative was to NOT preserve it and still destroy it, given the political atmosphere and power dynamics that were in place), but clearly even that would be seen as someone stealing your language then, as a representative of the government, helping to destroy it. In other cases linguists may have been actively problematic by learning a language in order to convert the speakers to Christianity or to teach them 'proper' English instead of their language.
A major problem were boarding schools that were designed to eradicate languages: children were taken from their homes and placed in boarding schools but moved around so that the groups of children did not share a common language. They were forced to communicate in English. (Do a web search for "Indian boarding schools" for more information.)
I don't actually know how active the role of linguists was in this, but of course many people don't distinguish between a language teacher and a linguistic scientist. But in this case I don't know that it's even that: some linguists were truly ill-intentioned, supporting or even actively aiding in the destruction of culture and languages. Obviously, that's a bad thing and not still true (well, I hope!). And it all relates to other kinds of mistreatment and social problems.
However, there are major distinctions between the different groups. Some were affected much worse than others and still are mistrusting. Others are more open. There are a number of generous and responsible researchers studying Native American languages who are appreciated by the tribes that they work with, but it is delicate. And in fact there are a few cases (very important) of Native Americans themselves becoming linguists (even receiving a formal education in Linguistics) and returning to their home to do documentation and revitalization efforts. I think the revitalization efforts are probably most important for improving relationships.
That's about all I know. I'm sure it's discussed somewhere, but I don't have a particular source in mind. I believe the timeline was around 1850-1900 especially, but I'd have to look up those dates to be sure.