I think of it in the micro sense. We live most of our lives with our spouse and children. There is a certain way we do things at home. We can tolerate visitors for short visits. But the longer they stay, the more the differences in how they live start to gnaw at you. We can teach our children to conduct themselves in the way we think is correct. But we would take pause in telling guests how they should live. They may respect our boundaries for a while before taking liberties.
On that note, some guests who take too many liberties in our homes tend to not be invited to stay again.
If you scale this out, immigration is not the problem. The problem stems from the differences that cause friction in the conventions a community follows.
One anecdotal example. I live on the border with Mexico. There are some things that Mexican drivers do that are bothersome to local drivers. For example, they tend to drive slower (which I think is a result of living with the metric system 😜). They also tend to disregard turning lanes or make sudden or awkward lane changes that could result in accidents. They don't do this maliciously. But their way of driving is a problem for the rest of us who have come to expect drivers to respect turning lanes or be more considerate when changing lanes. They're not bad people. But they can be disruptive and inadvertently dangerous.
Are locals attached to the system? I don't think so. We disregard traffic rules often. But we disregard them in certain ways and not others, like speeding or rolling stops. So, it's not the system we're attached to. It's more like social conventions that allow us to get along. And when somebody new comes along who does not know or blatantly disregards those conventions, it's upsetting. We go through the trouble of following those conventions for the benefit of our communities. It's a courtesy. Discourtesy is frowned upon.
I think what happens is that those discourtesies add up until people just don't want to tolerate it anymore than they would allow a rude guest to stay in their home.