I've never known them to be intrusive. Once I went outside asked that they move along because I had sleeping children and a FIL who just came from the hospital (car accident) to recuperate at my house. I didn't give them reasons, I just shooed them away kind of like you do stray dogs.
I've been caroling many times and enjoyed it every time. Usually I enjoy hearing them in the front lawn too.
I have never been aware of carolers acting intrusive. Perhaps you should clarify your question.
I have only been caroling once, and that was when so directed by one of those religion peddlers. I did not continue membership with that group, and the group soon fell apart because of lousy leadership.