Because I can't stand having my nose and mouth covered at the same time. I feel as though I'm suffocating and after breathing in my own carbon dioxide for a while, I get to feeling woozy.
I did at first, and I felt like I couldn't breathe even with a few minutes of wearing one. The sensation lasted even after I removed the mask. However, after a few wearings, my body adjusted and now it's fine.
That's was a couple of others have said, but I guess I don't wear it often enough to have gotten used to it. I'm in the house most of the day as I'm working remotely. When I do go out, I don't wear it in my car - only when I go into a store and if I'm sitting outside on the benches in front of my building, I don't wear it unless there's someone else going to sit on the bench which is almost never.