Yes, agree with you. This is exactly the reason I use manual sparingly for only extreme conditions.I use sport mode mostly when I need a little extra grunt when I want to pick up speed quicker. It holds the gears a little longer, thus keeping the revs up in a better power band.
I also have paddle shifters, and once the novelty wore off, I stopped using them. It takes a lot of concentration and constantly watching the rev counter in order to get it right. This unfortunately leads to distracted driving. It takes my attention away from the road. Also, many times I've forgotten to shift (up or down) and I end up either lugging the engine, or revving the crap out of it because I didn't up-shift. It's just too much work.
I do the polar opposite to scare the life out of them!I use Sport mode to piss off tailgaters that try to go around! >