So this evening I finally took one for a spin; the wife and I went to the dealership (she wasn't exactly thrilled to spend friday night at a car dealership, but hey, she went). It's a joint Mazda/Ford lot, and as soon as I pulled up, three guys came out to say hello; we were the only customers.
They only had two MS3s on the lot, both GTs, one with nav. As I poked around one and asked some questions the guy couldn't answer, he got another guy, who knew slightly more. Best part was, he let me take it for a drive, about 2 miles.
The exhaust was louder than I expected, but wasn't unpleasant sound (not sure if I would feel the same 2 hrs into a drive). The gearbox wasn't as bad as most reviews would lead me to believe, but I would want to shorten the throw, and yeah, compared to my S, it's not the same - but it's not horrible by far. The first time I did a hard acceleration, I wasn't all that impressed. yes, it's fast, and has more torque than my S (but what doesn't), but it didn't seem to be frenetic. Then I realized I hadn't put my foot to the floor, and tried again. Whoa, much faster, much more torque. I ended up doing that a few times.
The handling was...ok. There was some body roll, which I assume could be addressed with swaybars, but it really wasn't that bad. I mean, I am comparing it to my S, which isn't exactly the same. What I was kind of surprised with was when we drove over some railroad tracks at about 40mph, the amount of float seemed significant to me. I'm assuming coilovers might solve that.
I totally forgot to try out the radio, but all the buttons/knobs on the dash seemed well laid out, and the seats were reasonably comfortable, and the rear seats seemed to have ample space too.
Overall, it seems like a solid car (no creaks or rattles, but it only had 22 miles on it), and could fit my bill, assuming I can find a good deal.

Found out there's four at local dealership, and two farther away; unfortunately they're all red or black, and I was hoping for blue, white and silver, but might settle for red if the price is right.
I'll sleep on it, and go from there (probably another test drive), and I'll be sure to keep you all updated. Thanks again for all the input. Looks like there's a number of people who have also owned an S, which helps me understand the switch better.