If you want AWD go with a Speed6 (my bias opinion)...It really just depends on what you are looking for. I own a Speed6 and I was able to test drive the Speed3 a few times and I have driven a variety of STI's and EVO's. IMHO, the Speed3 has a better interior than the STI or Evo (again, just my opinion), however, if you are going for looks...the STI or Evo would definitely win if youre not a fan of the hatchbacks. Or if you just love the import racing look with the giant wing.
Yes, the AWD system in the Subi's rock. Its a given. The Speed3 does have a lot of torque steer to it, but I think its manageable once you get used to the car and the Speed3 wont give you that pushed into your chair (as much) feeling like the STI would just because the STI is AWD.
The Speed3 has a bit less horse (264 BHP) whereas an 04 STI is about 300 BHP. However, being the STI is AWD you do risk losing more power to the wheels which could even it out a bit -- never seen both on a dyno so I honestly couldn't say for sure...both are also torque-y as hell. Like many have said previously, with a used STI you run the risk of the previous owner beating the daylights out of the car and then selling it before it breaks. So that would be another pro for the Speed3. The Speed3 is also more of a sleeper, no big wing or giant hoodscoop. The stock STI pushes about 14.5lbs of boost -- I have yet to learn how much the Speed3 pushes but I'm sure its similar (my Speed6 pushes 15.6).
Not sure if any of this is meaningful. My best suggestion would be to find a Subi you like and test drive it and then test drive the Speed3. Everyone has their own preference and opinion...and really, both are good choices..but naturally I'm a Mazda fan

If you do go used be sure to get the vehicle checked out thoroughly.
Good night and good morning everyone!