Just to clarify and expand on what you said, aluminum is actually stiffer than steel in general, pound for pound. But to get the same torsional stiffness with an aluminum tube than a steel tube, the wall thickness needs to be considerably higher.
But it will still be lighter. Fatigue is the next major factor, aluminum doesn't like to be bent, and will crack and fail with repeated flexing much, much faster than steel will.
That's why nobody's ever been dumb enough to make an aluminum spring. But it's very good for relatively static parts like frame sections and body panels, you can build a stiffer part that's lighter than a steel equivalent of the same strength, you just don't want it to move very much, very often because it weakens much faster.
The aircraft industry figured this out long before the auto industry did. Yes, the wings you see flexing on a bumpy flight are mostly aluminum, but there's a steel alloy tube (chrome-moly) tube called a wing spar that spans the length of wing, and limits the amount of flex that the aluminum has to endure.
More exotic cars that make extensive use of aluminum still support high load areas with steel. Both metals are very well suited for certain things, but in many cases, a stamped or forged steel part will be stronger and sometimes lighter lighter than an aluminum equivalent. BMW and Audi in particular are struggling with this - despite their extensive use of aluminum in critical areas, they're finding that it cracks and bends a lot easier. Many, many BMW Z8's have bent frames, and Audi has a history of cracked aluminum control arms.