I'm trying to make the same decision. I haven't driven either of them yet, but each one has their pros and cons on paper.
The Matrix is only 3 inches taller than the P5, so there isn't that much difference in height.
What I've found...
Pros for the Matrix base/XR - Toyota reliability, gas...