It depends on your needs.
I actually leave the cargo cover in place almost 90% of the time. Unless you need the 3rd row all the time, or your cargo goes up to the roof, I find it comes in handy to cover up what you have stored back there.
I agree, it's a tradeoff but IMO, I am very glad I got it.