i would think it would need to be a fairly hard hit to do major damage. that being said, obviously your wheel is not meant to take impact in that direction, so there will be some sort of detriment, but probably one that you will notice. My experience tells me that even if you hit at ~5 - 10 mph, worst case scenario is that you need a fresh allignment.
so, whats the story, did you hop a median or something?? lol