Maybe something is leaking in the brake booster? But I'm no expert by any means...
A leaky brake booster will make the pedal hard, because engine vac isn't helping you press the brakes.
I had a similar spongy pedal feel that related back to a master cylinder replacement and not doing a good job bench-bleeding, maybe this is part of it?
A seized caliper would explain the pulling, have you checked brake/wheel temps after driving it hard? When a caliper started dragging on mine the wheel was quite hot to the touch. If you notice that one wheel is significantly colder or hotter than the others it would help pinpoint the issue.