Have you thought about adding a resonator? I am no exhaust expert by any means but it might be enough to quiet it down for you. I would give that a shot. Or we could try a swap, my stock manifold and cat for your header and mid pipe. I would think they would both give decent sound and not be too loud. It might be a good idea to try it out and decide from there. It would be a lot of work though.
Putting the stock header will make a huge difference. I'd say that's the way to go.