There are lots of reasons that people bring up. The answer varies from person to person and usually includes one or more of the following:
Diesel engines are perceived to be noisy and stinky in the US.
Diesel engines are perceived to be bad for the environment, so a lot of people who are willing to pay more for a car that uses less fuel seem to prefer hybrids.
Diesel is up to 15% more expensive than 87 octane gas (which most cars use), so the payback is very long since most diesels cost more at initial purchase.
Due to some bad engineering during the late 70s and early 80s, a lot of older people think diesel engines are unreliable.
Diesel car emissions have to be as clean as gasoline, which killed the plans of some car companies, like Honda and Subaru to bring diesels here.
The emissions requirements have lowered the mileage advantage that diesels used to have over gasoline.
Another reason is that they aren't offered in many cars that are affordable. The TDi cars from VW are the only affordable diesel cars that I can think of and VW doesn't have a reputation for reliability here either. Other than that, BMW, Mercedes and Audi seem to be the majority of passenger vehicles (besides pickups) that you can get a diesel in.
So, in my opinion, consumer ignorance is the main reason.
Edit: According to some people, we also don't like wagons!