There seems to be a confusion between four different issues here: environment, sustainability, health, and taste.

The last is just that - for those of us who love the taste of meat and dairy it's a pointless exercize.

Health - not really my main concern, and just like you can put together a veggie diet that is healthy, you can do the same for a non-veggie one.  

environment - like so much else, both animal and non-animal products can be produced in a more or less environmentally friendly way.

sustainability - that's the big one. But no, I don't feel particularly guilty.

My philosophy is to make those sustainability/environmental choices which are cause less inconvenience to me or, ideally, which I actually prefer, rather than make huge sacrifices (in the subjective sense). So I have no car, I live in a densely packed city, relying on walking and mass transit exclusively, and I tend to eat local organic stuff to the extent I can afford to. And I suspect that's what most of us do.