You're forgetting the nitrogen-fixing pulses: peas, beans, lentils, etc, that should be part of human nutrition even for non-vegetarians.

For our local food co-op, I deal with two organic farmers. One provides us with wheat flour, rapeseed and sunflower oil: he has beef cattle to feed manure into the system. His neighbours don't have animals, and put chick peas and lentils (that we buy from them) into the crop rotation.