I think you've pinpointed one of the difficulties of using Allport in some situations: It is far easier to focus on and discuss extreme forms of bigotry than to deal with subtle forms of bigotry that are more likely to characterise daily interactions.

Banter is fun, but I'm not sure it is harmless when it focuses on stereotyped differences. I think there is a difference between banter among people of two different, but interacting groups, who are working together, versus two groups that see each other as competing or of markedly different status, versus people who are joking solely within their own group.  

The first type of banter may be acceptable when there is a degree of balance between the groups, and both are comfortable challenging each other and the banter is not damaging to their working relationship, e.g. neither side has to swallow resentment. The second type of banter may be expected, but isn't necessarily good (athletic challenges descending into riots come to mind). The third type of banter may seem harmless, but I think it can slide over into acceptance of a prejudiced view that will affect relationships outside the group, e.g. reinforcing stereotypes.

I do agree, however, that expansion of prejudices into wiping out ethnic groups is not automatic, nor simple. It requires charismatic, coercive, and educational actions on a wide public scale over a period of time to build to the point that such actions become acceptable.