Buddhism isnt a passive religion, but it is just what the majority of lay buddhists would like to believe. In fact Buddhism is a religion that is very vigorous and challenging for the individual. Because being passive, avoiding and not retaliating is easier than doing so. And also because, most Buddhists would not like to lose out to christianity and its image as a "peaceful" religion.
No matter what, it is still a fact that many people become Buddhists because they wish to escape from the mainstream religion of christianity or that they want to be different, or because Buddhism is an exotic religion, or maybe perhaps they see buddhism as a less demanding religion with less commitments and less "superstition" compared to the religion they grew up in.
From all these, they start to push Buddhism in to fit their mould, to be something it is not. And thus came many misconceptions and sterotypes about Buddhism that are not true at all. Being passive and hippy-like is just one of the many. If Buddhism was a passive religion, it would not grow and stay stagnant. It is a religion that encourages us to take action against something after education.
The world can believe what they want, but it does not mean that they are right. For example, everyone else, except for a few believes that animals are not as intelligent as humans, and therefore it is all right to hurt them and breed them for food. This belief is obviously wrong even when the majority believes in it. So why is it that we have to follow people's preconception on Buddhism and maintain that when we know it is not true?