Here is an infographic citing the reasons for banning and challenges:
Often challenges are made for "language" and for sexual content. But note the "not suited for the age group" column. Frequently, the challenge has more to do with "this is something kids can't deal with."
More charts are available here: http://www.ala.org/advocacy/bbooks/frequentlychallengedbooks/statistics
You can see who is responsible for most of the challenges (parents) and which institutions are the hardest hit by challenges (school libraries).
The best defense is a strong offense. Know the facts.