Tales abound of Muslim friends and relations, especially in the New York area, who packed up and returned to their homelands in 2001. Other stories, of Muslims being detained, never to return, are harder to substantiate but no less vivid on the rumor circuit.
Then there are the day-to-day hurts. Not long after Sept. 11, Tooba Ahmed, then a University of Maryland student, decided for the first time to "begin covering" - wearing a hijab - as an affirmation of her faith. In chemistry class, she found the seats around her suddenly empty.
"It was a very different feeling," she says.
If tensions have eased in the past five years - and incidents are fewer and further between - it may be because of the initiative of Muslim-Americans like Jameel, Ansari and Bendebba, who work to counter the Muslim tendency toward self-effacement. "It is crucial to build bridges," Jameel says.
To that end, he speaks at churches, sharing Muslim perspectives. Ansari helps organize dinners, inviting the wider community. Bendebba, ISB President Umar Mustafa and others encourage members to take part in the U.S. political system. Just last month, ISB held two "Meet the Candidates" sessions that brought scores of aspirants to the center to answer questions. Jameel, Ansari and others have given their names and cell numbers to FBI and police officials in case translators - or ambassadors - are needed.
It's still an uphill battle. When Israel's war with Hezbollah began in July, a local TV reporter visited the ISB for reaction. Ansari was disappointed at the first (and only) question: "Do any members of [the terrorist group] Hamas belong here?"
"I mean no disrespect," he says patiently, "but how do you answer a question that shows such a gap in awareness? Hamas? To us, they are what the Ku Klux Klan would be to Christians. Why even ask us? If people like that were here, we'd have handed them over a long time ago. ... We don't want that kind of trouble any more than you do."
As years continue to pass and Sept. 11 recedes in memory - and given better communication from non-Muslims and those who practice the faith - the world outside the ISB's cyclone fence will come to see those inside it as they see themselves.
"Just look at the way we live," Abed Husain says. "We are true Americans. We are not very different from you."
9/11: Five years later