Agree. If the terrorists don't shout it when attacking and mobs don't shout it right after "Death to America, Death to the Infidel" it wouldn't be objectionable. If it was then used by friendly, peaceful Islamic leaders and was translated in the news as "God is Great", then it hints at far more common ground for cooperation.
As it is used by the terrorists, it becomes a clear battle cry and a divider, differentiating their god from the God of Christianity, Judaism, etc. Many in the West have come to link the phrase (with some basis) with Islamic elements that hate Christianity and/or the West.
A related phrase used in much of Islam is "As-Salaam-Alaikum" (Peace be Unto You), which is very similar to the "God's peace" observed in some Christian services) and similar to the Jewish Shalom "Peace" (with the sometimes implication that God is the source of true peace). I've heard mixed things on the etiquette of it being said to non-Muslims (or non-Muslims saying it), but I've used it myself to Muslims on the assumption it's like saying "God's peace" -- I believe my God and theirs are the same, they just interpret him quite differently.
Another Arabic/Islamic phrase with strong connotations to some in the West is "In Sha' Allah" -- God willing. Taken literally, it can simply demonstrate a faithful person's acknowledgement that God is the ruler of all and will dictate to what happens. However, have spent time working with locals in Afghanistan, Saudi Arabia, and visiting some of the gulf states, it was seen as something quite different. Instead of being a faithful acknowledgement of God's will, it was often used (and/or perceived) as a lazy person's out for not working too hard. On more than one occasion, I was negotiating work with locals and they'd say, "We'll be done on Wednesday, In Sha' Allah." What this really meant in practice (in the numerous cases I experienced) was, "It'll be done on Wednesday if we feel like it and don't decide to take tomorrow off or work on something else." I perceived it as them basically using God as a cover for their poor work or laziness. I'm sure it's often used with pure intent and true meaning, but that wasn't what I encountered.
"Allāhu akbar" simply translates as "God is great". just butterfly terrorist movements using it and it'll be more positively considered. iirc, it comes up in Islamic prayer as well, so you may well consider it as simply a Muslim equivalent to "Amen." so one way it could be used is simply in agreement where, to give an internally fictional example, there's a scene with a rousing speech by the main character and his allies all cheer and one yells out "Allāhu akbar!" rather than "Amen!"