A couple of comments:
1. The First Amendment issue is simple--the Constitution protects the building of the project at that site, whatever one thinks of the propriety of doing so.
The first amendment does not address the location of places of worship.
2. The project will not be built at Ground Zero, at the edge of Ground Zero, or within the sight of Ground Zero. It is two blocks away from the northeast edge, surrounded by other buildings.
Part of the landing gear of one of the planes that struck the towers, debris from the towers, and almost definitely parts of human remains struck the exact site of the mosque/Islamic Religious center to be. To try to call it other than Ground Zero is like saying "it was not Pearl harbor, it was Ford Island."
3. New York and New Yorkers haven't treated the area around Ground Zero as "hallowed ground" up to this point. There are one or more strip clubs and t-shirt vendors closer to the site. So? It is New York City. That's their taste in things.
4. The area has boarded up buildings, so the project actually would help the area economically. Do these boarded up buildings predate 9/11 or where they boarded up afterwards.
5. The project technically isn't a Mosque, but it does have a prayer center so that distinction probably isn't meaningful. One could argue that if there was a differentiation, then it would be meaningful. Unless the place was zoned for educational organizations. Still, it contains a mosque, and will be used as a mosque.
6. The area has included a Mosque since before the WTCs were built, so this isn't really the Muslim community moving in to declare victory.
When they get a $100 million Cordoba Islamic Center (and Mosque) while the Orthodox Church gets the bum's trush, you don't think that there will be Muslims that do not think of this as a victory? Even if the Greeks were allowed to rebuild, even if other faiths were to set up Cathedrals in the same zone, this will still be seen as a sign of Islamic Triumph by many of the extremists, and be disheartening to the moderate Muslims.
7. The asserted symbolism of the project has not been well received by Muslims. Those involved in the project are not required to take this into consideration, but they may want to do so. Right, back to the onging conflict between moderate and radical Muslims.
8. The project sponsors own one building needed for the project, and have the right to acquire the other. In that economically depressed section of Manhattan, they probably couldn't sell their rights for sufficient cash to build a project a few blocks further away that would serve the Muslim community already resident in the area. What to the opponents of the project think they should do about that? Crap. They have gotten $100 million from "private sources". They have enough cash to move the victory mosque elsewhere.
Oh, and if they do build it, they should be prepared for the city to exercise "eminent domain" once Bllomberg is voted out of office.