The defeat of the compromise means that lawmakers will return to the Statehouse today to consider either an outright ban on gay marriage or letting the state's constitution remain intact.
The joint House and Senate session adjourned for the evening about 8:30 p.m. after about six hours of debate.
The bipartisan proposal was crafted by Senate leaders who wished to overturn a high court decision legalizing gay marriage while still extending equal benefits to gay couples. It was rejected 104-94.
The compromise would have made Vermont-style civil unions automatically legal in Massachusetts in November 2006, the earliest an amendment could be placed on a ballot for voter approval. At that time, any gay couples married in Massachusetts would be stripped of their licenses and considered part of a civil union.
Massachusetts was thrust into the epicenter of the national gay marriage debate in November when the Supreme Judicial Court ruled 4-3 that that it was unconstitutional to bar same-sex couples from marriage.
The court definitively reaffirmed the decision last week, clearing the way for the first state-recognized gay marriages in U.S. history to start taking place in May.
Rep. Philip Travis, a Democrat who sponsored the original ban on gay marriage, said the compromise proposal would have asked voters to decide on two potentially conflicting initiatives - a ban on gay marriage and the legalization of civil unions - with one vote.
"It goes beyond what the people wish to vote on," Travis said. "It is almost like offering a true-false question. How do you respond by voting yes or no at the ballot box?"
"I think at this point if they did not endorse the amendment or if they backed off or whatever, they would have a tremendous problem among their most loyal supporters because of the assurances we've been given," said Gary Bauer, president of a pro-amendment organization called American Values.
White House press secretary Scott McClellan said the administration was closely watching events in Massachusetts.
"The president has said that he is committed to doing what is legally needed to protect the sanctity of marriage," McClellan said. "And he has said, if necessary, he would support a constitutional amendment. That is what he has previously said. But at this point, we continue to look very closely at this issue. Obviously, if there are any updates, I will keep you posted."
Impromptu rallies erupted outside the 2-century-old building, and hundreds of people lined both sides of the street, holding signs, waving flags and eliciting honks from passing cars.
Chants broke out spontaneously in pockets up and down the street, and included, "Hey hey, ho ho, homophobia's got to go." Others chanted, "Adam and Eve, not Adam and Steve," or held signs that read "Civil Unions Sitting on the Back of the Bus."
State police had to separate two crowds inside the Statehouse who began pushing and shoving after one group unfurled an American flag across from the chamber and began chanting "One Man. One Woman. Let the People Vote." Moments later, a contingent of gay-marriage backers arrived shouting, "Equality Now."