Born in Baltimore and raised on Belgian Avenue in Pen Lucy, he attended Blessed Sacrament School and was a 1944 graduate of Loyola High School, where he played football.
"He was a good athlete. He ran fast and he was durable," said a longtime friend and retired attorney, James O'Conor Gentry of Timonium. "You could always depend on him for help."
He then entered the Army and was sent to Officer Training School. A second lieutenant, he was assigned to the 24th Calvary Reconnaissance Squadron and commanded a five-tank platoon. He was stationed in Austria at the end of World War II as part of the occupation forces.
Family members said that after leaving the military, he used the GI Bill to attend Washington College, where he played on the school's football team and graduated in 1949. He then received a degree from the University of Maryland School of Law.
Mr. Brown became an assistant state's attorney in the office headed by Anselm Sodaro, a friend. He tried cases there from 1953 to 1956. He was later general counsel to the Maryland Department of Employment Security and in 1960 and 1961 was a Baltimore City Traffic Court magistrate.
In July 1961, Gov. J. Millard Tawes appointed him to fill a vacancy in the House of Delegates created when Francis X. Gallagher resigned to become people's council for the Public Service Commission. He served a year.
Active in Democratic politics in the old 4th District, Mr. Brown ran unsuccessfully for the U.S. House of Representatives against incumbent George H. Fallon in a 1962 primary election.
In 1963, Mr. Brown joined the Social Security Administration and was sent to a post in Louisville, Ky. He returned to Baltimore as an administrative law judge. He also served as deputy chief judge in Arlington, Va., from 1980 to 1986 and chief judge of his agency's Baltimore office from 1986 to 1990. He then became senior judge in 1990 and retired at 83 in 2010.
"As a judge, he was thorough-minded and fair," said Mr. Gentry. "He was well regarded in the Social Security system and certainly stayed a long time."
Family members said that Mr. Brown acquired a small rental unit in Ocean City more than 50 years ago. He named it the Ocean Jay after his first son. The family vacationed there and often allowed friends to use its second apartment.
In March 1962, a large storm swept the Atlantic Coast and washed the Ocean Jay out to sea.
"There was no flood insurance, and he lost everything as a young father," said his son, Neal M. Brown of Hunt Valley. "Even though he had nothing left — even the land disappeared — he kept up paying the mortgage until it was paid off."
Mr. Brown resolved to rebuild at Ocean City. He acquired another place, which he called the Challenge, to describe the process of rebuilding. He also built another home, the Christopher.
Friends recalled he enjoyed weekly games of poker. Among the players were two friends who were former Baltimore mayors, J. Harold Grady and Thomas J. D'Alesandro III.
Friends said Mr. Brown was an intrepid traveler. He and his wife traveled the world, including trips to Uzbekistan and Northern Africa and multiple trips to Ireland.
Mr. Brown was a supporter of the Mother Seton Academy on Greenmount Avenue, adjacent to St. Ann's Church, where he had married. He was also a Viva House benefactor.
A Mass of Christian burial will be offered at 10 a.m. Thursday at St. Joseph Roman Catholic Church, 100 Church Lane in Cockeysville.
In addition to his son, survivors include his wife of 58 years, the former Mildred Mullan; three other sons, J. Robert Brown Jr. of Denver, Jeffrey P. Brown of Saratoga Springs, N.Y., and Christopher L. Brown of Ellicott City; three daughters, M. Noreen Brown of Portland, Ore., Linda A. Thomas of Ridgely, W.Va., and Mary Carole Harrison of the United Arab Emirates; a brother, George Eugene Brown of Baltimore; two sisters, Sister Noreen Brown of the School Sisters of Notre Dame of Baltimore and Anita Ross of Fallbrook, Calif.; and 21 grandchildren.