The president announced the creation of the new panel, called the Commission on Enhancing National Cybersecurity, last week as part of a proposed $19 billion package of measures to improve the nation's ability to withstand attacks from hackers.
Obama met with Palmisano and Tom Donilon, a former National Security Advisor who will lead the commission, in the Oval Office Wednesday afternoon.
"The Internet has brought incredible opportunity, incredible wealth, good access to data and information, that are enhancing our lives in all sorts of ways," the president told reporters. "It also means that more and more of our lives are being downloaded and stored ... and as a consequence we're a lot more vulnerable."
"We are not as well organized as we need to be to make sure we're dealing with all of these threats in an effective way," he said.
The panel will include businessmen and technical specialists and has been instructed to report back to the president by December, the White House said.
Palmisano graduated from the Johns Hopkins University and had a lengthy career at IBM starting in 1973. He served as the company's CEO from 2003 to 2011, at a time when it was moving away from selling computers to consumers and focusing more on providing services to other businesses.
While the president plans to make a major cyber security push in his final year in office, the government and the private sector have clashed over the best way to keep Americans' information safe.
The announcement of the two appointments came as what could be a fierce battle over computer security is brewing between Apple and the FBI.
A federal judge ordered the company to help investigators unlock a phone linked to one of the two attackers in December's shooting rampage in San Bernardino, Calif. But Apple CEO Tim Cook has vowed to resist, saying providing assistance will undermine his customers' security.
At the Oval Office meeting, Obama ignored a question about the face off.