(LA Times) — The race is on to tap one of the most vital sources of campaign cash — Wall Street — and the early results are not looking good for President Obama.

The president’s campaign struggled this week to sell out a fundraising dinner Friday at Manhattan’s gilded Four Seasons restaurant despite its being hosted by America’s No. 1 capitalist, Warren Buffett, according to people close to the campaign who were not authorized to speak publicly. The dinner for 100 was also a relative bargain at $10,000 a plate; recent fundraisers in Hollywood and New York have gone for $35,800 a pop.

The episode highlights a worrying trend for the Obama campaign. Wall Street, a key contributor to Obama in 2008, seems to be switching allegiances.

“His record has been one of reform and that has been an uncomfortable process for some of the major sources of political cash,” said Sheila Krumholz, executive director of the Center for Responsive Politics, which tracks political fundraising.

Or as one big-ticket Wall Street fundraiser for Obama put it: “It’s more difficult this time around.”

The race is far from over, but Obama’s difficulties speak to his contentious relationship with the financial community. After raising $43 million from the industry in 2008, he has spent much of his first term railing against Wall Street’s excesses. Pushing for financial reform, he made a reference to “fat cat bankers” that still has executives smarting.

Obama has made three campaign swings through Manhattan in the past few months, raising money from financial industry leaders on each visit. He wants the banking industry’s support. The question is whether the bankers still want him.