Brown: Still time for GB Olympic team
A British team will feature at the London Games for the first time since 1960 but there will be no players from Scotland, Wales or Northern Ireland.
"There's still time for them to reconsider. It would be far better if it was the whole UK," Scotsman Brown told Sky news ahead of a Cabinet meeting on the Olympic site in east London.
"You can understand, and I do understand, their sensitivities ... but this is very special, 2012, it has come to London after many years."
The Scottish, Welsh and Northern Irish football associations are opposed to taking part in a joint team, fearful their independent status in future tournaments would be prejudiced, despite assurances that would not be the case.
The British government helped push through the all-England compromise between the soccer associations, which was backed by FIFA president Sepp Blatter.
"...I worked very hard to get Mr Blatter and the FIFA people to say it would not affect ... Scotland, Wales and Northern Ireland's position in the international status that they had, but we were unable to get them on board for this," Brown said.