GPC negotiator Dr Peter Holden described the figures as 'outrageous'.
'If they spent £50 million a year they could have completely renovated one in eight surgeries,' he said.
The NAO figures show that the under spend has grown over time. In 2004/5, PCTs spent £370 million out of £386 million that had been allocated for building work.
But in 2006/7, they spent only £437 million of an allocation of £550 million - leaving more than 20 per cent of the cash unspent.
David Stout, director of the PCT Network, which is part of the NHS Confederation, said that the underspend resulted from local factors.
'It will be PCTs balancing their books and making local decisions about the most appropriate investment decisions,' he explained.
Mr Stout added that the under spend on premises was a tiny share of funding for GP services as a whole, which stood at £7.8 billion in 2006/7.
PCTs spent far more than their allocation on other aspects of the new GMS contract.
For example, between 2004 and 2007, they spent £384 million more than their budgets on the quality framework.
The unspent premises money is for maintenance and renovation rather than new buildings. Mr Stout noted that many of the worst surgeries 'probably need replacing rather than tarting up. Therefore this particular type of investment probably wouldn't help'.
He also said that there was nothing to stop GPs from using practice income to replace rundown surgeries.