Speaking at the Primary Care Live Event in London last week, Sir Michael said measures to tackle health inequalities needed to look at the overall health of the population.
Increases over the past decade in life expectancy for all groups, including the worst off, were a 'major societal achievement', he said.
'We should have two aims,' he said. 'One, to improve health for everybody, including the worst off. The second is to narrow the gap. We've done the one but not the other.'
He added: 'We should be looking not only at the poor health of the poor, but also at the social gradient of health.'
Sir Michael said looking solely at the gap in health outcomes between rich and poor had led to the 'rumour' that outcomes had not improved for the worst off.
But he pointed out that from 1995 to 2005, life expectancy rose by 2.9 years for the poorest members of UK society.
'The idea that it is all hopeless, that these inequalities are so deeply entrenched that there is nothing we can do is actually contradicted by the evidence,' he said.
This rise was not as great as in other groups and overall average life expectancy rose by 3.1 years, so the life expectancy gap did not close.
But he said the figures showed 'everyone can get better'. The potential savings of improving public health for all were enormous, while the cost of doing nothing was 'simply unsupportable', he added.
Sir Michael argued that plans to raise the retirement age to 68 would not work without improved health outcomes for all.
'About three quarters of the population do not have disability-free life expectancy above 68,' he said. 'If everybody with disability was on disability benefits, the effect of prolonging working age to 68 would be to shift people from pensions onto disability benefits.'