These areas of damage could increase the risk of cognitive impairment, dementia, depression and stroke, say the researchers.
For the study, 79 men and 153 women aged 60-86 years completed a food frequency questionnaire to determine calcium and vitamin D intake. Forty-one per cent of the participants had a diagnosis of depression.
The researchers took MRI scans of each participant's brain to identify and calculate the volume of any lesions.
All participants had some form of brain lesion, with volumes ranging from 0.7-61.8ml.
Calcium and vitamin D intake was significantly and positively correlated with brain lesion volume, delegates were told at the Experimental Biology conference in Washington this week.
These positive associations remained significant after multivariate analysis involving age, hypertension, diabetes, heart disease and any known depression.
Lead researcher Dr Martha Payne, from the department of psychiatry and behavioural sciences at Duke University in North Carolina, said: 'Since this was a cross-sectional study, we weren't able to assess risk.
'We did find that individuals with higher intakes tended to have greater lesion volume.'