LMCs carried a motion calling for GPs to take action because the NHS data spine - the planned national record database - was a 'major threat to civil liberties'.
Dr Preston de Mendonca, chairman of Devon LMC, said: 'The NHS smartcard opens the door to the details of 50 million NHS patients. There will be more than 50,000 cards in general practice.
'Any lock that can be opened by 50,000 keys is not secure. It will be possible to find all your ex-lovers' addresses.'
He said GPs could send a message to the nation about the poor security by withdrawing their own records and those of their families.
Connecting for Health GP clinical lead Dr Gillian Braunold said an audit trail would track smartcard users' access to records, so that inappropriate access would be detected.
She added that access to the records of famous people or those who had suffered abuse could be blocked.
'If you vote for this, you will disable the benefits of the system for patients,' Dr Braunold said.
But Dr de Mendonca pointed out: 'It doesn't matter that there's an audit trail - there will be millions of these footprints and you will not be seen. We are the guardians of patients' data and we will be feeding it into a big monster.'