A study from the University of Cambridge found that while three quarters of patients are diagnosed within two visits, there are large variations in the promptness of GPs to diagnose some cancers.
Women, young people, and ethnic minorities were also more likely to face delays in referral for symptoms of cancer, researchers found.
A cancer charity said the study highlighted the 'challenges' of making appropriate referrals for such a relatively rare set of conditions.
Researchers called for better tools to help GPs calculate a patient's risk of cancer and to improve their ability to spot symptoms earlier.
Study authors said: 'Research and policy should explore and assess physician-level educational interventions, further development of point-of-care decision aids, risk calculators and diagnostic tests, and redesign of systems to enable more appropriate and timely use of specialist diagnostic tests, e.g. imaging or endoscopy.'
The study examined the records of 41,299 patients with 24 different cancers. Researchers found that 77% of patients who first presented to their GP with symptoms were likely to be referred to a specialist after just one or two consultations.
But some groups - including women, young people, non-white patients and those with some less common cancers - were more likely to have three or more visits before referral.
Patients with multiple myeloma, Hodgkin's lymphoma, and cancers of the pancreas, stomach, lung, colon, and ovaries were far more likely to see their GP three or more times before being referred.
Over half - 50.6% - of patients with multiple myeloma required multiple GP visits compared with just 7.4% of patients with breast cancer.
Researchers said the findings highlighted limits in current understanding of cancer symptoms for some conditions.
Lead author Georgios Lyratzopoulos from Cambridge University said: 'Medical research in recent decades has prioritised improving cancer treatments, but knowledge about the "symptom signature" of common cancers and practical solutions on how best to diagnose them is still emerging.'
Jessica Harris, health information manager at Cancer Research UK, said: 'This study highlights some of the difficulties in diagnosing cancer in primary care.
'A GP will see only around eight cases of cancer a year, on average, among hundreds of people with symptoms that might indicate cancer, so making appropriate referral decisions can be challenging, especially for rarer cancers or those with symptoms that are vague or common to other diseases.'
The findings agreed with a report from the RCGP in November last year. This also found that GPs investigate and refer three-quarters of cancer cases within two consultations.
GPs at practices involved in the study had blamed poor access to investigational tests and scans for delays diagnosing some cancers.