Researchers writing in the BMJ said although telehealth reduced demands on NHS services, equipment costs outweighed savings and was not an effective use of NHS funds at current prices.
At 2009 equipment costs, the cost per quality-adjusted life year (QALY) was £92,000 - three times the level considered by NICE to be cost-effective.
Even if equipment costs fall dramatically in the coming years and a large-scale roll-out of the devices takes place, there was only a 61% chance of it being a cost effective option, the researchers found.
The DH is currently embarking on a mass roll-out of the technology under its Three Million Lives campaign.
In March 2012, then care services minister Paul Burstow said telecare and telehealth devices could save the NHS up to £1.2bn over five years.
The study looked at findings from the Whole System Demonstrator (WSD) project, a DH-funded trial of telehealth believed to be the largest of its kind in the world to date.
The trial examined the benefits of the devices in 3,230 people with COPD, heart failure or diabetes from 179 GP practices in three areas of England.
An initial analysis published in 2012 found that using telehealth devices to monitor patients' health cut hospital admissions by 18% and deaths by 46%.
In this latest study, 1,573 patients taking part in the wider trial were given questionnaires to report their experiences and outcomes.
A total of 845 patients received telehealth devices and 728 had usual care. The devices included weigh scales, pulse oximeters, BP cuffs and glucometers, with data sent via a set-top box plugged into a television.
After 12 months 599 patients - 38% of those taking part - had dropped out of the study.
Researchers calculated costs of equipment, support such as monitoring staff, installation and maintenance. They analysed this against the cost of NHS services used by participants and the self-reported health outcomes of the care using the EQ-5D measure.
They estimated the average cost of telehealth equipment and support at £1,847 per patient per year at 2009 prices.
Telehealth users accessed fewer NHS services such as visits to GP practices, emergency hospital admissions and physiotherapy than those receiving standard care.
Although this led to lower NHS costs in the telehealth group, the addition of the telehealth equipment costs meant that costs were higher than standard care overall.
Researchers calculated that telehealth would cost £92,000 per QALY gained over usual care based on 2009 equipment prices and support costs.
They said there was only a 11% chance telehealth would be cost effective based on NICE's 'willingness to pay' threshold of £30,000 year.
Researchers conceded that equipment costs may fall over time. If costs fell by 80%, the cost per QALY fell to £52,000.
But even assuming lower equipment costs and a far larger roll-out of the system, there was only a 61% chance that telehealth would be cost effective at the £30,000 per QALY threshold used by NICE.
Researchers said the timeframe of the study may have been too short to show improvements in health-related quality of life.
Nevertheless, they concluded that telehealth 'does not seem to be a cost effective addition to standard support and treatment'.
They added: 'If investment in telehealth falls mainly to primary and social care purchasers, while most savings accrue to the acute sector - for which there is some weak evidence here - then reinvestment into community health and social care services would be vital.'
The DH recently confirmed plans to introduce a DES for remote monitoring for patients with long-term conditions in the 2013/14 GP contract.
A DH spokeswoman said: 'This is only one part of a much wider study carried out between 2008 and 2010. The whole study showed that using telehealth reduces mortality by 45percent, A&E attendances by 15% and emergency admissions by 20%.
'This part of the study confirms that to introduce the technology in isolation, at high cost and in low numbers does not bring the cost reductions we believe are there to be made. That is why our approach, known as the 3millionlives initiative, is different. It is about bringing in telehealth at scale and will create improvements in services, care and costs.'