The National Audit Office (NAO) said the quality of service at HMRC “collapsed” over an 18-month period between 2014 and 2015.
Call waiting times tripled during that time, as some customers were kept on hold for up to an hour.
In response, HMRC said most calls were now being answered in just six minutes.
As part of its most recent study, the NAO worked out how much money callers would have notionally lost, while waiting for a reply.
Using HMRC’s own criteria, it valued people’s time at an average of £17 an hour.
As a result it claimed callers would have wasted the following sums:
£66m while waiting on the phone
£21m while actually talking to HMRC
£10m on the cost of the call
The NAO blamed HMRC’s poor performance on its decision to cut 11,000 staff between 2010 and 2014.
As part of its strategy to persuade people to do their tax returns online, it had anticipated needing fewer employees to answer the phone.
But after call waiting times for self-assessment tax returns peaked at 47 minutes last autumn, HMRC was forced to bring in 2,400 staff to its tax helpline.
Amyas Morse, head of the National Audit Office, said he accepted that HMRC’s overall plan did make sense.
“This does not change the fact that they got their timing badly wrong in 2014, letting significant numbers of call handling staff go before their new approach was working reliably,” he said.
Citizens Advice said some people might fall into debt as a result of the problems – if they missed a tax deadline as a result.
“Long waiting times not only cause frustration and increase the cost of the call, but can also mean people miss important deadlines,” said Gillian Guy, the chief executive of Citizens Advice.
“For example if you don’t return your tax form on time you face a fine – which for some households can be an additional cost they can’t afford to pay.”
HMRC said its service levels had improved since the period in question. Over the last six months it said call waiting times had averaged six minutes.
“We recognise that early in 2015 we didn’t provide the standard of service that people are entitled to expect and we apologised at the time,” said Ruth Owen, HMRC’s director general for customer services.
We have since fully recovered and are now offering our best service levels in years,” she said.
However the NAO said it was also concerned that, because many taxpayers never got through to HMRC on the phone, they may have paid the wrong amount of tax.
In March this year there were 3.2m outstanding high priority cases that still required investigation.
MPs on the Public Accounts Committee will take further evidence on the issue on 13 June.