Mr Swan said on Monday those getting a share of the $10.4 billion stimulus package should spend it to create jobs and bolster the economy.
But a survey published in News Ltd newspapers on Tuesday showed that almost half of respondents said they would use the money to pay off debt and put on their mortgage.
“Families will make their decisions in the interest of their households,” Mr Swan told reporters in Brisbane.
“I’ve talked to a lot of families, pensioners and parents and many of them have been doing it tough in recent times.
“So this cash will come at at a time when they can spend responsibly, support the economy and support Australian jobs.”
Mr Swan said a comment by Opposition Leader Malcolm Turnbull that the stimulus package was a “one-off sugar hit” showed all opposition support for the package had dried up.
“Mr Turnbull has rescinded any support for the economic security strategy,” Mr Swan said.
“It is regrettable that Mr Turnbull is so negative.
“The economic security strategy is vital to strengthen the economy and to support jobs.”