This has become a somewhat inaccurate historical meme, but of course with some basis. The first British settlement in Australia was an expedition to build a military outpost not a penal colony. The plan was hurriedly conjured up to deny the French a strategic position in the Pacific. They knew from experience that to build a self sustaining outpost on the far side of the planet required large reserves of labour.
They used slaves.
African slavery had become unpopular. Transportation to the American Colonies had been a useful tool in law and order matters for nearly one and a half centuries as a way of softening the "bloody laws". It has been estimated that about 50,000 convicts were transported on one way tickets to the North American Colonies, especially New England. The Crown paid merchants to ship them privately. Some were sold as slaves on arrival. Those that weren't sold outright were still slaves, but were more or less characterised as indentured labour. Emancipation was supposed to be granted after 7 years. The OP is correct when he says "many of the early settlers were criminals of some sort". In North America, just as in Australia later, the vast majority of settlers were not convicts.
After independence it became more difficult and less useful to send convicts to North America. The military needed an outpost built in the South Pacific to forestall the French. British and Irish slaves were the only means available to do the job quickly and thoroughly.
Once Port Jackson had been established, limited free settlement was encouraged. This influx of free civilians was needed to expand agriculture and commerce so that the train of expensive supply convoys could be wound back. Early settlers were given convict slaves to assist in their farming and business ventures. At this time it was certain that there were two distinct classes of criminals being transported. Nobody wanted a murderer or horse buggerer for a stable boy.
Better to have a shonky accountant as a gardener and a prostitute as a house maid.
The reward for hard work and diligence was a promise of emancipation. When this eventually came it arrived without the right to hold land tenure and other basic freedoms. Free labour morphed to cheap labour. Private businessmen were "limited" eventually to a maximum of 70 slave labourers. Wealth was accumulated rapidly by those that used the system.
Getting the "criminals" away from the colony became a priority.
Two penal settlements were established. The young colony of New South Wales had already started transporting its convicts. They were sent to the remote and God forsaken Norfolk Island and to the misery that was Port Arthur in Tasmania. Convicts were not be used as settlers but rather as slaves where possible and if truly incorrigible, to be physically and mentally destroyed as far from the shores of New South Wales as possible. A further attempt was made to create a genuine penal colony on the mainland at Moreton Bay but that, along with all forms of transportation, became so opposed by the inrush of free settlers that it was suspended in 1840 and officially abolished 10 years later.
Interestingly, the colonies themselves repeated the Port Jackson experiment at King George Sound in Western Australia in 1826 to prevent the French from claiming the western half of the continent. A military settlement supported by convict slave labour established the port of Albany, which later became a whaling centre. Western Australia was founded by a free settlement organisation but could not attract a sufficient labour force until gold was discovered in the 1890s. It petitioned Great Britain for convict labour and was eventually granted limited "supplies" for a period of about 18 years. The British Government insisted that they would only be sent under conditions where they would be freed after a relatively short period of "service".
The number of transportees to Australia was relatively high for such a short period. Higher than North America for certain. This was out of nation building necessity and to some extent greed. But a glimpse of context in numbers can be seen in that in the year 1852 alone, nearly 3 times as many free settlers arrived in Australia than the combined total of all convicts transported ever. More Americans emigrated to Australia than convicts.
Modern Australia was founded as a military outpost not a prison colony. The use of British subjects as slaves has been an embarrassment for governments and historians alike. It is generally more acceptable to refer to "convict years" than "slave years". There won't be a rush to illuminate the record.