However, The chart is given in Sv. Sv takes into account that some radiation is more harmful than others. So, the biological effects from 1 mSv should be the same whether it came from an alpha emmiter or a beta emmiter.
Again, some radionuclides concentrate in parts of the body (others are eliminated quickly - see effective halflife which combines radiological halflife and biological halflife). So, how can we know how many mSv we might get from ingesting one isotope or another? You want to look at commited dose. This is a calculation of how much dose (mSv) you recieve from ingesting some radioisotope. You then use that figure, in mSv, to compare against the chart on xkcd. What you might be interested in is ALI (annual limit on intake). This will give you an amount of a radionuclide (measured in activity or mass) that, if ingested, will give you the highest allowable dose (measured in mSv).
So, you can compare the damage done by various radioisotopes done to you in various ways if you are comparing them in the right units, mSv. But you couldn't compare them just by giving the amount of substance (without considering what kind of radiation and what in the body was irradiated). But, those calculations can be done, and the answer is given in mSv or mrem. This is why the xkcd chart uses mSv for the units, so that a meaningful comparison can be made.