Everything is toxic. It depends on the dose as to when it reaches toxic levels. For sugar, the LD50 is >10,000 mg per kg of body weight. In comparison, caffeine's LD50 is 100 mg/kg and nicotine's is 1 mg/kg.
"All things are poison, and nothing is without poison; only the dose permits something not to be poisonous." Paracelsus, the father of toxicology.