I do the same thing. I let whatever wants to grow, grow, so long as it doesn't mind being no taller than about 4 inches. We have all kinds of strange things living in our yard that I am positive would not be there if we had your typical toxic waste pit of an American yard.
We're also in the woods, and I make no effort to remove felled trees (except to remove trees that are threatening falling on the house), allowing them instead to decompose on the floor of the woods like they are supposed to.
What is alarming to me is the presence of several invasive species. We have asian giant hornets, land planariums (which are ***seriously*** bad things to have), and other asian insects that presumably hopped rides in shipping containers from the far east.
These invasive species have no natural predators and their populations are soaring. We had a tree fall this summer in a period of heavy rain, and the root ball was just infested with planariums. These things compete with earthworms for resources, but do not excrete anything useful into the soil, so areas that get infested with them cannot grow flora very well, and trees can die.