Logic fail. Dark matter can be explained by such small objects if they are incredibly numerous. It's just math: divide the missing mass by the mass of one brown dwarf to get the number needed. If you want to disprove the brown dwarf explanation you need to explain why the number that is needed contradicts something.
The hypothesized dark matter does not emit or absord any type of electromagnetic radiation, in other words it does not interfere with or react to light. Numerous small objects would. Also, and this is the most important bit in your logic fail fail, if you have enough small objects to account for five times the mass of the visible universe, you would have something five times more visible than the visible universe. Matter attracts other matter (which is why there is a dark matter hypothesis to begin with, something invisible attracts the visible) and such a copious amount of "small objects" would form larger objects. Which is how stars and planets form to begin with.