As much as the claim "a bunch of Imams got together and basically stated that Math and Science were of the devil" plays into contemporary stereotypes, it is a very superficial and incomplete assessment.
What happened was a period of scientific breakthroughs and constant progress in conjunction with the expanding empires of Islam from Andalusian Spain, to Kashgar in Western China.
Then, several events started the slow but steady decline. The first was the Mongol invasion from the east, which destroyed Baghdad as a seat of science (and government) for the Muslim east. Great libraries were lost in the event. The Silk Road trade was eliminated, and with it all the hinterland that produced luminaries such as Al-Farabi, Al-Biruni and many more for many centuries.
The second was the Reconquista in Spain which took several centuries. Again, untold amounts of books were burned or lost.
Then following the fall of Constantinople to the Ottoman Turks, 1492 saw two events: the final fall of Muslim Spain, and Columbus' discovery of America. With the wealth of the Americas, Europe now had access to new trade and riches, and developed many technologies for sailing, trade, military, ...etc. No longer were they constrained by the Muslim Orient being a barrier between them and trade limited to India and China.
Then came the rejection of modern technology: the most stark example is the printing press. While Europe started the Renaissance, and printed books started an intellectual revolution, the printing press was rejected in Muslim lands. I am not sure why, but perhaps the Ottoman authorities feared it as a means of insubordination. Regardless, the end result was 3 centuries of relying on manuscripts only, causing poor dissemination of knowledge.
You can see the effect even in religious disciplines, for example, jurisprudence: the later commentators were just compilers/editors/summarizers of earlier texts. Even they declared that the "door to ijtihad has been closed", and all that has been said has been said, nothing new was to come about. This decline happened under late Mameluke and Ottoman rule.
This was soon followed by the colonialism period from Mughal India (1700s by the British East India Company), North Africa (France 1830s), Egypt (Britian 1882), Palestinian mandate, and the rest of it.
Following World War II, military dictators came to power (Nasser, Sukarno, Assad, Saddam, Qaddafi, ...etc.)
At least the Arab countries have not yet recovered from those last 2 stages.
For more on Science under Islam, watch this awesome BBC documentary: Science and Islam - Episode 1.
The three episodes are described here:
Also articles, books and talks by Dr. George Saliba (Columbia University) are highly recommended in this regard. He is interviewed in the above documentary.