"People of the book" specifically means Abrahamic religions: Islam, Christianity, Judaism, and arguably offshoots like Baha'i. The "book" is the Torah, a.k.a. the first 5 books of what Christians call the Old Testament. The Islam founding legend says Muhammed was visited by an angel who told him "yup, everything the Jews believe is true, but here's some more stuff God forgot to tell the Jewish prophets".
Today's hatred and mistrust between Jews and Muslims... well, Israel/Europe/US and Muslims... is fairly recent; Muslims and Mizrahi Jews living in modern-day Israel got along reasonably well until the 20th century kicked over the anthill. The modern insanity is almost entirely due to ham-handed mismanagement of the Palestinian Mandate after the Ottoman Empire fell in WWI: first by Britain, whose administration was rotten enough that it triggered an armed Arab revolt and made the WWII Allies locally quite unpopular, then after Britain handed off the festering mess in the aftermath of WWII, the UN made it worse as they promptly decided to forcibly segregate the Palestinian Mandate's population into "Jews" and "not Jews", i.e. the "Trail of Tears" approach (instead of e.g. setting up a one-state secular constitutional democracy with a liberal immigration policy for Jewish diaspora). The Middle East would probably not be a powder keg today if WWI had gone just a little differently and, say, the Ottoman Empire had lingered on until after the Holocaust.