What you've described has been known for centuries as a "book cipher". Benedict Arnold used one during the American Revolutionary War to protect his treasonous communication with England.
Anyway, there's a really fun way to beat this kind of encryption today. If Mallory can get Alice or Bob to send a copy of BLACK_SQUARE.BMP, it's literally game over. Imagine XORing your key against a bunch of binary zeros. The result is a big patch of the cleartext version of the data that is your key. Google will find that faster than you can.
I did this to a friend who had the same idea in a "you'll never guess my encryption" challenge. After getting him to download a copy of BLACK.GIF, I stared at the intercepted results for many seconds longer than I should have. It output a repeating string of something like SLASHDOTTODHSALS, so I said that's your key. He was arguing because his key was SLASHDOT, and his "algorithm" was to invert the letters of the key word and append a copy to the end of the key. My mind boggled because I was expecting encryption, not immediate success at recovering his key and data.
Now, let's say you're smart enough to avoid encrypting BLACK_SQUARE.BMP. I can still achieve most of the same results by predicting that your data stream will contain "Host:", "Content-Type:", "Accept: text/plain", "User-Agent:", "HTML", "BODY", and other such 'cribs' (I was all set up to apply this logic to the intercepted message from my friend mentioned above.) By matching fragments of my guesses with your message, I can look to see if I recover legible text. It only takes a surprisingly small amount of recovered text to be able to identify the source.