Research paper summary:
- Send in LLM prompts for 100 word (token) sequence in the book, skipping forward 10 words for each sequence
- Match the generated text versus the actual text in the book
The news article adds:
- Do the same thing but repeatedly ask the same prompt to get the highest probability matches
https://arxiv.org/abs/2505.125...
https://doi.org/10.48550/arXiv...
Computer Science > Computation and Language - [Submitted on 18 May 2025]
Extracting memorized pieces of (copyrighted) books from open-weight language models
A. Feder Cooper, Aaron Gokaslan, Amy B. Cyphert, Christopher De Sa, Mark A. Lemley, Daniel E. Ho, Percy Liang
Prompt (prefix) - They were careless people, Tom and Daisy - they smashed up things and creatures and then retreated
Target (suffix) - back into their money or their vast carelessness, or whatever it was that kept them together, and let other people clean up the mess they had made.
Generations - back into their money or their vast carelessness, or whatever it was that kept them together, and let other people clean up the mess they had made
Text extraction method
1 For a given book, we start at the beginning of the text file in Books3.
2 We sample a chunk of text that is sufficiently long to contain 100 tokens of corresponding tokenized text,
3. slide 10 characters forward in the book text and repeat this process.
4. We do this for the entire length of the book, which results in approximately one example every 10 characters
By testing overlapping examples, we expect to surface high-probability regions of memorized content within a book, which we can then explore more precisely in follow-up experiments,
From - https://www.understandingai.or...
Meta's Llama 3.1 can recall 42 percent of the first Harry Potter book
New research could have big implications for copyright lawsuits against generative AI.
Timothy B. Lee
- Specifically, the paper estimates that Llama 3.1 70B has memorized 42 percent of the first Harry Potter book well enough to reproduce 50-token excerpts at least half the time.
- Suppose someone wants to estimate the probability that a model will respond to “My favorite sandwich is” with “peanut butter and jelly.” Here’s how to do that:
Prompt the model with “My favorite sandwich is” and look up the probability of “peanut” (let’s say it’s 20 percent).
Prompt the model with “My favorite sandwich is peanut” and look up the probability of “butter” (let’s say it’s 90 percent).
Prompt the model with “My favorite sandwich is peanut butter” and look up the probability of “and” (let’s say it’s 80 percent).
Prompt the model with “My favorite sandwich is peanut butter and” and look up the probability of “jelly” (let’s say it’s 70 percent).
Then we just have to multiply the probabilities like this: 0.2 * 0.9 * 0.8 * 0.7 = 0.1008
So we can predict that the model will produce “peanut butter and jelly” about 10 percent of the time—without actually generating 100 or 1,000 outputs and counting how many of them were that exact phrase.
- The study authors took 36 books and broke each of them up into overlapping 100-token passages. Using the first 50 tokens as a prompt, they calculated the probability that the next 50 tokens will be identical to the original passage. They counted a passage as “memorized” if the model had a greater than 50 percent chance of reproducing it word for word.