ChatGPT, Author of The Quixote


Hugo Bowne-Anderson


March 24, 2024

In the era of generative AI, copyright won’t be enough. In fact, it’s the wrong place to look.


In Borges’ fable Pierre Menard, Author of The Quixote, the eponymous Monsieur Menard plans to sit down and write a portion of Cervantes’ Don Quixote. Not to transcribe, but re-write the epic novel word for word:

His goal was never the mechanical transcription of the original; he had no intention of copying it. His admirable ambition was to produce a number of pages which coincided—word for word and line by line—with those of Miguel de Cervantes.

He first tried to do so by becoming Cervantes, learning Spanish, and forgetting all the history since Cervantes wrote Don Quixote, among other things, but then decided it would make more sense to (re)write the text as Menard himself. The narrator tells us that “the Cervantes text and the Menard text are verbally identical, but the second is almost infinitely richer.” Perhaps this is an inversion of the ability of Generative AI models (LLMs, text-to-image, and more) to reproduce swathes of their training data without those chunks being explicitly stored in the model and its weights: the output is verbally identical to the original but reproduced probabilistically without any of the human blood, sweat, tears, and life experience that goes into the creation of human writing and cultural production.

Generative AI Has a Plagiarism Problem

ChatGPT, for example, doesn’t memorize its training data, per se. As Mike Loukides and Tim O’Reilly astutely point out,

A model prompted to write like Shakespeare may start with the word “To,” which makes it slightly more probable that it will follow that with “be,” which makes it slightly more probable that the next word will be “or” – and so forth.

So then, as it turns out, next-word prediction (and all the sauce on top) can reproduce chunks of training data. This is the basis of the NYTimes lawsuit against OpenAI. I have been able to convince ChatGPT to give me large chunks of novels that are in the public domain, such as those on Project Gutenberg, including Pride and Prejudice. Researchers are finding more and more ways to extract training data from ChatGPT and other models. As far as other types of foundation models go, recent work by Gary Marcus and Reid Southern has shown that you can use Midjourney (text-to-image) to generate images such as these1:

[Image from here]

This seems to be emerging as a feature, not a bug, and hopefully it’s obvious to you why they called their IEEE opinion piece Generative AI Has a Visual Plagiarism Problem. And the space is moving quickly: SORA, OpenAI’s text-to-video model, is yet to be released and has already taken the world by storm.

Compression, Transformation, Hallucination, and Generation

Training data isn’t stored in the model per se but large chunks of it are reconstructable, given the correct key (“prompt”).

There are lots of conversations about whether or not LLMs (and machine learning, more generally) are forms of compression or not. In many ways, they are, but they also have generative capabilities that we don’t often associate with compression.

Ted Chiang wrote a thoughtful piece for the New Yorker called ChatGPT is a Blurry JPEG of the Web that opens with the analogy of a photocopier making a slight error due to the way it compresses the digital image. It’s an interesting piece that I commend to you but one that makes me uncomfortable. To me, the analogy breaks down before it begins: firstly, LLMs don’t merely blur, but perform highly non-linear transformations, which means you can’t just squint and get a sense of the original; secondly, for the photocopier, the error is a bug, whereas, for LLMs, all errors are features. Let me explain. Or, rather, let Andrej Karpathy explain:

I always struggle a bit [when] I’m asked about the “hallucination problem” in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines.

We direct their dreams with prompts. The prompts start the dream, and based on the LLM’s hazy recollection of its training documents, most of the time the result goes someplace useful.

It’s only when the dreams go into deemed factually incorrect territory that we label it a “hallucination”. It looks like a bug, but it’s just the LLM doing what it always does.

At the other end of the extreme consider a search engine. It takes the prompt and just returns one of the most similar “training documents” it has in its database, verbatim. You could say that this search engine has a “creativity problem” - it will never respond with something new. An LLM is 100% dreaming and has the hallucination problem. A search engine is 0% dreaming and has the creativity problem.

As a side note, building products that strike balances between Search and LLMs will be a highly productive area and companies such as Perplexity AI are also doing interesting work there.

It’s interesting to me that, while LLMs are constantly “hallucinating”2, they can also reproduce large chunks of training data, not just go “someplace useful”, as Karpathy put it (summarization, for example). So: is the training data “stored” in the model? Well, no, not quite. But also…. Yes?

Let’s say I tear up a painting into a thousand pieces and put them back together in a mosaic: is the original painting stored in the mosaic? No, unless you know how to rearrange the pieces to get the original. You need a key. And, as it turns out, there happen to be certain prompts that act as keys that _unlock _training data (for insiders, you may recognize this as extraction attacks, a form of adversarial machine learning).

This also has implications for whether Generative AI can create anything particularly novel: I have high hopes that it can but I think that is still yet to be demonstrated. There are also significant and serious concerns about what happens when we continually train models on the outputs of other models.



  1. It’s ironic that, when syndicating this essay on O’Reilly Radar, we didn’t reproduce the images from Marcus’ article because we didn’t want to risk violating copyright–a risk that Midjourney apparently ignores and perhaps a risk that even IEEE and the authors took on!↩︎

  2. I’m putting this in quotation marks as I’m still not entirely comfortable with the implications of antropomorphizing LLMs in this manner.↩︎

  3. My intention isn’t to suggest that Netflix is all bad. Far from it, in fact – Netflix has also been hugely powerful in providing a massive distribution channel to creatives across the globe. It’s complicated.↩︎

  4. Also note that the outcome of this case could have significant impact for the future of OSS and open weight foundation models, something I hope to write about in future.↩︎