ChatGPT or Shakespeare? Readers couldn’t tell the difference and even preferred an AI-generated verse

If the whole world is a stage and all men and women just players, where does that leave non-human figures, such as artificial intelligence chatbots? It turns out that AI can hold its own against humans – even the Bard himself – when it comes to writing poetry.

AI chatbots can imitate famous poets like William Shakespeare well enough to fool many human readers, according to a new article published Thursday in the magazine Scientific reports. Furthermore, many study participants even preferred the chatbot’s poetry to the work of renowned writers.

Researchers asked OpenAI’s ChatGPT-3.5 to generate poems in the style of famous authors, including Walt Whitman, Geoffrey Chaucer, TS Eliot, Sylvia Plath, Allen Ginsberg, Emily Dickinson and William Shakespeare.

They then gathered 1,634 study participants and asked them to each read ten poems: five written by a human poet, and five written by the chatbot in the style of that same human poet. The poet was randomly assigned to each participant.

When scientists asked participants to indicate which poems were fake and which were real, participants guessed correctly about 46 percent of the time—slightly worse than if they had tossed a coin instead. This finding wasn’t necessarily a surprise, as ChatGPT-3.5 was likely trained on the works of the famous poets.

“Essentially, ChatGPT has demonstrated its knack for quasi-plagiarism,” says Keith Holyoaka cognitive psychologist at the University of California, Los Angeles, who was not involved in the study New scientist‘s Jeremy Hsu.

In a second experiment, researchers asked another group of 696 participants to read and rate poems on 14 qualities, ranging from rhythm to originality. They told a third of participants that they read poems written by an AI chatbot and another third that they read works written by a human. For the last third of the participants, the scientists did not reveal anything about the authorship of the poems. In reality, participants in all three groups were presented with a mix of poems written by humans and by AI

As expected based on previous research, participants who thought they were reading poems written by humans gave higher ratings than participants who thought they were reading AI-generated poems, regardless of what they actually read.

But the team also discovered a surprise: Participants who knew nothing about the poems’ origins gave higher average ratings to the poems written by the chatbot.

Why do readers seem to prefer AI-generated poetry? It’s not entirely clear, but the researchers’ best guess is that the AI ​​poems may be more appealing because they are relatively simple and easy to understand.

Because AI-generated poems cannot match the complexity of human-written verse, they are better at “unambiguously conveying an image, a mood, an emotion, or a theme to non-expert poetry readers,” the researchers write in the paper.

For example, they write the chatbots Plath style poem is clearly about sadness:

In addition to the themes and emotions, ChatGPT’s poems were also simpler in terms of their overall structure and composition.

“Emily Dickinson sometimes deliberately violates the expected rhyme scheme,” says co-author of the study Brian Portera researcher at the University of Pittsburgh, to New scientist. “But the AI-generated poems generated in her style never did that once.”

Understanding poems written by people requires deep and critical thinking – and that is a large part of poetry’s appeal, the researchers write in the paper. But modern readers don’t seem to want to do this work, preferring instead texts that give them “instant answers,” such as Andrew Deana literary scholar at Deakin University in Australia who was not involved in the study, writes in the Conversation.

“When readers say they prefer AI poetry, they seem to express frustration when confronted with writing that is not under their notice,” he adds.

In some cases, participants may have misunderstood the complexity of human poetry as AI incoherence. In other words, they could have been so confused by the real human-written work that they convinced themselves it was garbled chatbot nonsense.

This theory seems to be supported by participants’ responses to T. S. Eliot’s “The Boston Evening Transcript”, reports the WashingtonPostis Carolyn Y. Johnson. The poem, a satire about the readers of a once-popular newspaper, was most often misidentified as AI-generated. After reading Eliot’s words, one participant even wrote, in all caps, “IT MAKES NO EXPLANATION TO ME OR COMES FROM SOMEONE WHO HAS FEELINGS.”

The study’s findings appear to confirm many onlookers’ biggest fears about AI, which is that this will one day be the case replace human artists and puts them out of work. But Dorothea Laskythe only living poet whose writings were included in the experiments says it’s not necessarily a bad thing that readers enjoyed the AI-generated poems.

“Poetry will always be necessary,” says Lasky WashingtonPost. “If these people in the study read AI poems and liked that poem more than a human-generated poem, then I think that’s wonderful. They had a good experience with a poem, and I don’t care who wrote it. I feel like there is room for all poets, even robot poets.”