Generative AI and creativity.

The products of generative AI are becoming ever more present in daily life. As consumers of totally synthetic images and sounds with no discernible author other than a neural network to whom do we ascribe the act of creation? to the human who wrote the prompt? to the originator of the training data? or to the AI itself? Many social, legal, and financial issues also hinge on the answer to this question.

Computational creativity

These ideas have been previously addressed under the topic of computational creativity. Can a computer create art? Can a computer create anything in addition to that for which it was programmed by a human mind? In 1842 Ada Lovelace writes that a computer may one day be able to compose music.

The concept of creativity has been investigated in detail in the writing of Margaret Boden. Everyone is creative to some extent, the ability to speak and converse involves a generative grammer, solving the problems of daily life often requires some level of creative thought. Her analysis finds three types of creativity:

1. making unfamiliar combinations of familiar ideas, not just random but a combination which makes sense in some way (for example: collage in art; analogies in language; creative associations like a physicist comparing an atom to the solar system);
2. forming a new idea within a conceptual space (where a conceptual space is a structured styles of thought for example styles of poetry, prose, painting, choreography), this is an exploratory creativity which generates permutations within the space in a combinatorial fashion;
3. the final type of creativity changes the space itself. A conceptual space sets limits to the possibilites of thought. Certain thoughts are impossible, unthinkable. However with this type of creativity someone thinks something which, with respect to, the conceptual space, they could not have thought. The result of this is that the map of the conceptual space is transformed.

Can LLMs be creative?

This question can be approached as another example of the question of computational creativity. Artificial intelligence focuses on creating raw intelligence, however the most astonishing recent developments are impactful as a result of the surprise in their creative abilities. Something which is often assumed to be a solely human ability.

The transformer neural network which underlies the large language model is function by predicting the next term in a sequence. The astonishing interactions with LLMs are a result of scale, both in terms of a huge number of trainable parameters and the volume of training data. The LLM is definitely capable of surprising us with it generated utterances, but how creative is this process?

Here are some samples of text generated with a LLM (matthh/Mistral-7B-v0.1_poetry-fine-tuned) that I fine-tuned with a poetry dataset (DanFosing/public-domain-poetry). The author and title of the poem were provided as the prompt to generate the content of the poem.

Selected at random here is one of the poems from the dataset:

Author': 'Robert Browning',
Title': 'The Italian In England'
text:
That second time they hunted me
From hill to plain, from shore to sea,
And Austria, hounding far and wide
Her blood-hounds thro' the country-side,
Breathed hot and instant on my trace,
I made six days a hiding-place
Of that dry green old aqueduct
Where I and Charles, when boys, have plucked
The fire-flies from the roof above,

And here are some lines of the poem generated by the fine-tuned model:

1
Tis said, in days of yore, an Englishman
Went over to the Continent and there
Fell in love with a young Italian lass,
And brought her back across the Channel hither;
2.
Whereat his friends were wroth, as you may guess,
Saying 'twas folly for him thus to marry
A foreigner who could not understand
His language, nor he hers, save only

1.
'Tis said, in days of yore, an Englishman
Went over to the Continent and there
Fell into love with a young lady fair;
And when he came back home again, \'twas then
He found his wife had borne him two fine sons . . .
2.
The eldest boy was called "John," you see?
"George" was the name they gave unto the other!
Now what

1.
I'm an Italian in England,
And the English are my foes;
Yet I love them as a lover
Loves his mistress when she goes.
2.
She is fair to look upon,
Though her heart be full of guile;
Still it pleases me to see her,
When she walks along the Nile.
3.
Therefore do I sit and watch her,
Sitting by the river

Are any of them surprising? Can they be said to be creative? And if they are creative is it type 2 creativity or type 3. These examples seem to be more consistent with the second type of creativity: combinatorial exploration of a conceptual space. Does this mean that a language model is not capable of the third type of creativity?

This is also happening in other applications of AI, for example the discovery of new materials, or the discovery of a new class of antibiotics? Also in the example of move number 37 made in the “alien” style of Go playing made by AlphaGo, a move which was new in the history of Go. Are these examples combinatorial iterations thriogh an existing conceptual space or are they realigning the conceptual space itself?