The Making of One Drop in the Ocean – AI Art and Text Generation
For my latest original, free bedtime story, I decided to generate a lot of AI art to accompany the text. I’m getting a lot more comfortable with Photoshop, prompting the AI and using img2img and inpainting in Stable Diffusion.
Using InPainting for AI Art
In my experience, Stable Diffusion can really struggle if you specify too many elements in a single prompt. For example, we really struggled to generate one image containing both a frog and a fish in a pond. However, we learned from that experience and did several things this time around that really made making a composited AI image easier:
- Choosing an AI art style that lends itself to easily hiding artifacts left over from compositing via clipping masks, filters and adjustment layers. In the case of One Drop in the Ocean, I chose digital oil painting since Photoshop has an oil paint filter.
- Generating the background first. For Little Frog, Big Dragon, I made the mistake of trying to generate the background after we had the foreground subject. This created the problem of trying to match the camera angle and perspective between the two.
Creating the stream in the forest
The first step was making the background. I chose the initial composition by running a large batch at 20 sampling steps and a denoising level of 0.7. After picking the one I liked best, I had Stable Diffusion add more detail by running the picture through img2img at 150 sampling steps and a denoising level of 0.3.
20 sampling steps 150 sampling steps
Improving the Stream
Since the story is about the water cycle, it’s important for the stream to be depicted well. In the AI art above, the stream has motion blur. That style would be appropriate for a long-exposure photograph, but not for an oil painting. Here I used in-painting to add additional detail.
Area selected for improvement using inpainting The stream with
Generating the Stag for AI Art
Now that the background was looking nice, I wanted the image to directly reference the story. After the snow melts, the text says, “The river brought new life to the mountain, and the animals that lived there began to stir.” Given the scene composition, I decided to add a deer as a point of interest. I based the stag on a long-deceased artist whose work is in the public domain, Edwin Landseer, because of the ethical debate around using the work of living artists to generate AI art.
To get started, I took the background image and ran it through Stable Diffusion’s upscaler to increase its size by 4x. Then I took a 512×512 cutout of the area where I wanted to add the deer. I then took a cutout of Landseer’s famous painting, Monarch of the Glen, composited it into the cutout and fed it into img2img. This process is based on a tip from Albert Bozesan’s excellent Youtube video tutorial series on making composited AI art. Over several iterations, I got SD to create a deer that no longer resembled Landseer’s and also had sufficient contrast that its features would be distinguishable after I scaled it down.
Inpainting progression to generate the stag
Once I got a deer I liked, I composited it into the background and used Photoshop tools to adjust the color, lighting and masking to match the deer to the rest of the scene. Here is the final AI art:
Using Playground to Write the Text
For One Drop in the Ocean, the Open AI’s Playground really struggled to write from the perspective of a water drop. Part of this was the phrasing of my initial prompts:
Prompt: Write a description of a drop of an anthropomorphic drop of water in the ocean.
Playground: A small anthropomorphic drop of water falls into the vast ocean. It’s a little scared at first, but soon becomes exhilarated by the experience. It tumbles and flips through the waves, enjoying the salty water and the feel of the sand between its toes.
That’s cute, but a drop with toes wasn’t what I was going for. After I manually rewrote a couple paragraphs and fed them to Playground as example text to follow, it significantly improved. I will follow up more on using Playground in a subsequent blog post.