Image generation using Neural Style Transfer and Kangas

Sandy M.

·

Follow

Published in

Heartbeat

·

4 min read

·

Jul 24, 2023

--

Listen

Share

Algorithms for deep learning have been used to create art, enabling us to create original images by fusing the style and content of other embodiments. In this lesson, we will examine the idea of neural style transfer and discover how to use deep learning frameworks to execute it. By the conclusion of this tutorial, you will be able to produce your own artistic images that combine the style and substance of several photos.

Overview of Neural Style Transfer

Neural style transfer is a technology that blends two images’ content and style to produce a new image incorporating both features. In order to extract the content and stylistic features from the input photos, a pre-trained convolutional neural network (CNN) is used in the process. We will also be using kangas for visualizing images because it's a great tool for exploring and visualizing large-scale multimedia data.

The steps covered in the lesson are as follows:

• Install Kangas
• Preparing and loading the content and style pictures
• Being familiar with the VGG19 model architecture
• Taking the pre-trained model’s content and style attributes
• defining the loss function to balance the weights of the content and the style
• Reducing the loss by optimizing the resulting picture

Kangas Installation

pip install kangas

Neural style transfer implementation

Using Python and a deep learning framework called TensorFlow, we will go into the code implementation of neural style transfer in this section.

First, let’s import all the libraries that will be needed for this tutorial

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import kangas as kg
from kangas import DataGrid, Image

Now let’s load the pre-trained VGG19 model that will be used for training the model

vgg_model = tf.keras.applications.VGG19(include_top=False, weights='imagenet')

After that, we’ll preprocess the images that will be used to build the model. We’ll be using two images, the first image is the content image is the image whose content or subject we want to retain in the final generated image and the second is the style image which is the image that possesses a distinct artistic style.

content_image_path = 'content_image.jpg'
style_image_path = 'style_image.jpg'

# Load and preprocess content and style images
def load_and_preprocess_image(image_path):
    img = tf.io.read_file(image_path)
    img = tf.image.decode_image(img, channels=3)
    img = tf.image.convert_image_dtype(img, tf.float32)
    img = tf.image.resize(img, (256, 256))
    img = img[tf.newaxis, :]
    return img

content_image = load_and_preprocess_image(content_image_path)
style_image = load_and_preprocess_image(style_image_path)

After processing the image let’s load them with kangas and set up a data grid so we can have a good view of the images

content_image = kg.Image(content_image, name='content.jpeg') 
dg = kg.DataGrid(name='Art Images') 
dg.extend([{'Image': content_image}])

style_image = kg.Image(style_image, name='style.jpeg')
dg.append([{'Image': style_image}])

# kickstart the UI
dg.show()

After preprocessing the images the next step will be to define the styles and content layer that will be used to carry out feature extraction.

content_layers = ['block5_conv2']
style_layers = ['block1_conv1', 'block2_conv1', 'block3_conv1', 'block4_conv1', 'block5_conv1']

Now it’s time to build the style transfer model

def style_transfer_model(vgg_model, style_layers, content_layers):
    vgg_model.trainable = False
    style_outputs = [vgg_model.get_layer(name).output for name in style_layers]
    content_outputs = [vgg_model.get_layer(name).output for name in content_layers]
    model_outputs = style_outputs + content_outputs
    return tf.keras.models.Model(vgg_model.input, model_outputs)

style_transfer = style_transfer_model(vgg_model, style_layers, content_layers)

And now define the content and style weights

content_weight = 1e3
style_weight = 1e-2

The next step will be for us to define the loss function

def style_content_loss(outputs, style_targets, content_targets, style_weight, content_weight):
    style_outputs = outputs[:len(style_layers)]
    content_outputs = outputs[len(style_layers):]

    style_loss = tf.add_n([tf.reduce_mean(tf.square(style_outputs[i] - style_targets[i])) for i in range(len(style_outputs))])
    style_loss *= style_weight / len(style_outputs)

    content_loss = tf.add_n([tf.reduce_mean(tf.square(content_outputs[i] - content_targets[i])) for i in range(len(content_outputs))])
    content_loss *= content_weight / len(content_outputs)

    total_loss = style_loss + content_loss
    return total_loss

Now let’s extract the content and style features from the pre-trained model

def get_feature_representations(model, content_image, style_image):
    content_features = model(content_image)['block5_conv2']
    style_features = [model(layer)[name] for layer, name in zip(style_image, style_layers)]
    return content_features, style_features

content_features, style_features = get_feature_representations(style_transfer, content_image, style_image)

Next, we’ll create the initial generated image as a random noise and define the learning rate.

generated_image = tf.Variable(content_image + tf.random.normal(content_image.shape) * 0.01)
optimizer = tf.optimizers.Adam(learning_rate=0.02, beta_1=0.99, epsilon=1e-1)

It’s time to perform neural style transfer and define the number of training interactions

@tf.function()
def train_step(image):
    with tf.GradientTape() as tape:
        outputs = style_transfer(image)
        loss = style_content_loss(outputs, style_features, content_features, style_weight, content_weight)
    gradient = tape.gradient(loss, image)
    optimizer.apply_gradients([(gradient, image)])
    image.assign(tf.clip_by_value(image, clip_value_min=0.0, clip_value_max=1.0))

# Number of training iterations
num_iterations = 1000

And then perform style transfer for each of the iterations

for i in range(num_iterations):
    train_step(generated_image)
    if i % 100 == 0:
        print("Iteration:", i)

Now let’s use kangas again to visualize the generated image

image = kg.Image(generated_image[0]).to_pil()
image

Conclusion

In this lesson, we looked at the fascinating field of neural style transfer in deep learning for the arts. We discussed the main ideas underlying the method and gave detailed directions for implementing it with Python and a deep learning framework. We may create appealing and beautiful images via neural style transfer by fusing the content of one image with the style of another. You can carry on experimenting, changing the settings, and investigating variants of the approach to make it better.
Keep in mind that neural style transfer is just the start of the amazing possibilities that deep learning presents for the creation of art and images.

Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.

Editorially independent, Heartbeat is sponsored and published by Comet, an MLOps platform that enables data scientists & ML teams to track, compare, explain, & optimize their experiments. We pay our contributors, and we don’t sell ads.

If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletter (Deep Learning Weekly), check out the Comet blog, join us on Slack, and follow Comet on Twitter and LinkedIn for resources, events, and much more that will help you build better ML models, faster.