Unlocking Artistry: A Guide to Neural Style Transfer

Imagine having the ability to recreate Leonardo da Vinci's artistic style in your own photographs. To make your images appear convincingly as if da Vinci himself painted them – sounds quite remarkable, doesn't it? In this article, we'll delve into the world of Neural Style Transfer (NST) and show you how to apply it to images, videos, and even real-time video streams.

Content

1. What is Neural Style Transfer?
2. How does it work?
3. Neural Style Transfer Use Cases
4. How to Stylize Images
5. How to Stylize Videos
6. Real-Time Video Stylization
7. Conclusion

What is Neural Style Transfer?

Neural Style Transfer is a captivating technique that involves blending the artistic style of one image with the content of another, all while preserving the core essence of the content image. This process involves supplying the model with two input images: one representing the desired style and another portraying the content to be transformed.

How does it work?

Neural Style Transfer might seem like a magical feat in the realm of computer science, specifically within the field of machine learning. Think of it as teaching a computer to mimic the artistic techniques of various renowned painters. Let's break down the process into simpler terms:

1. Neural Networks: Picture a neural network, a sophisticated computer program that learns patterns from images. This network can identify shapes, colors, and objects present in images.

2. Content and Style Images: For this magic to unfold, you need two images – a content image and a style image. The content image represents the image you want to transform, while the style image is the masterpiece whose artistic style you wish to apply.

3. Layers of Understanding: The neural network operates in layers, much like how our brain processes different aspects step by step. Each layer focuses on distinct elements in the images, like lines, textures, or colors.

4. Finding the Perfect Blend: The key lies in adjusting the content image to retain its essence while adopting the stylistic elements of the style image. The neural network does this by altering the pixels of the content image bit by bit.

5. Achieving Balance: As the network modifies the content image, it continuously compares it with the style image. The goal is to make the content image mirror the style image's artistic expression while preserving the underlying content.

6. Iterations: This process iterates repeatedly, similar to a painter refining their artwork. With each iteration, the content image gets closer to harmonizing the desired content and admired style.

7. Revealing the Magic: After several iterations, the content image emerges with an appearance as though it were painted using the chosen style. It's akin to the computer mastering the fusion of two images in a truly artistic manner.

Neural Style Transfer Use Cases

The applications of Neural Style Transfer span a wide spectrum of creativity and innovation:

1. Artistic Rendering and Filters: NST serves as a digital canvas, transforming photographs into captivating artworks inspired by diverse artists and art movements.

2. Fashion and Design: Designers can experiment with unique visual styles, inspiring fresh ideas for textiles, clothing, and interior design.

3. Entertainment and Media: In cinema, gaming, and virtual reality, NST breathes life into virtual worlds by applying distinct visual aesthetics that evoke various eras, genres, or artistic sensibilities.

4. Educational Enrichment: Complex concepts find a new dimension as NST contributes to scientific illustrations and diagrams, making learning engaging and relatable.

5. Architectural Visualization: Architects and real estate developers can showcase designs with diverse artistic styles, helping clients envision projects from novel perspectives.

How to Stylize Images

To embark on the journey of stylizing images, follow these steps:

Begin by installing the required libraries using the following terminal command:

   pip install tensorflow tensorflow_hub matplotlib numpy --upgrade

Next, import the necessary libraries and create a function that leverages TensorFlow's fast style transfer capabilities.

import matplotlib.pyplot as plt
import tensorflow as tf
import tensorflow_hub as hub
import numpy as np

def preprocess_and_view_image(content_image_path, style_image_path, save_path=None):
    content_image = plt.imread(content_image_path)
    style_image = plt.imread(style_image_path)
    content_image = content_image.astype(np.float32)[np.newaxis, ...] / 255.
    style_image = style_image.astype(np.float32)[np.newaxis, ...] / 255.

    style_image = tf.image.resize(style_image, [256, 256])
    hub_module = hub.load('https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2')
    outputs = hub_module(tf.constant(content_image), tf.constant(style_image))
    stylized_image = outputs[0]
    stylized_image_array = stylized_image.numpy()
    stylized_image_array = np.clip(stylized_image_array, 0, 1)

    plt.figure(figsize=(15, 5))
    plt.subplot(131)
    plt.imshow(content_image[0])
    plt.title('Content Image')
    plt.axis(False)
    plt.subplot(132)
    plt.imshow(style_image[0])
    plt.title('Style Image')
    plt.axis(False)
    plt.subplot(133)
    plt.imshow(stylized_image_array[0])
    plt.title('Stylized Image')
    plt.axis(False)
    plt.show()

    if save_path is not None:
        plt.imsave(save_path, stylized_image_array[0])

preprocess_and_view_image('content_image.jpg', 'style_image.jpg', 'output_image.jpg')

Use the function to preprocess and view your stylized image:

    preprocess_and_view_image('content_image.jpg', 'style_image.jpg', 'output_image.jpg')

Output:

How to Stylize Videos

Moving beyond images, you can extend the magic of Neural Style Transfer to videos. Let's explore how to stylize videos frame by frame:

Install Dependencies:
Before diving in, ensure you have the necessary libraries. You'll need OpenCV for video processing and TensorFlow for style transfer.

   pip install opencv-python-headless tensorflow tensorflow_hub numpy

Import Libraries: We start by importing the necessary libraries. OpenCV (cv2) is used for video processing, TensorFlow (tf) is employed for style transfer, and numpy is used for numerical computations.

   import cv2
   import tensorflow as tf
   import tensorflow_hub as hub
   import numpy as np

Preprocess Video: This function, preprocess_video, takes a video file path, a style image path, and an optional output path for the stylized video. It captures the video and applies style transfer to each frame.

   def preprocess_video(video_path, style_image_path, output_path=None):
       cap = cv2.VideoCapture(video_path)
       style_image = plt.imread(style_image_path)

Preprocessing Style Image: Before applying style transfer, the style image is loaded, and its pixel values are normalized to the range [0, 1]. It's also resized to match the dimensions used in the style transfer model.

       style_image = style_image.astype(np.float32)[np.newaxis, ...] / 255.
       style_image = tf.image.resize(style_image, [256, 256])
       hub_module = hub.load('https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2')

Applying Style Transfer: Within the video processing loop, each frame is converted to the appropriate format for style transfer. The style transfer model from TensorFlow Hub is used to stylize each frame, which is then post-processed.

       while cap.isOpened():
           ret, frame = cap.read()
           if not ret:
               break

Stylizing and Post-processing: Each frame is preprocessed and stylized using the TensorFlow Hub model. The stylized frame is then converted back to BGR format for displaying or saving.

           stylized_output = hub_module(tf.constant(frame), tf.constant(style_image, dtype=tf.float32))
           stylized_video_output = stylized_output[0].numpy()[0]
           stylized_video_output = (stylized_video_output * 255).astype(np.uint8)
           stylized_video_output = cv2.cvtColor(stylized_video_output, cv2.COLOR_RGB2BGR)

Displaying or Saving: Depending on the presence of an output path, the stylized frame is either displayed using cv2.imshow or saved as part of the stylized video using out.write.

          if output_path is not None:
               out.write(stylized_video_output)
           else:
               cv2.imshow('Stylized Video', stylized_video_output)
               if cv2.waitKey(1) & 0xFF == ord('q'):
                   break
       cap.release()
       if output_path is not None:
           out.release()
       cv2.destroyAllWindows()

Applying the Function: To stylize a video, use the preprocess_video function with the input video path, style image path, and an optional output path.

   preprocess_video('input_video.mp4', 'style_image.jpg', 'output_video.mp4')

Output:

Real-Time Video Stylization

Taking it a step further, let's achieve real-time video stylization using your computer's camera:

Initialize Webcam:
Begin by capturing video from your webcam.

   def real_time_stylization(style_image_path):
       cap = cv2.VideoCapture(0)  # 0 refers to the default camera
       style_image = plt.imread(style_image_path)
       style_image = style_image.astype(np.float32)[np.newaxis, ...] / 255.
       style_image = tf.image.resize(style_image, [256, 256])
       hub_module = hub.load('https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2')

       while cap.isOpened():
           ret, frame = cap.read()
           if not ret:
               break

           frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
           frame = frame.astype(np.float32)[np.newaxis, ...] / 255
           frame = tf.image.resize(frame, [256, 256])

           stylized_output = hub_module(tf.constant(frame), tf.constant(style_image, dtype=tf.float32))
           stylized_video_output = stylized_output[0].numpy()[0]

           stylized_video_output = (stylized_video_output * 255).astype(np.uint8)
           stylized_video_output = cv2.cvtColor(stylized_video_output, cv2.COLOR_RGB2BGR)

           cv2.imshow('Real-Time Stylization', stylized_video_output)
           if cv2.waitKey(1) & 0xFF == ord('q'):
               break

       cap.release()
       cv2.destroyAllWindows()

Apply the Function:
Use the real_time_stylization function to view real-time stylization through your webcam.

   real_time_stylization('style_image.jpg')

Conclusion

Our voyage through the realm of Neural Style Transfer (NST) has opened the doors to a world where commonplace images and videos can be transformed into mesmerizing masterpieces. By seamlessly blending the artistic essence of one image with the content of another, NST elevates creativity and innovation to unprecedented heights. Let's summarize the key waypoints on our journey:

• Comprehending NST: Our journey began with an introduction to Neural Style Transfer, an ingenious technique that harmonizes the artistic flair of one image with the substance of another, conjuring enchanting outcomes.

• Unveiling the Process: We delved into the intricacies of NST, unveiling the mechanics that enable a computer to emulate diverse artists' techniques. From understanding neural networks to unraveling iterative refinement, we demystified the wizardry behind this technique.

• The Tapestry of Applications: Our exploration extended to the multifaceted applications of NST. From infusing artistic renderings and inspiring fashion designs to animating cinematic visuals and enriching education, the horizons of NST span as wide as the imagination.

• Crafting Image Marvels: Our journey deepened as we explored the practical implementation of NST on images. Armed with the tools to merge content and style seamlessly, you're empowered to fashion your own captivating artworks.

• Animating Visual Narratives: Venturing further, we harnessed NST's magic for videos. By imbuing each frame with distinct stylistic elements, we witnessed how NST transmutes moving sequences into aesthetic visual stories.

• Live Stylization Spectacle: Our odyssey culminated in real-time video stylization. Through your webcam, you gained a front-row seat to the captivating fusion of content and style, unfolding before your eyes in an instant.

As you embark on your personal journey with Neural Style Transfer, always remember that innovation recognizes no confines. Whether you're an artist seeking fresh perspectives or a technologist pushing the boundaries of visual creativity, NST provides a blank canvas where imagination commands the stage.

Set forth, experiment boldly, and let your artistic expression thrive through the boundless potential of Neural Style Transfer.

Find the code repository on GitHub and connect with me on LinkedIn.