Enhancing Your Shopping Experience with AI and LORA

While wisiting a cloating e-commerce shop, I had a random thought: "Wouldn't it be cool to preview clothes on myself while shopping?" As a developer by trade, and having prior experience with stable diffusion using Dream Booth, I decided to build an MVP to do just that.

Exploring LORA

First, I attempted to use Dream Booth to learn my face. However, I stumbled upon something even cooler: LORA (Low-Rank Adaptation). LORA proved to be faster and easier to train and had the potential for combining different aspects. For instance, you could combine a LORA model of a user's face with a LORA model of the clothes you want that user to wear.

I started by creating a LORA model of my face and had some fun with it, generating cool images like this one:

Then, I created a LORA model of jackets from images found on an e-commerce website:

Next, I combined these LORA models in a prompt and achieved pretty decent results:

However, I noticed that the images weren't perfect, especially when it came to small details of the jacket and my face. I ran more prompts and tried to improve the LORA model, but my face was consistently off.

Troubleshooting the LORA Model

I experimented with several solutions:

1. I attempted to label a person in the jacket dataset as a model and included this as a negative prompt, but it didn't yield significant improvements.

2. I refined the prompts and used WD4 to annotate images, but the results remained suboptimal.

After trying numerous approaches without groundbreaking success, I decided to explore using a control net to correct the person's pose and apply masks to allow LORA to work on each part of the image independently. Here's what it looked like:

The results were better, though my face was still not quite right. At first glance, everything seemed to be working okay:

Encouraged by these results, I attempted to add another piece of clothing to see if it would work with three LORA models. Here are the original images from the shop:

And the results:

This time, the jacket and jeans looked really good, but the face was worse than in the last batch. Unfortunately, I couldn't pinpoint the exact reason for this.

I'm still working on solving this issue, and when I do, I hope to share it with the community. If anyone has any insights or solutions, I would be incredibly grateful.