Building a Reverse Image Search app with Manticore Search

TL;DR: Learn how to build a reverse image search app with Manticore Search, including a look at the history of reverse image search, the technology behind it, and practical approaches for image retrieval systems.

Introduction

Reverse image search has changed how we discover digital content by allowing users to search using images instead of text. This technology has numerous applications, from helping shoppers find products to enabling designers to check their work against existing designs. It has become a key component in many digital platforms.

You can try out our reverse image search demo at image.manticoresearch.com, and check out the open-source implementation on GitHub.

We'll dive into how reverse image search works, discuss its real-world uses, and show how vector search technology makes this tool efficient and accessible for diverse scenarios, from everyday browsing to specialized applications.

Understanding Reverse Image Search

How Does Reverse Image Search Work?

Reverse image search allows users to search by uploading an image or providing an image URL, and the system returns visually similar images along with related information. The process involves several key steps, utilizing vector search technology to efficiently handle high-dimensional image data:

Feature Extraction: The system analyzes the image to identify key visual elements.
Embedding Generation: Visual features are turned into a numerical vector representation.
Similarity Comparison: This vector is compared against a database of stored image vectors using vector search.
Result Ranking: Results are ordered by similarity scores.

The Role of Machine Learning Models

Machine learning models, particularly deep learning, have revolutionized reverse image search. Early systems in the 2000s relied on basic color histograms and edge detection, which limited accuracy. The introduction of Convolutional Neural Networks (CNNs) in 2012, like AlexNet, significantly improved the capability to understand complex visual patterns.

Over time, services like Google Images and Pinterest evolved from basic metadata matching to using advanced deep learning models, providing precise search results. TinEye also showcased the ability to track image modifications and reuses across the web.

Modern machine learning models excel at understanding intricate visual patterns and generating detailed image embeddings that capture each photo's essence. These models can understand context, recognize objects from different perspectives, and even grasp artistic styles—achievements that were beyond the reach of older computer vision methods.

Building a Reverse Image Search System with TinyCLIP and Manticore Search

At Manticore Search, we explored several approaches to building an efficient reverse image search system. We wanted a practical solution that leveraged Manticore's powerful vector search while remaining accessible, and this led us to TinyCLIP — a streamlined model that runs efficiently on standard CPUs.

TinyCLIP is a perfect fit for our implementation because it balances processing efficiency with precision. Unlike larger models that require specialized GPU infrastructure, TinyCLIP can run on standard hardware, making it ideal for lightweight, containerized deployments.

Part of the CLIP (Contrastive Language-Image Pre-training) family, TinyCLIP can create vector embeddings for both images and text, enabling:

Searching for images using other images (reverse image search).
Searching for images using text descriptions.
Finding related text content with an image.
Matching text-to-text based on similarity.

TinyCLIP retains good accuracy while reducing:

Model size (less storage and memory).
Computational requirements (runs on CPUs).
Inference time (faster response).
Container footprint (easy deployment and scaling).

By integrating TinyCLIP with Manticore Search's vector storage, we created a practical foundation for building deployable multimodal search systems.

Vector Search and Manticore Search

Vector search is an effective approach for reverse image search, enabling fast and accurate comparisons of high-dimensional vectors. Manticore Search, an open-source engine, supports vector search, making it a powerful choice for implementing reverse image search.

Implementing Reverse Image Search with Manticore Search

We brought reverse image search to life with Manticore Search's capabilities. Below is a step-by-step guide:

Set up Manticore Search: Install and configure Manticore Search by following these instructions.

Create a table for image vectors: Define a table schema for storing image vectors:

CREATE TABLE IF NOT EXISTS <table name> (
    id bigint,
    image_path text,
    caption text,
    embeddings float_vector knn_type='hnsw' knn_dims='512' hnsw_similarity='COSINE'
)

Generate image embeddings: Leverage TinyCLIP to transform your images into vector embeddings.
Generate image embeddings: Use TinyCLIP to transform images into vector embeddings. We created a lightweight Python server to generate and retrieve these embeddings.
If you're curious about using the model with Python, it's quite straightforward and looks something like this:

from transformers import CLIPProcessor, CLIPModel, AutoProcessor, AutoModelForCausalLM
clip_model = CLIPModel.from_pretrained("wkcn/TinyCLIP-ViT-61M-32-Text-29M-LAION400M")
clip_processor = CLIPProcessor.from_pretrained("wkcn/TinyCLIP-ViT-61M-32-Text-29M-LAION400M")
try:
    image_bytes = base64.b64decode(request.image)
    image = Image.open(io.BytesIO(image_bytes))
    inputs = clip_processor(images=image, return_tensors="pt")
    with torch.no_grad():
        image_features = clip_model.get_image_features(**inputs)
    print(image_features.squeeze().tolist())
except Exception as e:
    raise HTTPException(status_code=400, detail=f"Error processing image: {str(e)}")

Index the image vectors: Import your vector data into the Manticore Search table.
First, process each image to obtain its text embedding representation, then store these vectors in your Manticore Search table. This crucial step transforms your visual data into searchable numerical values, making them ready for efficient retrieval during searches.
Look at this script that we developed to import data into our demo using the Python server API mentioned earlier. All you need to do is process your image collection through the model, which will generate vectors that can then be stored in Manticore Search.

Implement image search: Integrate Manticore Search's vector capabilities to find visually similar images.
Here's a practical implementation and basic workflow demonstrating what you can achieve with Manticore. For simplicity, some details have been omitted, but you can find the complete code in our repository.

// Upload image and get embeddings first
$image = Image::upload($file['tmp_name'])->unwrap();
$embeddings = $Embed->getImageEmbeddings($image->getPath())->unwrap();
// Search with Manticore Search
$client = new Manticoresearch\Client(config('manticore'));
$query = new Manticoresearch\Query\KnnQuery('embeddings', $embeddings, 10);
$docs = $client->index('image')->search($query)->get();
// Finally do something with images found
foreach ($docs as $doc) {
    $row = ['id' => (int)$doc->getId(), ...$doc->getData()];
    $items[] = $row;
}

Also, in this script you can see how we process image embeddings and perform vector similarity search using Manticore Search to retrieve relevant results.

Manticore Search Image Demo

We have developed a demo showcasing the capabilities of vector search, which you can explore at image.manticoresearch.com. This demo leverages TinyCLIP's AI model to transform images into vectors and perform fast similarity searches.

Key Features

Reverse Image Search (Image-to-Image Search): Upload or link an image to find visually similar content.
Text-to-Image Search: Use text descriptions to find relevant images. This feature is made possible by TinyCLIP's ability to create text embeddings that match image embeddings, offering effective cross-modal retrieval. While this article focuses on reverse image search, we will create a separate article dedicated to text-to-image search in the future.
Efficient CPU Processing: TinyCLIP delivers fast results without needing specialized hardware.

Applications of Reverse Image Search

Reverse image search unlocks many possibilities:

E-commerce: Reverse image search can help customers find visually similar products by uploading a photo. For example, if a customer sees a dress they like but cannot find it online, they can upload a picture of it, and the system will show them visually similar dresses available for purchase. This feature helps improve product discovery and enhance the shopping experience.
Content Management: Identifying duplicate images is a major challenge for organizations with large content libraries. Reverse image search allows content managers to quickly identify and remove redundant images, optimize storage space, and ensure efficient content management. It also helps track unauthorized usage of images across different platforms, maintaining copyright compliance.
Recommendations: Providing visually relevant suggestions helps enhance user engagement by making it easy for users to discover related content. For instance, a user who uploads a photo of a particular piece of furniture could be shown similar items that match the style or color, making it easier for them to complete a set or explore similar options. This capability can be applied to fashion, home decor, and many other sectors to deliver a more intuitive and visually driven recommendation experience.

Conclusion and Future Directions

Reverse image search has come a long way, evolving from simple color matching to advanced vector-based similarity analysis. With models like TinyCLIP and Manticore Search, building a reverse image search system is now feasible for developers of all scales.

Our image search demo available at image.manticoresearch.com lets you explore these technologies firsthand. Whether you're looking to add visual search to your app or are simply curious, this demo is a great place to start exploring the power of modern reverse image search.