Building a Web Search Engine in Go with Elasticsearch

Web search engines are essential for indexing vast amounts of online information, making it accessible in milliseconds. In this project, I built a search engine in Go (Golang) named RelaxSearch. It combines web scraping, periodic data indexing, and search functionality by integrating with Elasticsearch—a powerful search and analytics engine. In this blog, I’ll walk you through the main components of RelaxSearch, the architecture, and how it efficiently scrapes and indexes data for fast, keyword-based search.

Overview of RelaxSearch

RelaxSearch is built around two primary modules:

1. RelaxEngine: A web scraper powered by cron jobs, which periodically crawls specified websites, extracts content, and indexes it in Elasticsearch.
2. RelaxWeb: A RESTful API server that allows users to search the indexed data, providing pagination, filtering, and content highlighting for user-friendly responses.

Project Motivation

Creating a search engine project from scratch is a great way to understand web scraping, data indexing, and efficient search techniques. I wanted to create a simple but functional search engine with fast data retrieval and easy extensibility, utilizing Go’s efficiency and Elasticsearch’s powerful indexing.

Key Features

• Automated Crawling: Using cron jobs, RelaxEngine can run at regular intervals, scraping data and storing it in Elasticsearch.
• Full-Text Search: RelaxWeb provides full-text search capability, indexing content by keywords, making retrieval fast.
• REST API: Accessible through a RESTful API with parameters for pagination, date filtering, and content highlighting.
• Data Storage: The indexed content is stored in Elasticsearch, allowing for scalable and highly responsive queries.

Architecture of RelaxSearch

1. RelaxEngine (Web Scraper and Indexer)

RelaxEngine is a web scraper written in Go that navigates web pages, extracting and storing content. It runs as a cron job, so it can operate at regular intervals (e.g., every 30 minutes) to keep the index updated with the latest web data. Here’s how it works:

• Seed URL: RelaxEngine starts scraping from a specified seed URL and then follows links within the site up to a configurable depth.
• Content Parsing: For each page, it extracts titles, descriptions, and keywords, constructing an informative dataset.
• Indexing in Elasticsearch: The scraped content is indexed in Elasticsearch, ready for full-text search. Each page's data is stored with a unique identifier, title, description, and other metadata.

2. RelaxWeb (Search API)

RelaxWeb provides a RESTful API endpoint, making it easy to query and retrieve data stored in Elasticsearch. The API accepts several parameters, such as keywords, pagination, and date filtering, returning relevant content in JSON format.

• API Endpoint: /search
• Query Parameters:
  • keyword: Main search term.
  • from and size: Pagination control.
  • dateRangeStart and dateRangeEnd: Filter results based on the timestamp of data.

Key Components and Code Snippets

Below are some important components and code excerpts from RelaxSearch to illustrate how it works.

Main Go Code for RelaxEngine

The core functionality is in the main.go file, where RelaxEngine initializes a scheduler using gocron to manage cron jobs, sets up the Elasticsearch client, and begins crawling from the seed URL.

func main() {
    cfg := config.LoadConfig()
    esClient := crawler.NewElasticsearchClient(cfg.ElasticsearchURL)
    c := crawler.NewCrawler(cfg.DepthLimit, 5)
    seedURL := "https://example.com/" // Replace with starting URL

    s := gocron.NewScheduler(time.UTC)
    s.Every(30).Minutes().Do(func() {
        go c.StartCrawling(seedURL, 0, esClient)
    })
    s.StartBlocking()
}

Crawler and Indexing Logic

The crawler.go file handles web page requests, extracts content, and indexes it. Using the elastic package, each scraped page is stored in Elasticsearch.

func (c *Crawler) StartCrawling(pageURL string, depth int, esClient *elastic.Client) {
    if depth > c.DepthLimit || c.isVisited(pageURL) {
        return
    }
    c.markVisited(pageURL)
    links, title, content, description, err := c.fetchAndParsePage(pageURL)
    if err == nil {
        pageData := PageData{URL: pageURL, Title: title, Content: content, Description: description}
        IndexPageData(esClient, pageData)
    }
    for _, link := range links {
        c.StartCrawling(link, depth+1, esClient)
    }
}

Search API Code in RelaxWeb

In relaxweb service, an API endpoint provides full-text search capabilities. The endpoint /search receives requests and queries Elasticsearch, returning relevant content based on keywords.

func searchHandler(w http.ResponseWriter, r *http.Request) {
    keyword := r.URL.Query().Get("keyword")
    results := queryElasticsearch(keyword)
    json.NewEncoder(w).Encode(results)
}

Setting Up RelaxSearch

1. Clone the Repository

   git clone https://github.com/Ravikisha/RelaxSearch.git
   cd RelaxSearch

1. Configuration
   
   Update .env files for both RelaxEngine and RelaxWeb with Elasticsearch credentials.

2. Run with Docker
   
   RelaxSearch uses Docker for easy setup. Simply run:
   

   docker-compose up --build

Challenges and Improvements

• Scalability: Elasticsearch scales well, but handling extensive scraping with numerous links requires optimizations for larger-scale deployments.
• Robust Error Handling: Enhancing error handling and retry mechanisms would increase resilience.

Conclusion

RelaxSearch is an educational and practical demonstration of a basic search engine. While it is still a prototype, this project has been instrumental in understanding the fundamentals of web scraping, full-text search, and efficient data indexing with Go and Elasticsearch. It opens avenues for improvements and real-world application in scalable environments.

Explore the GitHub repository to try out RelaxSearch for yourself!