How to Implement Dynamic Multilingual Sitemap with 40,000 URLs in Next.js App Router

#webdev #programming #sitemap #nextjs

Introduction

This article explains implementing a dynamic sitemap for a large-scale website using Next.js App Router. I'll share the implementation method for a multilingual site containing over 40,000 content items.

TL;DR

By combining Next.js sitemap API with 24-hour caching, we implemented a practical dynamic sitemap solution for large-scale sites. This approach minimizes the impact on frontend build time while enabling daily data updates.

Development Environment

Frontend: Next.js (App Router)
Backend: Express + MongoDB
Infrastructure: Vercel (Frontend), Heroku (Backend)
Languages: 2 (bilingual)

Context

Over 20,000 unique pages
Each page has 2 language variations (total: 40,000+ URLs)
Daily data updates
Sitemap required for SEO requirements

Understanding Sitemap Generation

There are three main approaches to sitemap generation:

Static Generation
- Fast response time
- Low server load
- Longer build times
- Difficult to handle immediate data updates
Dynamic Generation
- Easy reflection of latest data
- No impact on build time
- Higher server load
- Potential slower response times
ISR (Incremental Static Regeneration)
- Balanced approach between static and dynamic
- Easy cache control
- Slightly more complex implementation

I had written about basic sitemap generation in the following document:
Implementing Multilingual Sitemap with next-intl in Next.js App Router

Implementation

We adopted an implementation combining Next.js sitemap API with 24-hour caching:

// app/sitemap.ts
import { MetadataRoute } from "next";

const BASE_URL = process.env.NODE_ENV === "production" 
  ? "https://example.com"
  : "localhost:3000";

const API_BASE_URL = process.env.NODE_ENV === "production"
  ? `${process.env.NEXT_PUBLIC_API_BASE_URL}/api/util`
  : "http://localhost:3500/api/util";

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const locales = ["en", "ja"];
  const routes = ["/", "/about", "/products", "/contact"];

  // Generate static routes
  const staticRoutes = routes.flatMap((route) =>
    locales.map((locale) => ({
      url: `${BASE_URL}/${locale}${route === "/" ? "" : route}`,
      lastModified: new Date(),
      changeFrequency: "monthly" as const,
      priority: route === "/" ? 1 : 0.8,
    }))
  );

  try {
    // Fetch product data with caching
    const response = await fetch(`${API_BASE_URL}/products`, {
      next: { revalidate: 86400 }, // Cache for 24 hours
      headers: { "Cache-Control": "public, max-age=86400" },
    });

    if (!response.ok) {
      throw new Error(`Failed to fetch products: ${response.statusText}`);
    }

    const products = await response.json();

    // Generate product URLs
    const productRoutes = products.flatMap((product: any) =>
      locales.map((locale) => ({
        url: `${BASE_URL}/${locale}/product/${product.id}`,
        lastModified: new Date(product.updatedAt),
        changeFrequency: "daily" as const,
        priority: 0.7,
      }))
    );

    return [...staticRoutes, ...productRoutes];
  } catch (error) {
    console.error("Error generating sitemap:", error);
    return staticRoutes; // Fallback to static routes on error
  }
}

Backend API implementation:

// Express route handler
router.get("/products", async (req, res) => {
  try {
    // Fetch minimal required data from MongoDB
    const products = await Product.find(
      {},
      {
        id: 1,
        lastUpdateDate: 1,
      }
    ).lean();
    res.json(products);
  } catch (error) {
    console.error("Failed to fetch products:", error);
    res.status(500).json({
      message: "Error fetching products",
      error: error instanceof Error ? error.message : String(error),
    });
  }
});

Implementation Results

Total URLs: 42,000 (21,000 for en, ja)
File size: 4.6MB
Generation time: Under 5 seconds
Impact on build time: Minimal
Cache duration: 24 hours

Checking results

You can check the generated sitemap in dev environment, localhost:3000/sitemap.xml.

Key Implementation Considerations

Caching Strategy
- Use both Next.js revalidation and HTTP caching
- 24-hour cache balances freshness and performance
- Implement fallback to static routes on errors
Error Handling
- Graceful degradation to static routes
- Structured error logging
- Monitoring integration recommended
SEO Optimization
- Proper lastModified dates
- Appropriate priority settings
- Consistent language codes

Conclusion

Next.js Sitemap API proves capable of handling large-scale multilingual sites effectively. The key to success lies in implementing proper caching strategies and error handling, enabling a balance between performance and maintainability.