DEV Community

Christopher Wray
Christopher Wray

Posted on • Edited on • Originally published at chriswray.dev

How to Create a Sitemap in Laravel for a Website that Contains Thousands of Records.

#laravel #php #seo #tutorial

In Laravel, if you want to build a sitemap, there are several packages built that can help you with smaller websites. One of them is the popular Spatie Sitemap package, which crawls your site when you ask it to and builds a sitemap for you based on the pages your frontend is linked to.

For smaller websites that only contain a few hundred pages, these packages will work great, and I recommend checking them out.

For a larger website with thousands of pages, you will not want to use them.

In my case, I was working on the Nursery People website a few months ago, and I wanted to build a sitemap for the site to help Google and other Search Engines see the pages on the site more easily. The Nursery People website has well over 30,000 pages, so I learned quickly that sitemap crawlers were not able to get the job done efficiently.

This is when I learned from a Laracast discussion that the best way to go about building a sitemap for a site with a lot of records is to build a sitemap that generates markup for various collections based on records in Alphabetical order.

In this article, I'll share how you can build a sitemap for a large website like I did for the Nursery People site.

Getting Started

First, you need to have a running Laravel application set up. For this tutorial, I created a new Laravel 8 application which you can fork on Github if you would like to follow along with the code, or you can just build this in your application.

For this example, I built a simple Model, Controller, and Migration for Posts. Of course, in your app, you will have more models, but for this example, I kept it simple.

I also used the Inertia Jetstream UI scaffolding so that you can see how this works with an updated Laravel 8 SPA. This isn't necessary, and you can build this just fine with the traditional Laravel UI.

I won't go into the details of creating the Post functionality or setting up Jetstream. You can read in other articles how to do that.

Creating a Sitemap Controller

Once your application is built, you will want to build routing for your Sitemaps.

In our app, we will generate two controllers for the website but for your app, you may want to generate several controllers.

For example, you may want to create a PostSitemapController for your Post model, and a TagSitemapController for Tags.

Generate the SitemapController

In your terminal:

php artisan make:controller SitemapController
Enter fullscreen mode Exit fullscreen mode

Add the Index Route

In routes/web.php add a route for the main entry point of your sitemap.

//Sitemap Routes
Route::get('/sitemap.xml', [SitemapController::class, 'index'])->name('sitemap.index');
Enter fullscreen mode Exit fullscreen mode

This will be the main sitemap that will contain links to the other sitemaps on the site.

We are also giving our sitemap a name so that we can easily route to it in our blade file.

Write the index() method in the SitemapController

Next, you will want to create the index method that we just referenced in the web.php route file.

For my app, the controller looks like this:

<?php

namespace App\Http\Controllers;

use App\Models\Post;
use Illuminate\Http\Request;

class SitemapController extends Controller
{
    public function index()
    {
        $post = Post::orderBy('updated_at', 'desc')->first();

        return response()->view('sitemap.index', [
            'post' => $post,
        ])->header('Content-Type', 'text/xml');
    }
}
Enter fullscreen mode Exit fullscreen mode

As you may notice, we are getting the latest post in this method. I will share why in a moment.

Also, we return a blade view with a header of "Content-Type", "text/xml". This is how we tell out app that we are returning an XML file, not an HTML page.

Build the XML Blade View

Create a sitemap folder in your /resources/views directory and create a file named index.blade.php.

Inside this file, add the following. You can also see how I am using the $post that I got inside our index method here to show the last time our sitemap was updated.

<?php echo '<?xml version="1.0" encoding="UTF-8"?>'; ?>

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    @if($post != null)
        <sitemap>
            <loc>{{ route('sitemap.posts.index') }}</loc>
            <lastmod>{{ $post->updated_at->tz('UTC')->toAtomString() }}</lastmod>
        </sitemap>
    @endif
</sitemapindex>
Enter fullscreen mode Exit fullscreen mode

Since this is the entry point to your other sitemaps, you can add as many links to other sitemaps as you would like. These will be generated in the next step.
In my simple example, I am only creating one sitemap index for my posts.

Now, my sitemap looks like this when visiting the /sitemap.xml link:
sitemap-project.png

Create a Post Sitemap Controller

In your terminal:

php artisan make:controller PostSitemapController
Enter fullscreen mode Exit fullscreen mode

This is the sitemap controller I will use for posts.

Create a route for the posts sitemap

Add a second route to your routes/web.php file that will go to the sitemap that we just linked to in the blade file.

//Sitemap Routes
Route::get('/sitemap.xml', [SitemapController::class, 'index'])->name('sitemap.index');
Route::get('/sitemap/posts.xml', [PostSitemapController::class, 'index'])->name('sitemap.posts.index');
Enter fullscreen mode Exit fullscreen mode

We added another route and name for easy linking.

Add the Index method to PostSitemapController

In this method, we will need to get a list of letters in the alphabet. That way we can build sitemaps for posts based on which letter of the alphabet they begin with. This isn't necessary for small collections, but in our case where we want to generate thousands of routes, this is a good way to separate content so that it is limited.

<?php

namespace App\Http\Controllers;

class PostSitemapController extends Controller
{
    public function index()
    {
        $alphas = range('a', 'z');

        return response()->view('sitemap.posts.index', [
            'alphas' => $alphas,
        ])->header('Content-Type', 'text/xml');
    }
}
Enter fullscreen mode Exit fullscreen mode

We are returning the array of the alphabet to the blade view.

Create the post index blade view.

Now, in the /resources/views/sitemap directory create another folder for posts and add an index.blade.php file to it.

You will use the $alphas variable to loop through each character of the alphabet and generate a sitemap to posts starting with that character.

<?php echo '<?xml version="1.0" encoding="UTF-8"?>'; ?>

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    @foreach($alphas as $alpha)
        <sitemap>
            <loc>{{ route('sitemap.posts.show', $alpha) }}</loc>
        </sitemap>
    @endforeach
</sitemapindex>
Enter fullscreen mode Exit fullscreen mode

See that since we created a route name, it is easy to use the $alpha character as the route param.

Add the Final Route and Controller Method

Sweet! We are almost done. Now, add a route to the /sitemap/posts/{$letter}.xml in your routes/web.php file.

//Sitemap Routes
Route::get('/sitemap.xml', [SitemapController::class, 'index'])->name('sitemap.index');
Route::get('/sitemap/posts.xml', [PostSitemapController::class, 'index'])->name('sitemap.posts.index');
Route::get('/sitemap/posts/{letter}.xml', [PostSitemapController::class, 'show'])->name('sitemap.posts.show');
Enter fullscreen mode Exit fullscreen mode

We are using the show() method here that will show posts based on the letter in the URL.

Post Sitemap Controller 

<?php

namespace App\Http\Controllers;

use App\Models\Post;

class PostSitemapController extends Controller
{
    public function index()
    {
        $alphas = range('a', 'z');

        return response()->view('sitemap.posts.index', [
            'alphas' => $alphas,
        ])->header('Content-Type', 'text/xml');
    }

    public function show($letter){
        $posts = Post::where('title', 'LIKE', "$letter%")->get();

        return response()->view('sitemap.posts.show', [
            'posts' => $posts,
        ])->header('Content-Type', 'text/xml');
    }
}
Enter fullscreen mode Exit fullscreen mode

Here, we are returning the $posts in that range to our view.

Create a show blade view.

Now, create the final sitemap needed for this app. In resources/views/sitemap/posts create a new file: show.blade.php.

<?php echo '<?xml version="1.0" encoding="UTF-8"?>'; ?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    @foreach ($posts as $post)
        <url>
            <loc>{{ route('posts.show', $post->id) }}</loc>
            <lastmod>{{ $post->updated_at->tz('UTC')->toAtomString() }}</lastmod>
            <changefreq>weekly</changefreq>
            <priority>0.6</priority>
        </url>
    @endforeach
</urlset>
Enter fullscreen mode Exit fullscreen mode

On this page, we are looping through each of the post pages and generating the URL to the post page!

Conclusion

Well, that is how you generate a custom sitemap for a website with thousands of records. Make sure to build as many sitemaps/controllers as you need, and make sure that each one is linked to from the main sitemap.

Hope you enjoy this and can help you as you are building your next website.

Extra!

Make sure to go and submit your sitemap in the Google Search Console as soon as your done, and Google will then know it exists.

I originally posted this article on my website here.

Top comments (8)

Collapse
 
lito profile image
Lito • • Edited

The idea is great!

Here my version with months instead alphabet:

# Routes
Route::get('/sitemap.xml', SitemapIndexController::class)->name('sitemap.index');
Route::get('/sitemap/posts.xml', SitemapPostsIndexController::class)->name('sitemap.posts.index');
Route::get('/sitemap/posts/{date}.xml', SitemapPostsShowController::class)->name('sitemap.posts.show');
Enter fullscreen mode Exit fullscreen mode 
<?php declare(strict_types=1)

namespace App\Http\Controllers;

use App\Models\Post;
use Illuminate\Http\Response;

class SitemapIndexController extends Controller
{
    public function __invoke(): Response
    {
        $post = Post::select('published_at')
            ->orderBy('published_at', 'DESC')
            ->first();

        return response()->view('sitemap.index', [
            'post' => $post,
        ])->header('Content-Type', 'text/xml');
    }
}
Enter fullscreen mode Exit fullscreen mode 
<?xml version="1.0" encoding="UTF-8"?>

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    @if ($post)
    <sitemap>
        <loc>{{ route('sitemap.posts.index') }}</loc>
        <lastmod>{{ $post->updated_at->tz('UTC')->toAtomString() }}</lastmod>
    </sitemap>
    @endif
</sitemapindex>
Enter fullscreen mode Exit fullscreen mode 
<?php declare(strict_types=1)

namespace App\Http\Controllers;

use App\Models\Post;
use Illuminate\Http\Response;

class SitemapPostsIndexController extends Controller
{
    public function __invoke(): Response
    {
        $months = Post::selectRaw('DATE_FORMAT(`published_at`, "%Y-%m") AS `date`, MAX(`published_at`) AS `published_at`')
            ->groupBy('date')
            ->get();

        return response()->view('sitemap.posts.index', [
            'months' => $months,
        ])->header('Content-Type', 'text/xml');
    }
}
Enter fullscreen mode Exit fullscreen mode 
<?xml version="1.0" encoding="UTF-8"?>

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    @foreach ($months as $month)
    <sitemap>
        <loc>{{ route('sitemap.posts.show', $month->date) }}</loc>
        <lastmod>{{ $month->published_at->tz('UTC')->toAtomString() }}</lastmod>
    </sitemap>
    @endforeach
</sitemapindex>
Enter fullscreen mode Exit fullscreen mode 
<?php declare(strict_types=1)

namespace App\Http\Controllers;

use App\Models\Post;
use Illuminate\Http\Request;
use Illuminate\Http\Response;

class SitemapPostsShowController extends Controller
{
    public function __invoke(Request $request): Response
    {
        $posts = Post::select('id', 'published_at')
            ->whereRaw('DATE_FORMAT(`published_at`, "%Y-%m") = ?', [$request->input('date')])
            ->get();

        return response()->view('sitemap.posts.show', [
            'posts' => $posts,
        ])->header('Content-Type', 'text/xml');
    }
}
Enter fullscreen mode Exit fullscreen mode 
<?xml version="1.0" encoding="UTF-8"?>

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    @foreach ($posts as $post)
    <url>
        <loc>{{ route('posts.show', $post->id) }}</loc>
        <lastmod>{{ $post->updated_at->tz('UTC')->toAtomString() }}</lastmod>
        <changefreq>weekly</changefreq>
        <priority>0.6</priority>
    </url>
    @endforeach
</urlset>
Enter fullscreen mode Exit fullscreen mode

Thanks for sharing!

Collapse
 
cwraytech profile image
Christopher Wray •

I really like your method! Thank you!

Collapse
 
l3lackheart profile image
l3lackheart •

nice approach :O will try it on my next website

Collapse
 
cwraytech profile image
Christopher Wray • • Edited

Thanks! Yeah. I think you probably will want to set up caching as well so that this isn't generated every time on the fly! That is one thing I didn't cover in this article.

Collapse
 
l3lackheart profile image
l3lackheart •

set up catching? you mean caching?
if so then yes, will keep notice for that :3

Thread Thread
 
cwraytech profile image
Christopher Wray •

Haha yeah 😂😂😂

Collapse
 
manzadey profile image
Andrey Manzadey •

Good idea, but there is a better option to avoid loading the database.

Use the \DomDocument class to generate a sitemap.xml and thus make the sitemap.xml static. And use the laravel scheduler.

Collapse
 
cwraytech profile image
Christopher Wray •

Love it. Still be using a controller?

Read next

mematthew123 profile image

How To: Interactive Pricing Component with Tailwind CSS

mematthew123 -

sarthak_niranjan_3bd6cb5f profile image

The 'this' Keyword in JavaScript: A Beginner’s Guide

Sarthak Niranjan -

vigneshiyergithub profile image

Exploring the ECMAScript 2024 Updates: A Beginner’s Guide

vigneshiyergithub -

mshsayket profile image

How to Calculate the Average of a Column Using Eloquent

Saddam Hossain -