Integrate IndexNow with Umbraco CMS to submit content to search engines

One of Umbraco's superpowers has always been how easy it is to extend the CMS as a .NET web developer. As an ASP.NET developer, you can develop your websites on top of Umbraco's content database, but you can also extend the CMS itself and its backoffice. The backoffice is Umbraco's web interface for managing content, developing the website, and other administrative tasks. You can empower the backoffice users by adding more functionality, making it the central hub for managing your website. For example, I added Google Analytics to the backoffice, capabilities to clear Cloudflare's cache for specific Umbraco content, and just recently, I added IndexNow integration to easily submit content to search engines for crawling.

In this tutorial, you'll learn how to integrate IndexNow, so you can easily submit Umbraco content URLs to search engines that they will (re)crawl. Along the way you'll also learn how to:

1. hook into Umbraco's startup
2. extend the backoffice menu's
3. add JavaScript to the backoffice
4. add an API controller for authenticated backoffice users
5. submit an Umbraco content URL to IndexNow

The result of the IndexNow integration will look like this:

I spent this afternoon adding IndexNow support for Bing and Yandex to my Umbraco website.

Will write a blog post about it ... someday 🙈#dotnet #aspnet #umbraco pic.twitter.com/GyxxB03Ewi

— Niels Swimburger.NET 🍔 (@RealSwimburger) January 30, 2022

What's IndexNow?

IndexNow allows software developers to work with a single simple protocol, for all search engines who adopt it, to requests search engines to crawl your website's URLs.

What's more, when you submit URLs to one search engine, that search engine will inform all other search engines with IndexNow support. That means you only have to make an API call to one IndexNow service, not to every search engine's IndexNow service.

How to implement IndexNow

1. Create a key between 8 and 128 hexadecimal characters. Allowed characters: lowercase characters (a-z), uppercase characters (A-Z), numbers (0-9), and dashes (-). You can choose this key yourself, or you can generate a valid key on Bing's IndexNow site for your convenience.
2. Upload a text file to the root of your website. The name of the file is your key with the ".txt" extension , and the contents of the file is also your key.
3. Submit HTTP requests to any IndexNow service:
   • IndexNow.org: https://api.indexnow.org/indexnow
   • Bing IndexNow: https://www.bing.com/indexnow
   • Yandex IndexNow: https://yandex.com/indexnow

You can send HTTP requests 2 ways. The first and simplest way is to send a GET HTTP request with two querystring parameters:

• url: this is the URL you want search engines to crawl
• key: this is the key you generated earlier

This will submit a single URL to IndexNow.

To send multiple URLs, you can use the second way. The second way is to send a POST HTTP request with the following JSON body:

{
    "host": "example.org",
    "key": "db132248f7cb4d309ee9d2aa54131af6",
    "keyLocation": "https://example.org/db132248f7cb4d309ee9d2aa54131af6.txt",
    "urlList": [
        "https://example.org/url1",
        "https://example.org/folder/url2",
        "https://example.org/url3"
    ]
}

Here's what the JSON properties should be:

• host: The domain name of your website. The host must match the URLs you are submitting.
• key: The IndexNow key you generated.
• keyLocation: Where you host your key text file. This property is optional if you store the key text file at the root of your website.
• urlList: An array of URLs you want to submit to search engines for crawling.

Read this post to learn more about IndexNow and why you should care as a web developer, content editor, marketeer, or webmaster.

Prerequisites

To follow along, you'll need the following things:

• Windows OS
• .NET Framework
• A .NET framework friendly IDE (Visual Studio, JetBrains Rider, etc.)
• An Umbraco 8 website (this tutorial used the NuGet installation method with the default starter kit)

You can find the source code for the IndexNow integration on this GitHub repository. If you run into any problems, feel free to submit an issue or reach out through social media.

This tutorial uses Umbraco 8 which only supports Windows using the .NET Framework, but what you'll learn here you can also apply to Umbraco 9 which use .NET (Core) and supports Windows, macOS, and Linux.

Hooking into Umbraco's startup

You'll need to be able to run code at Umbraco's startup to extend the backoffice menu.

Open your Umbraco website using an editor or IDE and create a new folder called IndexNow.

In this folder, add a new C# file called IndexNowComposer.cs with the following contents:

using Umbraco.Core;
using Umbraco.Core.Composing;

namespace UmbracoIndexNow.IndexNow
{
    public class IndexNowComposer : IUserComposer
    {
        public void Compose(Composition composition) => composition.Components().Append<IndexNowComponent>();
    }
}

During startup, Umbraco will scan for all composers and run them. Composers will add components and thus running all composers will create a composition of components.

The composition of components is how Umbraco configures itself, and where you can add new components or replace existing components of Umbraco.

The IndexNowComposer will append the IndexNowComponent to the list of existing components.

The IndexNowComponent does not exist yet, so create a new file at IndexNow/IndexNowComponent.cs with the following content:

using Umbraco.Core.Composing;

namespace UmbracoIndexNow.IndexNow
{
    public class IndexNowComponent : IComponent
    {
        public void Initialize() { }

        public void Terminate() { }
    }
}

In the component, there are two methods: Initialize and Terminate. The Initialize method executes during Umbraco's startup which is when you can run code to extend Umbraco.

The Terminate method runs when the application is stopped, and this will allow you to cleanup whatever resources you created during initialization.

Extending the Umbraco backoffice menu

To extend the backoffice menu, you actually need to do something in the IndexNowComponent, so update the IndexNow/IndexNowComponent.cs file as below:

using Umbraco.Core.Composing;
using Umbraco.Web.Models.Trees;
using Umbraco.Web.Trees;

namespace UmbracoIndexNow.IndexNow
{
    public class IndexNowComponent : IComponent
    {
        public void Initialize() => TreeControllerBase.MenuRendering += OnMenuRendering;

        public void Terminate() => TreeControllerBase.MenuRendering -= OnMenuRendering;

        private void OnMenuRendering(TreeControllerBase sender, MenuRenderingEventArgs e)
        {
            if (sender.TreeAlias != "content")
            {
                return;
            }

            var content = sender.Umbraco.Content(int.Parse(e.NodeId));
            if (content == null)
            {
                return;
            }

            var contentType = content.ContentType;
            var compositions = contentType.CompositionAliases;
            if (compositions.Contains("contentBase") ||
                compositions.Contains("navigationBase") ||
                contentType.Alias == "home")
            {
                var i = new MenuItem("submitToIndexNow", "Submit To IndexNow");
                i.ExecuteJsMethod("submitToIndexNow(node)");
                i.Icon = "share-alt";
                e.Menu.Items.Add(i);
            }
        }
    }
}

Now the IndexNowComponent registers a callback on the TreeControllerBase.MenuRendering event during initialization and unregisters the callback during termination.

Every time a user opens the menu for a tree in the backoffice, the OnMenuRendering method will be called. For example, when you right-click on a content node in the content tree.

Though, this method will also be called for other trees in the backoffice such as settings, members, etc.

Since you're only interested in changing the menu in the content tree, the code above checks if this is the 'content' tree and immediately backs out if it is not.

Now that you know you're in the content tree, you know that the NodeId will point to Umbraco content. Thus, you can fetch the Umbraco content by the NodeId.

Although not all content nodes in the tree will be published. If the content isn't published, the content variable will be null. If so, you can safely back out as it doesn't make sense to submit unpublished content to IndexNow.

Which leads you to the question, what type of content do you want to submit to IndexNow? That depends on how you built your content architecture.

For the starter kit, only content that either is composed of the contentBase or navigationBase composisition makes sense to submit to IndexNow. The only exception being the home page, which doesn't have compositions for some reason.

Now that you've verified the selected content makes sense to submit to IndexNow, you can finally add a menu item to the menu.

You can configure the menu item to do different things like open a dialog, navigate to a route, or execute a JavaScript function.

As a lazy developer, the easiest option seemed to execute a JavaScript function. When this menu item is clicked, a global JavaScript function will be invoked named submitToIndexNow.

If you want to do something else with menus and tree, check out the Umbraco docs for extending trees.

Add JavaScript to submit Umbraco Node to web API

Interestingly, Umbraco will use the eval-function to evaluate the JavaScript string passed to ExecuteJsMethod and run it. This also means you have access to whatever variables are within scope when Umbraco runs your JavaScript expression.

Most importantly, there is a node variable which holds the details about the selected node in the tree. That's why the node variable is passed as an argument to the submitToIndexNow function.

Create a new folder named IndexNow under App_Plugins, and then create a new JavaScript file IndexNow.js with the following contents:

function submitToIndexNow(node) {
  fetch(`/umbraco/backoffice/api/IndexNow/Submit?nodeId=${node.id}`,
    {
      method: 'post',
    })
    .then(response => response.json())
    .then(response => {
      if (response.success === true) {
        UmbSpeechBubble.ShowMessage('success', 'IndexNow', 'Submitted successfully');
      }
      else {
        console.log(response.errorMessage);
        UmbSpeechBubble.ShowMessage('error', 'IndexNow', response.errorMessage);
      }
    })
    .catch(err => {
      console.log(err);
      UmbSpeechBubble.ShowMessage('error', 'IndexNow', 'Submit failed');
    });
}

The submitToIndexNow sends an HTTP request to /umbraco/backoffice/api/IndexNow/Submit passing in the id of the selected tree node. The application currently doesn't listen to this endpoint, but later on the HTTP request will be handled by the Submit method on the IndexNowController.

The response from this controller will be a JSON object with two properties: success and errorMessage.

success is a boolean indicating whether the content was successfully submitted to IndexNow, and errorMessage will provide a user-friendly message if success is false.

In both cases, a little toast will be displayed to the user by calling the UmbSpeechBubble.ShowMessage function provided by Umbraco.

In case of an error, the error is logged to the console and a toast will also be shown with a generic failure message.

For Umbraco to actually load this JavaScript, you need to create a plugin manifest. Create a new file name package.manifest with the contents below:

{
  "javascript": [
    "/App_Plugins/IndexNow/IndexNow.js"
  ]
}

With these two files, Umbraco will start including your JavaScript function in the backoffice.

Add backoffice API controller to submit URL to IndexNow

You'll need to create a controller to handle the HTTP request made from the submitToIndexNow function.

Create a new file named IndexNowController.cs in the IndexNow folder (not App_Plugins/IndexNow) with the following contents:

using Newtonsoft.Json;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
using System.Web.Http;
using Umbraco.Core.Logging;
using Umbraco.Core.Models.PublishedContent;
using Umbraco.Web;
using Umbraco.Web.WebApi;

namespace UmbracoIndexNow.IndexNow
{
    public class IndexNowController : UmbracoAuthorizedApiController
    {
        public static readonly HttpClient httpClient = new HttpClient();
        private readonly ILogger logger;

        public IndexNowController(ILogger logger)
        {
            this.logger = logger;
        }

        [HttpPost]
        public async Task<object> Submit(int nodeId)
        {
            // get URL of the Umbraco content
            var nodeUrl = Umbraco.Content(nodeId).Url(mode: UrlMode.Relative);

            // create IndexNow object
            var indexNowObject = new
            {
                host = IndexNowConfiguration.Host,
                key = IndexNowConfiguration.Key,
                keyLocation = IndexNowConfiguration.KeyLocation,
                urlList = new string[]
                {
                    $"{IndexNowConfiguration.BaseUrl}{nodeUrl}"
                }
            };

            // convert IndexNow object to JSON string
            var jsonString = JsonConvert.SerializeObject(indexNowObject);

            // send JSON string to IndexNow service
            var content = new StringContent(jsonString, Encoding.UTF8, "application/json");
            var response = await httpClient.PostAsync(IndexNowConfiguration.Url, content);

            // if response is status code 200, return object with `success` = true
            if (response.IsSuccessStatusCode)
            {
                return new { success = true };
            }

            // if response statuscode is not 200
            // log an error message
            logger.Error<IndexNowController>(
                "Submitting URL to {IndexNowUrl} failed, status code: {StatusCode}",
                IndexNowConfiguration.Url,
                response.StatusCode
            );
            // log the response body as error for debugging purposes
            var responseContent = await response.Content.ReadAsStringAsync();
            logger.Error<IndexNowController>("IndexNow response: {IndexNowResponse}", responseContent);

            // return object with `success` = false and a user-friendly error message
            return new
            {
                success = false,
                errorMessage = $"Submitting URL to {IndexNowConfiguration.Url} failed, status code: {response.StatusCode}"
            };
        }
    }
}

By inheriting from UmbracoAuthorizedController, this controller can only be used by authenticated backoffice users and the default routing will be /umbraco/backoffice/api/[controller]/[action] which will resolve to /umbraco/backoffice/api/IndexNow/Submit in this case.

The Submit action receives the nodeId which was passed as a querystring parameter and uses it to fetch the relative URL for the Umbraco content. The relative URL is combined with a base URL to create an absolute URL. The base URL comes from the IndexNowConfiguration class which will be introduced next.

Next, an anonymous object is created following the IndexNow JSON schema using the absolute URL and more properties coming from the IndexNowConfiguration class.

The object is converted to a JSON string and submitted using an HTTP POST request to the IndexNow service.

If the response returns an HTTP status code 200, a JSON object {"success" = true} is returned. Otherwise, an error message is logged, the response body is logged, and the action will return {"success" = false, "errorMessage" = "..."}.

The last missing piece is the IndexNowConfiguration class. Create a new file IndexNowConfiguration.cs in the IndexNow folder (not App_Plugins/IndexNow) with the following contents:

using System.Configuration;

namespace UmbracoIndexNow.IndexNow
{
    public static class IndexNowConfiguration
    {
        public static string BaseUrl => ConfigurationManager.AppSettings["IndexNow.BaseUrl"];
        public static string Url => ConfigurationManager.AppSettings["IndexNow.Url"];
        public static string Host => ConfigurationManager.AppSettings["IndexNow.Host"];
        public static string Key => ConfigurationManager.AppSettings["IndexNow.Key"];
        public static string KeyLocation => ConfigurationManager.AppSettings["IndexNow.KeyLocation"];
    }
}

This static class will resolve all of its static properties using the ConfigurationManager.AppSettings. The app settings are populated by the Web.config file in your project.

Add the configuration to your Web.config file like this:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <!-- existing configuration omitted for brevity -->

  <appSettings>
    <!-- existing appSettings omitted for brevity -->

    <add key="IndexNow.BaseUrl" value="[YOUR_WEBSITE_BASE_URL]" />
    <add key="IndexNow.Url" value="https://api.indexnow.org/indexnow" />
    <add key="IndexNow.Host" value="[YOUR_WEBSITE_HOST_NAME]" />
    <add key="IndexNow.Key" value="[YOUR_INDEX_NOW_API_KEY]" />
    <add key="IndexNow.KeyLocation" value="[URL_TO_YOUR_INDEX_NOW_API_KEY]" />
  </appSettings>

  <!-- existing configuration omitted for brevity -->
</configuration>

Here's what the properties should be configured as:

• IndexNow.BaseUrl: this will be used as the base URL to convert your relative URLs to absolute URLs. Configure this with your public website root URL, for example: https://swimburger.net
• IndexNow.Url: this is the URL for the IndexNow service you will submit your requests to. You can leave this as is, or swap it out for another IndexNow service. You can find the other URLs on IndexNow's FAQ.
• IndexNow.Key: this is the key which you need to upload as a text file to the root of your website, and also pass with every HTTP request. You can create a key yourself as explained earlier, or generate a valid key on Bing's IndexNow site for your convenience.
• IndexNow.KeyLocation: this is the URL pointing to the text file with your key in it, hosted on your public website.

There are many other ways to handle configuration, but this is a simple and efficient way.

Test your IndexNow integration

With everything in place, you can start your Umbraco website and navigate to the backoffice at /umbraco. Go to the Content tab and right-click some of the content.

A new menu item should appear saying "Submit to IndexNow". Click on the menu item and the content should be submitted to IndexNow.

Did everything work out as expected? What services have you integrated into Umbraco? I'm curious to find out, let me know!