Cloudflare Pages and Origin Rulesets

In my current job we use Cloudflare (CF) extensively, most recently to migrate our main website written as an Angular SPA to a series of smaller apps in Remix (SSR) and next.js (SSG). To do this, we use Cloudflare Pages. It's a great way of hosting your site directly on the edge.

It would be cool if we could roll out new apps on new routes without impacting existing Angular app hosted on our mobile domain.

Scenario

All traffic for our mobile website, m.mydomain.com goes to Angular SPA hosted in Azure App Service. Product wants to deploy a brand new blog site on sub-path: m.mydomain.com/blog/.

Blog is a static site generation (SSG) app using next.js. Perfect candidate for hosting on CF Pages. I won't go into detail on how we build and deploy it - here's a link on how to use Wrangler to publish your build output to CF Pages.

Once we deploy our assets to a CF Pages project, we get a <project-name>.pages.dev hostname. Now, the main problem - how to host it on a sub-path of an existing domain?

Attempt 1 - Pages domain

You can bind a domain to your CF Pages project as described here. You need to create a proxied CNAME record:
m.mydomain.com -> my-project.pages.dev and bind it to that project. However, a proxied CNAME already exists! It points to our Azure App Service. Changing it would break the main site. This method is mostly for projects that are entirely served by CF Pages project, and not just part of it.

Attempt 2 - Origin Rules

Origin rules are part of the new ruleset engine and an evolution of Page Rules.

The main difference between origin rules and page rules is that you can use expressions in your rule condition instead of just path-matching patterns. So you can use hostname, IP, country, headers, cookies, paths, query strings etc. in your matching expression.

They are still in beta, but we use them extensively in our Production without issues - except this one! Let's hope they fix it before GA. :)

We can create an origin rule like this:

Now, let's go to https://m.mydomain.com/blog/. Oh no! 522 error. The connection did not time out by the way. I got this error instantly.

After contacting Cloudflare, this turns out to be some security feature preventing you from calling Pages from Cloudflare - even though the hostname we put in the rule belongs to our project. Technically, we do not own the .pages.dev domain - that is shared between all Cloudflare customers, but it would have been good for them to realize we use it!

I also found this doc which explains that your DNS override should be a hostname on your zone. That's OK - I can make a brand new CNAME on my zone: <project-name>.mydomain.com -> <project-name>.pages.dev.

Now update the origin rule:

Still nothing! Always 522 error. Believe me when I tell you - I tried every permutation of DNS records with proxied, not proxied, even tried where brand new DNS record was also a Pages domain like in attempt 1. Nothing worked.

Attempt 3 - Workers

We also have a Cloudflare Workers bundle purchased. Workers are a very powerful tool that allow you to write custom JavaScript to modify your requests, responses, fetch responses from different origins etc. We use it in some places to augment Cloudflare functionality where some rulesets don't work as we expect them. To date, we have been using this trick to implement our scenario:

You can create a CF worker script, read the URL of the incoming request, change the hostname and use fetch() to get it, as if it was a 3rd party URL.

export default {
  async fetch(request, env) {
      const url = new URL(request.url);
      url.host = "<my-project>.pages.dev";
      return fetch(url.toString(), request);
  }
}

You can bind a route to be handled by your worker code. This binding is similar to Page Rules - only path based matching allowed, not full expressions.

And voila! We have our blog site on m.mydomain.com/blog/ while the rest of m.mydomain.com remains the same. I had to change some domain names for the demo, but you can probably guess.

There are some disadvantages with this approach:

• You are paying extra for Worker requests on top of your throughput.
• Complex and bespoke fetch() behaviours. We put our entire mobile site through Workers for few reasons, but logic in the script becomes complicated and hard to understand.
• Performance - it seems a little silly that we need an extra hop through Workers to hop over to our own CF Pages project. This hop always adds just a little bit more milliseconds to our overall latency.

Attempt 4 - Origin Rules (again!), with a workaround

Attempt 2 got me thinking - We use origin rules successfully in other places where DNS host is overwritten. In those cases, we create a CNAME record on our zone to Azure resource hostnames. e.g.
<my-app>.mydomain.com -> <my-app>.azurewebsites.net.

Maybe Cloudflare has issues with it's own domain, but not if an external DNS resolves it. What if I create a CNAME that points to a hostname on Azure DNS that ultimately resolves to my CF Pages project hostname?

Let's create a new public Azure DNS zone. To add insult to injury, I made it a sub-domain of my Cloudflare zone: omtest-sub.mydomain.com.

Now link this sub-domain to root domain using NS records in Cloudflare:

Now I can resolve my custom domain:

PS> dig blog.omtest-sub.mydomain.com

; <<>> DiG 9.14.2 <<>> blog.omtest-sub.mydomain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10431
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;blog.omtest-sub.mydomain.com. IN    A

;; ANSWER SECTION:
blog.omtest-sub.mydomain.com. 600 IN CNAME   <my-project>.pages.dev.
<my-project>.pages.dev.     300     IN      A       188.114.96.7
<my-project>.pages.dev.     300     IN      A       188.114.97.7

;; Query time: 123 msec
;; SERVER: 10.64.140.254#53(10.64.140.254)
;; WHEN: Tue Oct 03 20:33:20 GMT Summer Time 2023
;; MSG SIZE  rcvd: 124

Last step is to put my new CNAME directly into DNS override box in the origin rule from Attempt 2. After all, it's already on mydomain.com zone!

And we're back in business..

I assume this would also work if you used another DNS provider like NS1 or Route53. You obviously have to pay for external DNS resolution. Azure charges by millions of queries, but it's cheaper than having to go with Workers. Ultimately, I hope they fix this problem, so we can directly use .pages.dev domains in origin rules.