Web application firewall on Netlify for free

Talented folks built Netlify to reduce development friction. They offer a generous free tier, which is super helpful for the open-source ecosystem. But Netlify needs resources to grow and stay healthy. They offer paid plans and have also hired a special force called sales.

Sales operate strategically. They target areas rich in special kind of prey called enterprise. Like experienced hunters, they conserve energy and wait at the watering hole called security.

The internet is a wild place. And I’ve said it before — please bear with me. I like to explore the mechanics of Newtonian action and reaction in daily life.

Openness and freedom create incredible opportunities for growth. And with great value comes opportunistic behavior. I personally see it as a balancing act and enjoy observing the laws of physics in many forms.

Startup perspective

Last year, I moved all my web apps from an AWS self-managed Kubernetes cluster to Netlify. Since then, I’ve saved 800€ on infrastructure bills and eliminated some complexity. In the process, I did something unusual: I tried an upsell product — GDPR-friendly site analytics for $9/month.

Privacy-friendly observability is a fair deal for me. It offers insights without compromising on core values. I was surprised and genuinely puzzled by the results.

It turned out that Valisa attracted thousands of daily visitors. Most came from Singapore and the US. That was strange because my service focused on European flights. It sparked my curiosity to dig deeper.

I inspected the logs and noticed distinct visitor groups: aggressive crawlers, credential harvesters, vulnerability scanners, reasonable crawlers, and legitimate users. Bad bots, good bots, and humans. My initial reaction was to wait and see.

Bad bots

A few weeks in, I received an alert from Netlify. It warned me that more than half of my server functions quota (125k per month) had been used. Frankly, I like the reality in which resources are limited.

In my experience, the most annoying are credential harvesters and vulnerability scanners. They fire 20-30 concurrent requests per second for around five minutes. I guess we can call it a baby DDoS.

Reckless bots are the mosquitoes of the net. They get particularly active around holidays. 5k requests per day is a common occurrence, and their favorite food is WordPress.

[{
  message: "Blocked request 🚫",
  url: "https://valisa.io/.git/config",
  ua: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0",
  location: "Amsterdam, The Netherlands",
  ip: "45.148.10.80"
},
{
  message: "Blocked request 🚫",
  url: "https://valisa.io/wp-content/uploads/gravity_forms/g/f/f/b/",
  ua: "",
  location: "Dublin, Ireland",
  ip: "52.169.211.111"
},
{
  message: "Blocked request 🚫",
  url: "https://valisa.io//wp-includes/wlwmanifest.xml",
  ua: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36",
  location: "Boardman, United States",
  ip: "52.34.1.203"
},
{
  message: "Blocked request 🚫",
  url: "https://valisa.io/configs.php",
  ua: "",
  location: "Washington, United States",
  ip: "52.146.39.115"
}]

These requests invoke server functions. Naturally, I went to check Netlify's offer for security and was disappointed to see a "Contact Sales" button.

It was like me coming to the watering hole. I see hunters; they see me. We both shrug because they are after the wild beast. And nothing happens; we continue with our daily business.

Build WAF

Savanna rewards creativity. I hope this post will help founders like me build their products with fewer distractions. However, this solution is not for every scale.

Netlify edge functions offer great power and flexibility. I am fascinated by how tools multiply human potential. Let's build a layered defense for the Astro website.

I started by filtering out traffic targeting WordPress and PHP. Then, I focused on requests without user-agent header and those containing specific keywords like .git or .env. These rules deny access to many unwanted visitors.

It took me some time to refine them by observing usage patterns. Eventually, I decided to be unwelcoming to bots that don't respect robots.txt, for example, Bytedance. Later, I blocked common automation tools for data scraping by their default user-agent.

The hard part is adjusting rules for the specific use case. I remind myself that simplicity prevents bugs. But with granular control, shooting yourself in the leg is easier than ever. Security is in constant conflict with usability. Therefore, it's essential to test changes.

netlify/edge-functions/firewall.ts

import type { Config, Context } from "@netlify/edge-functions"

export default async (request: Request, context: Context) => {
  const bannedBotsRegex =
    /Bytespider|ByteDance|PetalBot|Scrapy|Go-http-client|python-requests|python-httpx|axios\//i

  const userAgent = request.headers.get("user-agent") || ""
  const referer = request.headers.get("referer") || ""
  const country = context.geo?.country?.name
  const countryCode = context.geo?.country?.code
  const city = context.geo?.city
  const location = [city, country].filter(Boolean).join(", ")
  const ip = context.ip
  const url = new URL(request.url)
  const pathname = url.pathname?.toLowerCase()

  // Catch good part of total junk
  const badBot = bannedBotsRegex.test(userAgent)
  const blankUserAgent = userAgent.length === 0

  // Be careful not to block file extensions you actually use
  const bannedExtensions = [
    ".php", // most popular among scanners
    ".config",
    ".cgi",
    ".bak",
    ".asp",
    ".aspx",
    ".jsp",
    ".py",
    ".rb",
  ]

  // Any match in url, review before deploy
  const bannedKeywords = [
    ".env",
    ".git",
    ".vscode",
    ".aws",
    ".ssh",
    "/wordpress",
    "/wp-admin",
    "/wp-content",
    "/wp-includes",
  ]

  // Exact url pathnames
  const bannedPathnames = ["/admin", "/backup", "/wp"]

  // Cool Hazker detection algorithm
  const coolHazker = bannedExtensions.some((ext) => pathname?.endsWith(ext)) ||
    bannedKeywords.some((keyword) => pathname?.includes(keyword)) ||
    bannedPathnames.some((path) => pathname === path)

  // Countries
  const denyStates = []
  const bannedState = denyStates.some((state) => state === countryCode)

  // Alright, let's do something about it!
  if (bannedState || badBot || blankUserAgent || coolHazker) {
    console.log({
      message: "Blocked request 🚫",
      url: request.url,
      referer: referer,
      ua: userAgent,
      location: location,
      ip: ip,
    })
    return new Response("Access denied", {
      status: 403,
      headers: {
        "Content-Type": "text/plain; charset=utf-8",
      },
    })
  }
}

export const config: Config = {
  // Review before deploy
  path: ["/*"],
  excludedPath: ["/*.css", "/*.js"],
}

Source code - https://github.com/softbeehive/waf

Results

Around eighty percent of my visitors were rogue bots. Smoking them out at the edge is an effective approach that helped me reduce server load and keep my usage within pro plan limits.

Pros

• ✅ Efficient, does the job
• ✅ Transparent, full control of the logic
• ✅ Easy to customize
• ✅ Affordable

Cons

• 🍋 Redeploy on changes
• 🍋 Maintenance
• 🍋 Prone to human errors

Support

If you like this article, give WAF a star on GitHub and consider supporting my knowledge sharing efforts. Thank you!

Donate via PayPal