How Russia's government blocked our business - and how we dealt with it

As many other services, we also fell prey to the recent banning campaign conducted by Roskomnadzor. It resulted in significant traffic loss, which deprived us of many potential customers. At first users of several mobile operators couldn't open mkdev and then even the major internet service providers started suffering from the blocks.

For the first two days we were holding out hope that the common sense would prevail. But it didn't happen and after a while it only changed for the worse, so we personally had to take control of that. And now we are going to talk about how we were fighting against the blocking of our service and finally won and saved the part of the market, which is critically important for our business.

How to officially unblock your website without any technical skills

It's really easy to end up on a list of blocked websites and not so easy to be officially removed from it.

We didn't even have a chance to think of what to do and where to write, since on the second day of the war the Internet was full with the variably useful how-tos from different human rights organizations.

First of all we wanted to write a claim directly to Roskomnadzor, as they have some sort of special procedure concerning that. But unfortunately we couldn't do that as the website itself and the claim form were either DDoSed or blocked. At the time of writing it still doesn't work properly.

In the meantime, they officially declared that the tons of the statements about harmless websites being blocked were nothing but fiction. And after that they asked to message their hotline about such situations so they would be checked thoroughly.

We did that and even added the screenshots of the blocked website from our customers and the one from Roskomnadzor's official website, mentioning that our IP was blocked. The answer we got was error 554 (when the host is banned for spamming). Some other businesses came across the same issue. It seemed like the executive body just blacklisted all the email providers well in advance.

Some days later, they finally came to senses and created a Yandex email account. We messaged it too with the same claim, but as of today, we still haven't got any answer. We got nothing from ROCIT (Regional Center for Internet Technologies) and RAEC (Russian Association for Electronic Communications) as well, even though their websites mention successful negotiations with Roskomnadzor.

The same dialogue we had with the Internet service providers, especially the major ones. Most of the time it was something like that:

— Why is the website not available?

— We have nothing to do with that, update the browser, contact your hosting provider

— But it's available with VPN. Which means the browser is fine and the provider is not responsible for that as well.

— We have no more recommendations for you.

— But is mkdev.me available for you?

— No, it isn't.

— Why?

— Have a good one.

Some of them actually admitted that the site wasn't available because of the Telegram blocking campaign. However, they were unable to provide the court decision which bans our service.

Translation:

Good evening! Your claim was processed. The website is not fully available as it uses the web services blocked due to the fact of Telegram blocking campaign, which is being conducted by a court decision. The websites (as well as IP networks, subnets etc.), that are subjects to blocking, are determined by Roskomnadzor and our company, as a communication provider, is obliged to block such websites.

Should you have any further questions, answer this message.

Thank you for choosing Beeline!

Catering to your needs, Daria Shevtsova

----- Previous message ----- Beeline users cannot enter https://mkdev.me/

It’s been 3 days.

It seems that the providers either masked off Roskomnadzors deals or were rather unprofessional (or maybe scared to be sued?). The conversation was fruitful only with some small regional companies.

Translation:

Good afternoon!

The network which your domain (18.194.0.0/15) belongs to was blocked by a decision of the Prosecutor General’s Office 2018-14-16 14:17:32.

The communications providers are obliged to block the websites and IP addresses from the Unified Register of Forbidden Internet Resources in compliance with the federal law of 27 July 2006 № 149.

Best regards,

VGS Technical Support

———

Good afternoon!

Thank you for your commitment concerning this question.
We are ready to help you as much as we can.
There is some Information on the Internet…

Two weeks after the ban we were still not provided with the reason for it and no official statement. We're absolutely sure we'll get our answer in court, but it'll take so much time. The traffic from Russia since then has lowered by 60% and material losses, as well as reputation damage, are growing in numbers.

At the end of the day we realized that we shouldn't hopelessly wait for the justice to be brought. And so we decided to think how to solve our problems with technical tools.

A few words on mkdev infrastructure

Here on mkdev we don't just use a couple of AWS services. The whole infrastructure is located on the Amazon cloud. Moreover, mkdev cannot do without EC2, RDS, S3, CloudWatch, CloudFrount, SES and SNS. Since mkdev is a de jure German enterprise, we use Frankfurt AWS Region (eu-central-1) as our data-center and cannot just move all our data and servers to some other country.

Moving the infrastructure to another country or some other Internet service provider is also pointless because the blocking is so blunt that it's almost impossible to break free from. They just block all the IP addresses in the whole subnets. As you might have already seen, the list of those addresses is growing every moment. There is no guarantee that after we've moved our data-center somewhere else Roskomnadzor won't block it too with all its IP addresses. We didn't want to spend our time and money on relocation just to be blocked after that again, it made no sense.

Even though we cannot get rid of the block completely, there are some ways to circumvent it. But first let's talk about IPv6.

Why IPv6 cannot help with Roskomnadzor's blocks

The executive body blocks only IPv4 addresses. So in theory you can just finally start using IPv6. The problem is that both the website server and end user should have it. As it turned out Russia has pretty low IPv6 availability. So there was no point for us to move to IPv6 as the providers of our customers might not have done the same yet.

But we got ourselves an IPv6 address anyway, even though we don't actually need it now, but that's another story. Let's continue talking about our final solution.

Why it's still possible to bypass the ban

Some say that at some moment Roskomnadzor blocked 1% of all the IP addresses. It might seem as a whole lot, which is partly true. But lets not forget about the fact that 99% are still available, millions of AWS addresses among them.

If we had a chance to get a public address not blocked in Russia and a server not restrained by the blocks (anyone not based in Russia), we would be able to route all traffic through this intermediate server. And this is how we bypassed the ban.

Step 1. Looking for the addresses not blocked by Roskomnadzor

We need that very same not blocked address to start with. You can check if the address is banned using a special page at Roskomnadzors website. If you need the list of the banned subnets, you can just google it.

Luckily for us, on April 28 the executive body unblocked two major AWS subnets. But on the other hand, the German AWS Region wasn't a part of those two subnets. Amazon publishes all its current IP address ranges on the Internet. We searched through the list a little and found the regions with the IP addresses needed. Unfortunately, most of them were located in Japan and the USA, but it was enough for us to make a move.

We picked the Japanese Region and started playing some sort of the slots game trying to get the needed address. We were looking for an Elastic IP, which could always be available for us only. AWS request generates random EIPs, so getting the necessary one is only a matter of luck. Or our programming skills.

Since AWS API is suitable for pretty much anything, we created a small script, which makes infinitely long address requests, checks if the results are parts of the needed subnet and releases them if they are not. The script is so simple that we're quite embarrassed to show it, but we will, anyway:

require 'aws-sdk-ec2'

ec2 = Aws::EC2::Client.new

while true
  sleep 2 # Not to exceed API request limit
  resp = ec2.allocate_address({
    domain: "vpc",
  })

  ip = resp.public_ip
  alloc = resp.allocation_id

  puts ip

  # Here starts one of the subnets not blocked in Russia.
  if ip.start_with?("54.170")
    puts "Win!"
    break
  else
    ec2.release_address({
      allocation_id: alloc,
    })
  end
end

Having played around with the script for some time, we got the non-blocked address we wanted. Well, it might have been that all the addresses were taken by other AWS customers, so it was only a matter of luck. And we were lucky.

A little sidenote: here we are talking about AWS and how to find a needed address in their data-centers. But the solution itself is not only for Amazon. You just need a server with a non-blocked public address. If you find one on Digital Ocean, Google Cloud, Azure or somewhere else, that's great too. It's more convenient for us to have the infrastructure and pay for it in the same place, but if all the Amazon Web Services EIPs are blocked some day, we'll just move our tunnel to some other place.

Step 2. Configuring the server tunnel

After we started the server and assigned an IP address for it, we needed to configure it that way so it would transport all traffic to the mkdev main server. There are several ways to do that, maybe even dozens of them. I'm going to talk about only two of them, one is insanely easy while the other is a bit more difficult.

Way 1: port forwarding at the third OSI layer

The easiest way is to forward the traffic at the third OSI layer. On all modern Linux distributions you just need to enter several commands:

echo "net.ipv4.conf.all.forwarding = 1" >> /etc/sysctl.conf
sysctl -p
firewall-cmd --permanent --add-masquerade
firewall-cmd --permanent --add-forward-port=port=80:proto=tcp:toport=80:toaddr=TARGET_IP
firewall-cmd --permanent --add-forward-port=port=443:proto=tcp:toport=443:toaddr=TARGET_IP
firewall-cmd --reload

That'll be enough for all the traffic sent to the proxy server to be forwarded to the real server hosting the website. But the thing is that you need to track the original client IP addresses. As a rule, they are stored in HTTP title such as X-Forwarded-For or X-Real-IP. Since we forward the traffic at that level without any HTTP titles, we cannot notify the destination server about the client IP.

What's so bad about it? Well, our website is going to take all the traffic as if it comes from one IP address (the one of our proxy) and that's not really cool.

There's no doubt that we can configure the proxy server in a different way, we can just turn off the masquerading function and use proxy for all the traffic between a client and a website. To do this we need to adjust the routing so that all the traffic packets go through the proxy. But we don't want that for all the traffic, only for that from Russia. That's why we need to talk about the second way - the fully featured proxy with nginx as the main part.

Way 2: proxy with nginx

In this case you need to configure nginx proxy server and make it forward all the traffic to the server hosting the website. You also need to notify the mentioned server about that and show it where to get the real client IP from.

The proxy configuration might look like that:

stream {
    server {
        listen 443;
        proxy_pass TARGET_IP:8443;
        proxy_protocol on;
    }
}

http {
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /var/log/nginx/access.log main;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    server {
        listen 80 default_server;
        listen [::]:80 default_server;
        server_name _;

        location / {
          proxy_pass http://TARGET_IP;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       }
    }
}

You also need to adjust the configuration on the website nginx-server:

server {
  listen 443 ssl;
  listen 8443 ssl proxy_protocol;
  # ....
  add_header Cache-Control public;

  set_real_ip_from PROXY_IP;
  real_ip_header proxy_protocol;
  # ...
}

Let me say that it's not a perfect configuration, but a roughest and the fastest one. You can use it as a reference when you implement your own.

This configuration will make nginx to forward all the traffic and the destination server will be able to get the real client IP from proxy_protocol. That'll be more than enough at the end of the day. There's just one more step: DNS.

Step 3. Adjusting the DNS settings

We want all the client traffic except for that from Russia to go directly to our mkdev servers. We also want to route our Russian customers traffic through our proxy IP address when they enter mkdev.me. Route53 can do that with no trouble using the geolocation routing policy. We just indicate which address should be used by our Russian clients and which by others. As a result Russians go to the proxy server and the rest of the world goes directly to mkdev.

That's it?

Mostly, yes. We followed the steps, configured the simplest nginx proxy and now mkdev works in Russia again. If our proxy IP is blocked once more, we can always find another one and move to it.

What's important is that the proxy server shouldn't be too far from the main servers. Originally we tried to set the server in Tokyo, but it worsened the response rate. So after a quick proof of concept, we moved it to Paris.

So if your service was blocked by Roskomnadzor or some other official body, don't fret! There are some ways to bypass the block. You would definitely need to start the official investigation, but it might take weeks and you need your service available right now.

The conclusions we came to after the block

There is one and it's rather disappointing: it makes no sense for us to focus on the Russian market anymore. At some point a group of people, who are not fully aware of their actions, might just nip your business in the bud or at least seriously damage it and we are not ready to work under such conditions. We will always provide our services to Russian customers, but after what happened we'll give preference to some other markets.

Quick update: this whole story happened in April 2018. Around July 2018 our primary IP address was finally unblocked and we could drop the proxy for time being. We still decided to translate this article and share it with broader audience, because who knows who and when will block your business like this and when you will need some quick tech tips to resolve the problem.

This is an mkdev article written by Kirill Shirinkin. You can always hire our System Administration mentors and learn how to deal with such problems yourself.