Zero to Anycast in 20 Days


Inspired by my friend, Samir Jafferali and his blog post, Build Your Own Anycast Network in 9 StepsI decided to have a go at building my own anycast network. The following breakdown will describe the steps that I had to take using guidance and best practices while ignoring guidance along the way. Additionally, you will find some easy copy paste stuff to make those short-lived moments even shorter.

My experience is in CDN, DNS, Web Performance and Development, i.e., HTTP, TCP, WebSockets, and a bunch of other stuff are my forte and my Sys Admin experience is on the lighter side. With that in mind, on a scale from Utterly Painful to Slept Through It, I rate this project a solid That Wasn’t So Bad.

Chronology of Events

Samir mentioned in his guide the need to do some prep work. The bulk of the time was spent waiting for things to happen. During the waiting periods, system hardening and automation became a key focus. Additionally, there was a desire to build some custom software and to create a feedback loop. Here’s a timeline of that experience.

Day 1

Register with a Regional Internet Registry (RIR). Registration with RIPE is quick and easy. Nuff said.

Request to Acquire /24 & /48 from a LIR.  Prager-IT e.U. [AT], based on Samir’s advice, was the best LIR. My experience with Stefan Prager was awesome. This portion of the transaction took the longest mostly because of identity verification and acquisition of IP space. For identity verification, utility bills, government issued identification, bank transfer verification, and verification of various email addresses associated with RIPE data were all required. Prager sent an ASN verification letter via International Post from Austria.

Day 1 – 3

Build a Network: Prager warned that there would be a minimum 10 day delay to get a /24. This was prime time for learning and setup. Per Samir and Prager’s suggestions, Vultr seemed to be the better choice. They have a decent presence globally, their web portal was super easy to work with, APIs are pretty clean, and they allow BGP sessions in 3 reaally easy steps.

After 4 hours, start to finish, 3 North American Vultr nodes were running a Centos 7 build with all the necessary updates and boot scripts. For the next 10 days, various tweaks had to be made to further automate. See below for a guide including what I did with Vultr.

Day 4

Server Hardening. Some nodes kept going down due to DDoS attempts on open ports, specifically ssh. It was apparent that script kiddies were looking for an in. After asking around, fail2ban seemed to be the right choice. Written in python so it’s easy to understand, it’s clean and effective.

Why not just change ssh ports?  These days were meant to be a learning experience. fail2ban could come in handy for monitoring other logs and banning IPs elsewhere, if not nsd (default configuration exists for nsd already).

Day 5-8

Automation. If this system is going to serve its purpose, a few things have to be done to ensure successful boot up, teardown, graceful service recovery (especially with custom software), and synchronization.

tmux+cssh: Salt, Chef, Puppet, and Ansible are all great options. However, in the interest of time and past experience, tmux-cssh worked well. Synchronized shell with live output was more interesting than simply using mssh. tmux-cssh also looks cool in a Terminal Window.

systemctl: Part of this process involved enabling service to easily be brought up and down, including custom software. That custom software also has external dependencies. systemd/systemctl are reliable standbys on RedHat/CentOS and allow for dependency chains. This is key for services that will rely on bird later. Again, rather than use push automation, tmux+cssh provides an interface for scripted updates. This includes pulling new patches, updating yum, updating system configs, etc.

Day 10

Fix Automation. Part of the boot script includes updating yum and installing some packages. In 10 days, a new version of fail2ban had been released. A new node in Sydney had become available, so a new node launch was attempted, but fail2ban was not running. As it turned out,  the packagers failed to create directories for the fail2ban sock. A bug had been filed earlier that day and is fixed in version 0.10.

Day 11

Acquired /24 and /43. Prager sent the good news that a /24 and /43 (yes, more space) were ready. Unfortunately, the space sat idle until the route object could be created, which required ASN (origin) assignment and MNT (maintainers).

Day 12-18

Spurious Updates. With things moving slowly, this time was spent waiting, thinking, architecting, playing with ideas, writing software. Progress was made, but nothing pertinent to Anycast.

Day 15

Paperwork. From Day 2 to now, Prager had been sending paperwork. This included a Letter of Authorization, Lease Agreement, Verification Letter (for ASN), and other information crucial to the registration process. The day following my request, an ASN verification was posted. Once it was received, signed and scanned, the process could continue moving forward, however, slowly.

Day 19.0

Acquired an ASN. From verification letter to ASN sponsorship request, only 4 days elapsed. Prager’s ASN sponsorship and guidance throughout allowed the process to go very smoothly. He also spent some time to guiding through RIPE updates, made suggestions for what metadata to include or modify, provided pertinent links to shorten the amount of time otherwise spent hunting for them, etc. With an ASN assigned, route objects could be added and anycasting was well within reach.

NOTE: Prager’s sponsorship service is €200 as it is time consuming. Worth every penny.

Day 19.5

Establish BGP Session. This process was truly easy with Vultr. They only need your ASN, IP Prefix, and Letter of Authorization (forIP space). It’s as simple as creating a ticket. Given the amount of time spent thus far, expectation of rapid turn around was low. However, within minutes of filing a request, Vultr handed the process to their Network Administration team. They only needed a few more tidbits of information: description of use case, BGP session password (just make one up), and what routes were needed (no routes at all).

Started announcing. With pertinent information in hand from Vultr, setting up dummy interface and announcing my IP were easy. This was a quick process, however, the routes needed to be whitelisted upstream.

Back to speaking in First Person …

Disappointed, I went to bed.

Day 20

Anycast! My eyes popped open at 4:50 AM. I rolled out of bed and went to my machine. To my amazement, everything was working. The rest of my day was spent playing with what I could do to bring down parts of my network and watch my traffic fail over to another host.


With my anycast project done, it’s time to move onto other things. Access to resources and knowledgable people made this process ludicrously seamless. Using Samir’s blog as a guideline and adding my own tweaks along the way, I was able to get some stable servers in place, running both Apache and a custom GeoDNS server to proceed with some of my experiments. All-in-all, I’m happy and excited to move forward.

Recipe: Anycasting Vultr Nodes with CentOS 7


• 2+ Vultr Nodes (Centos 7, BGP Session already negotiated with Vultr)

• vim/emacs/whateva

• fail2ban

• bird

• net-tools

• bind-utils

• instance IP addresses

• an IP to anycast + ASN

• BGP Session password

  1. Update yum and install the necessary tools …
    1. yum update -y
      yum install vim fail2ban bird net-tools bind-utils net-tools -y
  2. Open up the firewall to allow DNS requests
    1. firewall-cmd --permanent --zone=public --add-interface=dummy1
      firewall-cmd --permanent --zone=public --add-port=53/tcp --add-port=53/udp
      firewall-cmd --reload
  3. (Optional) Update fail2ban to protect yourself. You could also change your ssh port, but this could be used for DNS DDOS protection as well.
    • (Sample) /etc/fail2ban/jail.local
    • [DEFAULT]
      # Ban hosts for one hour:
      bantime = 3600
      # Override /etc/fail2ban/jail.d/00-firewalld.conf:
      banaction = iptables-multiport
      enabled = true
      port = ssh
      logpath = %(sshd_log)s
      backend = %(sshd_backend)s
      # This jail corresponds to the standard configuration in Fail2ban.
      # # The mail-whois action send a notification e-mail with a whois request
      # # in the body.
      enabled = true
      port = ssh
      logpath = %(sshd_log)s
      backend = %(sshd_backend)s
    • Restart fail2ban
    • systemctl restart fail2ban
    • (Examples) Check that fail2ban is running
    • # fail2ban-client status
      |- Number of jail: 2
      `- Jail list: sshd, sshd-ddos
    • # fail2ban-client status sshd
      Status for the jail: sshd
      |- Filter
      | |- Currently failed: 1
      | |- Total failed: 19451
      | `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd
      `- Actions
       |- Currently banned: 5
       |- Total banned: 55
       `- Banned IP list:
  4. Create a dummy interface
    • ip link add dev dummy1 type dummy
      ip link set dummy1 up
      ip addr add dev dummy1
  5. Update bird config and restart
    • (Sample) /etc/bird.conf
    • router id;
      protocol static {
       route via;
      protocol bgp vultr {
       local as 43011;
       source address;
       import none;
       export all;
       graceful restart on;
       multihop 2;
       neighbor as 64515;
       password "VULTUREPASSWORD";
      protocol device {
       scan time 5;
      protocol direct {
       interface "dummy*";
       import all;
    • Restart bird (presumes it was previously enabled)
    • systemctl restart bird
    • (Example) Confirm bird is announcing
    • # birdc show proto all vultr
      BIRD 1.4.5 ready.
      name proto table state since info
      vultr BGP master up 19:27:00 Established
        Preference: 100
        Input filter: REJECT
        Output filter: ACCEPT
        Routes: 0 imported, 2 exported, 0 preferred
        Route change stats: received rejected filtered ignored accepted
          Import updates: 0 0 0 0 0
          Import withdraws: 0 0 --- 0 0
          Export updates: 2 0 0 --- 2
          Export withdraws: 0 --- --- --- 0
       BGP state: Established
          Neighbor address:
          Neighbor AS: 64515
          Neighbor ID:
          Neighbor caps: refresh restart-able AS4
          Session: external multihop AS4
          Source address:
          Hold timer: 167/240
          Keepalive timer: 15/80
  6. Start services…
    • NOTE: Apache did well in binding to the dummy interface, but my custom DNS server needed to be told explicitly to bind to and respond for my Anycast IP. This is also the case for nsd.
  7. Drink scotch.