Reza Toghraee

Technical Manager (CCIE, Cloud, SDN, Netwrok, Security Expert) at ArpaWare

All Articles by Reza Toghraee

Internet service providers and their subscribers. From Dial-up days with access servers such as Cisco AS5300, Patton RAS boxes. They improved their services as connectivity technology evolved. xDSL, Fiber xPON, DOCSIS, Wireless, are all different delivery platforms that ISPs are using to deliver Internet to their subscribers. 

Apart from different delivery technologies, the ISP high level designs are similar. Mostly use PPPOE over the physical communication platform to establish a PPP tunnel for subscribers, to control the bandwidth and AAA (Authentication, Authorization, Accounting). 

A generic traditional ISP network looks like this:

In traditional design, BNG (Broadband Network Gateway) or BRAS (Broadband Remote Access Server), is a critical part of the network. BNG terminates the PPP subscriber tunnels, and is the single point where the subscribers interact with. BNGs are in different sizes, they are normally high end expensive routers with hardware accelerated encapsulation and tunneling capabilities. Cisco , Huawei , Juniper Routers are all example of ISP BNGs. 

BNG is a very critical component of ISP network and you should be aware that:

1. It is not scalable. each BNG router has specific limit of throughput and number of tunnels it supports simultaneously.
2. In most cases its a single point of failure, If BNG fails, all subscribers will loose connectivity until standby BNG gets activated and tunnels get re-established.
3. BNG is expensive. 

How to use Open Hardware and Software to design a scalable ISP network?

In this new design we are eliminating the BNG, and use network automation to control the ISP network. 

What we have changed:

1. Added Whitebox Aggregation switches running Cumulus Linux next to DSLAMs to terminate the subscriber's VLANs and L3, control the bandwidth and control the subscriber usage.
2. Added FreeRadius with DHCP module which authenticates the DHCP request against the accounting database hosted on a MySQL cluster.
3. Added Puppet automation tool (Ansible as an alternative) to automatically publish the subscriber policies to aggregation switches. 

How it works?

The CPE devices simply run DHCP client to retrieve IP address. The aggregation switch acts as a DHCP relay and forwards the DHCP request to the FreeRadius-DHCP. 

FreeRadius-DHCP authenticates the user based on MAC address of the CPE router, and returns back the subscriber properties such as IP address, bandwidth, quota , lease expiration, etc.  A DHCP response will be sent to the subscriber's CPE , also FreeRadius triggers the Puppet to publish the user's restrictions on aggregation switches. 

Puppet communicates the restrictions to the switches, applies the bandwidth, secures the user by applying anti mac-spoofing rules and runs the house keeping for IP accounting. 

Why we choose whitebox baremetal switches?

• Because they are robust and flexible. We need a switch which can support lots of ACL and Traffic shaping rules in its silicon. We looked at 48 port 1G switches, based on Broadcom Helix4 silicon asic. Helix4 supports 1024 Atomic policy rules (also 2048 non-atomic).  Edge-Core AS4610-54T switch was nominated which is based on Broadcom Helix4 chip. 

Why we choose Cumulus Linux Network Operating System for baremetal switch?

• Cumulus linux is a Linux based network operating system. For this solution, we instantly integrated Cumulus with Puppet by installing the Puppet agents on Cumulus switches.
• Puppet master, communicate the network changes to the particular switch. When a subscriber joins the network or it's lease period expires, Puppet pushes the bandwidth limits, ACLs, bandwidth quotas, IP accounting parameters to the Cumulus switches running Puppet agents.

What we achieved?

1. Scale out ISP network. you can add many aggregation switches and DSLAMs in different areas and cities. No limitation.
2. Reduced the TCO
3. Free Radius
4. Choice of L3 routed or L2 in transit network
5. Fully automated network provisioning and configuration management.
6. Zero touch provisioning for subscribers and additional switches.

How much does it cost?

A single baremetal switch (Edge-Core AS4610) with Cumulus linux will cost around $3,200.

If we calculate 

The above solution is a extract from a real deployment.