Richard A Steenbergen <ras@nlayer.net> nLayer Communications, Inc. NANOG 51 . January 30, 2011

.
What Is Peering?

.
Why Do Networks Peer?

.
Where And How Do Networks Peer With Each Other?

.
Understanding The Motivations Behind Peering Behaviors

.
Peering Policies And Requirements

.
Peering Strategies And Techniques

.
Does Peering Make Sense For Me?

.         
To understand peering, first we must understand how networks connect to each other on the Internet.

.         
The Internet is a collection of many individual networks, who interconnect with each other under the common framework of ensuring global reachability between any two points.

.         
There are 3 primary relationships for this interconnection:

.         
Provider . Typically someone you pay money to, who has the responsibility of routing your packets to/from the entire Internet.

.         
Customer . Typically someone who pays you money, with the expectation that you will route their packets to/from the entire Internet.

.         
Peer . Two networks who get together and agree to exchange traffic between each others’ networks, typically for free.



.         
Reduced operating costs

.         You’re no longer paying a transit provider to deliver some portion of your traffic. Peering traffic is “free”, so this reduces your transit bills.

.         
Improved Routing

.         By directly connecting with another network whom you exchange traffic with, you’re eliminating a middle-man and potential failure point.

.         
Distribution of traffic

.         By distributing traffic over interconnections with many different networks, you can potentially improve your ability to scale.

.         
What is an Exchange Point?

.         
A layer 2 network where multiple network entities meet, for the purposes of interconnection and exchanging traffic with one another.

.         
Often referred to as an Internet Exchange (IX), or “public peering”.

.         
Today most Exchange Points are Ethernet based LANs, where all members sharing a common broadcast domain, and each member is given a single IP per router out of a common IP block (such as a /24).

.         
Exchange points are typically dedicated to facilitating peering between members, and may sometimes prohibit other types of traffic.



.         
Why do people peer at Exchange Points?

.         
An Exchange Point acts as a common gathering point, where networks who want to peer can find each other.

.         
A network new to peering will typically go to an exchange point as their first step, and be able to find dozens or even hundreds of other like-minded networks interested in peering with them.

.         
The more members an exchange point has, the more attractive it becomes to new members looking to interconnect with the most






















other networks. This is commonly referred to as “critical mass”.
.         Exchange points also simplify the process of peering, by allowing any member to logically talk to any other member without requiring that they establish a dedicated physical interconnection. The typical cost model is that the member pays for the port, and can then establish as many peering sessions as they would like.


.         
Bilateral Peering

.         When two networks negotiate with each other and establish a peering session directly, this is called Bilateral Peering.

.         
Multilateral Peering


.         Bilateral peering offers the most control, but some networks with very open peering policies may wish to simplify the process, and simply
“connect with everyone”. To help facilitate this, many Exchange Points offer “multilateral peering exchanges”, or an “MPLE”.
.         
An MLPE is typically an exchange point operated “route-server”, allowing a member to establish a single BGP session and receive routes from every other member connected to the MLPE.

.         
Effectively, connecting to the MLPE is the same as agreeing to automatically peer with everyone else connected to the MLPE, without requiring the configuration of a BGP session for every peer.




.         
Unfortunately Exchange Points have inherent limitations.

.         
A single layer 2 network is relatively easy to disrupt, and any such disruption results in impact to potentially every member.

.         
It is difficult to measure traffic being exchanged between members.

.         
Any member can communicate with any other member, whether this communication is desired or not. This can result in security or theft of service issues which are difficult and/or costly to protect against.

.         
Scaling large layer 2 networks is inherently difficult, resulting in rapidly increasing overhead and costs as the exchange point grows.

.         
Most Exchange Points offer little or no control over the traffic being exchanged, allowing one member to congest the port of another.

.         
Shared broadcast domains make it difficult to achieve redundancy.

.         
Ultimately, for peers exchanging a large amount of traffic with each other, the cost of an Exchange Point port will simply be too high.



.
Private Peering is a direct interconnection between two
networks, using a dedicated transport service or fiber.


.         It may also be called a Private Network Interconnect, or PNI.

.         
Inside a datacenter this is usually a dark fiber “cross-connect”.


.         
It may also be a Telco-delivered circuit as well.

.         Though these typically cost a lost of money, and are avoided whenever possible.

.         
Often the cost of the interconnection itself is shared.

.         A common model is “I’ll buy this one, you can buy the next one”.

.         
What are the considerations of public vs. private peering?







. An Exchange Point is typically the optimal choice for a network
maintaining a large number of “small” interconnections.
.
Trying to maintain private interconnections with dedicated physical links for every peer is often financially or logistically prohibitive.

.         
For example, maintaining 100 GigE cross-connects to peer with 100 small peers would probably exceed the cost of an Exchange Point port.

.         
Not to mention the overhead of provisioning and maintaining the ports.



.         
But a Private Peer is typically the optimal choice for two networks exchanging a large volume of traffic.


.         For example, if two networks exchange 10Gbps of traffic with each other, it is probably cheaper and easier to provision a dedicated 10GE between them, rather than have them each pay for another 10GE exchange port.
.         Many networks maintain a mix of public and private peers.



. Most peering relationships are for “Mutual Benefit”

.
Peering is (almost always) a voluntary arrangement, which requires the agreement of both parties to exist.

.
If a peering relationship doesn’t benefit both parties,



then it probably shouldn’t exist.
.
But there are many possible definitions of what constitutes mutual benefit.

.
Some say “as long as you both benefit”….

.
While others want the benefit itself to be equal as well.



.         
Consider this small example of mutual benefit

.         
You own a hammer, and your neighbor owns a saw.

.         
But you each need both tools for projects you are working on.

.         
To save money, you agree to loan each other your tools.

.         
At this point both parties achieve an equal and mutual benefit, and it is natural for the free exchange of tools to continue.



.
Now imagine what happens when the balance changes


.
Your neighbor purchases additional tools, while you do not.

.
Soon he owns 30 tools, and you still have only a hammer.

.
At this point, your neighbor may not consider it “fair” that you






come over to borrow all of his tools, and offer few in exchange.


.         
Now imagine that your neighbor buys his own hammer

.         
At this point he no longer sees any benefit at all from your tool lending relationship, only the cost/overhead of lending.

.         
He might still let you borrow some tools from time to time because of the pre-existing (or “grandfathered”) relationship, or just to be nice, as long as the burden is considered minimal.



.         
Now imagine your neighbor goes into the tool renting biz


.         
And he is now successfully charging money for loaning his tools.

.         
Then imagine if you not only borrowed his tools for your own use, but also rented them to people he considers potential customers.

.         
At this point he probably considers your relationship a liability.



.         
Peering relationships often work in exactly this way.

.         
One of the first things a new peering network discovers is that not everyone wants to peer with them.

.         
The willingness of a network to peer with others is often summarized into the following 3 basic categories:

.         
Open . The network is generally willing to peer with anyone (or almost anyone), without imposing specific tests or conditions.

.         
Selective . The network is generally willing to peer with those who meet a specific set of requirements, but may not peer with everyone.

.         
Restrictive . The network is generally inclined not to add any new peers. Restrictive networks may list a set of specific requirements, but with the bar so high that the intention is for no one to reach it.







.         It is important to not only understand a networks’ stated
policies, but their thinking and motivations as well.


.         
Open Peering Phase

.         
When a network first begins to peer, their initial policy is usually Open.

.         
They typically don’t know who they should be peering with, or where.

.         
The default behavior is to try and offload as much transit traffic as possible.



.         
Selective Peering Phase


.         As the network becomes larger and gains more peers, they may realize that
they don’t need to peer with everyone to obtain the same results.
.         
Peering requirements are often imposed to eliminate the smaller peers.

.         
Focusing on larger peers may improve capacity and redundancy too.


.         Restrictive Peering Phase
.         
As the network grows still larger, they may reach a point where they say “we already peer with everyone we need to peer with”.

.         
Consolidating traffic down to a few large peers may even allow them to meet


others’ restrictive requirements, or even become a Tier 1 themselves.
.         At this point, potential peers are often viewed as potential customers instead.


.         Backbone Capacity and Diversity
.         A peer may be required to operate a redundant backbone with circuits of a certain minimum capacity. This can be done to ensure the peer is
of a similar size, or simply to select only “reliable” peering partners.
.         Minimum Locations or a List of Specific Locations
. A peer may be required to interconnect at a minimum number of
locations, for example “5 locations in the US across 3 time zones”.
.         
POP Requirements

.         A peer may be required to operate a certain number of POPs/regions.

.         
Traffic Requirements

.         A peer may be required to exchange a certain minimum amount of traffic to be considered. Small peers are often seen as not worth the overhead of configuring BGP sessions.

.         
Consistent Routing

.
Most peers require that you announce routes consistently (i.e. the same routes, with the same attributes) at all peering locations.

.
This is because most peer routing is “hot potato” (closest exit).



.         
No Simultaneous Peer and Customer Relationships

.         Many networks prohibit you from being both a customer and a peer at the same time. There are many reasons for this, but one very important one is that it is possible to rig the BGP announcements to achieve free transit if this was allowed.

.         
Operations Requirements

.         
A 24/7 NOC and maximum acceptable response times for network issues are common requirements.

.         
Nobody wants a peer who doesn’t respond when there is an issue.



.         
Generic Peering Requirements

.         
Announce only your customer prefixes, i.e. the definition of peering.

.         
Don’t static route, point default, or alter BGP next-hops to route something other than a prefix that has been advertised to you.


. Many of these items could be considered theft of service too.

.         
Good Neighbor Requirements

.         
You must implement prefix-list filtering on your customers.

.         
You must be responsive to spam, DoS, or other abuse issues.



.         
Others Policies

.         
Some peers require a Non-Disclosure Agreement (NDA).

.         
Some peers have “term lengths”, but most allow termination of the








peering by either party, for any reason, typically with X days notice.



.
The peering requirement so infamous it gets its own section.


.
Traffic ratios are both extremely popular, and hotly contested/hated.

.
The concept is simple, measure the amount of traffic being sent vs the amount of traffic being received, as a ratio between the two.



.         
Why do people care about traffic ratios?


. In a typical peering relationship, traffic is “hot potato” routed (i.e. the
goal is to get it off your network as quickly as possible).
.         
Push traffic coming from Network A gets hauled primarily by Network B.

.         
Push traffic coming from Network B gets hauled primarily by Network A.


.         
If the ratio is close to 1:1, both peers share backhaul costs equally.

.         
But if Network A is very push heavy, and Network B is very pull heavy, Network B ends up paying for most of the backhaul costs.

.         
Network B may consider this “unfair”, and deny/terminate peering.





.         
But many argue that ratio requirements don’t make sense.

.         
For example, what if Network B is actually pull heavy overall?

.         
Is it Network A’s fault that Network B has only eyeballs, no content? They’re just delivering the traffic requested by Network B’s customers.

.         
Network A may believe that Network B is simply using ratios as an excuse not to peer with them, with no technical or financial merit.



.         
CDN traffic also disrupts the simple analysis of ratios.

.         
CDN providers may have made significant infrastructure investments to replicate the content closer to the edge, minimizing backhaul.

.         
Their ratios may be heavy outbound, but the “fair” sharing of costs




between networks may actually be quite balanced.
.         And ultimately, does it even matter that the costs be “equal”?

.         If Network A charges $10/Mbps and Network B charges $100/Mbps for
the same bit, why isn’t Network B willing to pay 10x the backhaul cost?



.         
A network’s interconnection relationship is often summarized using the informal “Tier #” hierarchy:

.         
Tier 1 . A network which does not purchase transit from any other network, and therefore peers with every other Tier 1 network to maintain global reachability.

.         
Tier 2 . A network with transit customers and some peering, but which still buys full transit to reach some portion of the Internet.

.         
Tier 3 . A stub network, typically without any transit customers, and without any peering relationships.



.         
A key point here is that these definitions are INFORMAL.

.         
There is no standardizing body, and no fixed definitions for Tier #’s.

.         
These definitions are often (i.e. almost always) twisted by sales and marketing departments to find a way to claim that they are a Tier 1.



.         
What is the distinction between Tier 1 and Transit Free?

.         
We already mentioned that a Tier 1 network is Transit Free.

.         
But that doesn’t mean every Transit Free network is a Tier 1.



.         
Consider the following scenario:

.         
You go out and purchase transit from every Tier 1 network.

.         
You use BGP Communities to restrict the propagation of routes to their customer only, effectively simulating a peering session.

.         
Now any new network who comes along and wants to become a Tier 1 must interconnect with you in order to have global reachability.

.         
This arrangement makes you Transit Free, but not a Tier 1.



.         
To be a true Tier 1, you need settlement free peering.






. But any settlement paid between networks can easily be hidden
under an NDA, making it difficult to objectively “prove” Tier 1 status.


.         
Many people view “Tier 1” as the ultimate end-goal.

.         
After all, “settlement-free to everyone” sounds like a good thing, right?

.         
But being a Tier 1 also imposes a significant set of limitations.



.         
So what’s wrong with being a Tier 1?

.         
By definition, a Tier 1 must peer with (or sell to) every other transit free network in order to have global end-to-end reachability.

.         
If two transit free networks choose to discontinue peering with each other, they will become partitioned, which causes customer impact.

.         
Without a transit provider, there can be no “backup plan”.

.         
Many Tier 1’s have peering requirements which explicitly forbid hearing a peers’ announcements via any other path.

.         
And if a Tier 1 was willing to buy transit, they would likely be depeered by






other Tier 1’s in response, as this proves that they are not “equals”.
.         Tier 1 peering disputes are often severely disruptive to both networks.



.         
Paid peering is just like regular peering, except that one side pays the other some form of settlement for the privilege.

.         
In some circles it may also be called “partial transit”.

.         Though this could also mean any other arrangement involving “not full transit”, while “paid peering” specifically means customer routes.

.         
Paid peering also covers a wide range of relationships:


. Everything from two networks who are VERY close to meeting each
others’ settlement free policies, but just need that little extra kick.
. To very dissimilar networks who would normally never be peers, but
where one side is willing to pay for “peering” and not transit.


.         Some typical arguments in favor of offering paid peering are:

.         
The provider is also being paid by their customer on the other side of the traffic, thus they can now be paid twice for the same bit.

.         
If your customers are multi-homed, offering “cheaper” paid peering can incentivize others to use your network (rather than someone


else’s) to reach those customers, thus increasing total billable traffic.
.         One network may not be willing to buy transit from another for political reasons, but offering a paid peering product can give them a way to save face publicly while still being a paying customer.
.         
On the customer side, there are advantages as well:

.         As a settlement-free peer, you may have difficulty getting capacity upgrades from the other party. But as a paid peer, you have prioritization over others. It may be worth spending a little money to guarantee port availability when you need it.

.         
Some typical arguments against offering paid peering (or




against offering it at “less than full transit” prices) are:
.         Paid peer traffic must almost always be hauled across the providers’ network, whereas an off-net customer’s traffic may be largely hot potatoed. Transport costs are typically very high, so this traffic may
actually cost the provider more to deliver than “regular” transit traffic.
.         
Customers who want to purchase this “non-standard” product may have other external considerations, such as a need to appear directly connected for political reasons, and may be willing to pay a premium.

.         
Many customers intentionally buy from networks with restrictive peering policies, hoping to motivate others to peer with them directly. Giving the appearance that two networks are peers, even if it is in reality settlement based, may reduce this perceived restrictiveness.





.         
The majority of the Internet is multi-homed at some level.

.         
Even if the end-customer is not multi-homed, their ISP probably is.

.         
And even if their ISP isn’t, their ISP’s ISP probably is....

.         
The vast majority of prefixes can be reached via more than one peer.



.         
Donut Peering is the concept of using alternate paths to bypass some of the larger more restrictive peers.

.         
A large Tier 1 network may easily have 100,000 customer prefixes.

.         
But only ~5,000 single-homed customer prefixes.

.         
The other 95% can be reached via peering with other networks, or by peering with their customers directly, diverting traffic away from the restrictive networks completely.

.         
This technique has been extremely effective, and today many Tier 2’s do far more total traffic than some of the classic Tier 1 networks.



.
Buying transit from a network with an extremely restrictive
peering policy is a classic technique to increase peering.


.         
Many networks choose not to peer with customers of existing peers.

.         
By buying from someone with a restrictive peering policy, you increase the probability that your potential peer does not already peer with your transit provider.

.         
If their only option is to pay for transit to reach you, they may be more willing to peer with you directly.

.         
Similarly, if you started buying transit from someone whom they already peer with, you may find yourself depeered by a network trying to increase peering traffic with your transit provider.



.         
How can having poor routing increase your peering?

.         
Some networks may be reluctant to peer with you, but if they start receiving a large number of complaints from their customers about performance to your network, they may be willing to reconsider.

.         
This is effectively going one stop beyond the “Buy From A Restrictive Peer” technique. Instead of just costing the other peer money, you make certain that they can’t reach you successfully for ANY price.



.         
Some networks also use this technique to drive sales.






. Even if a certain network won’t peer with you, their customers may
be willing to buy from you directly to resolve performance issues.
.         
But this can EASILY backfire.

.         
You’re not just hurting their customers,you’re hurting yours as well.

.         
Networks who engage in this type of activity quickly develop a reputation for their misdeeds, and may lose customers as a result.



.         
How can having a restrictive policy get you more peering?


.         
Networks with open policies may find that they are distributing their traffic across hundreds of different networks instead of just a few.

.         
While the total traffic may be high, the traffic to each individual network may be rather low. This can keep you off the radar map of bigger networks who might be willing to peer with you.



.         
Strategic peering is often more effective than open peering


.         
You can also motivate some networks to peer with you by offering to send the traffic to them, rather than peering with their customers.

.         
Think of it as an Anti-Donut policy, you’re specifically sending traffic to people who are willing to work with you as a result.

.         
This increases the total traffic that your peer can bill their customer






for, and keeps them from being Donut’d around.


.         Another classic strategy to obtain settlement free peering is to tie it to other non-IP revenue.
.         For example, even though you aren’t buying transit, you may still be
able to purchase fiber, transport, or colocation services from a network who might not otherwise be willing to give you free peering.
.         If your revenue from those other services is sufficiently large, you may be able to negotiate settlement-free peering as a condition of the contract.
.         Large networks may also be able to tie an existing transit contract into a future settlement-free peering relationship.
.         For example, by promising X years of transit revenue now, with a guaranteed option to convert to settlement-free peering in the future if certain conditions are met.


“This peering thing ain’t so easy as the people profiting from it make it out to be.”
-Anonymous IX Operator
“Peering is a scam.”
-Anonymous IX Employee

.         
In theory, peering is “free” right?

.         
Not necessarily. The overhead associated with peering can easily become higher than the cost of purchasing transit.

.         
Many people who heavily promote peering actually have their best interests in mind, not yours (i.e. they want you to buy an IX port, etc).



.         
What do you need to consider before peering?

.         
What does it cost you for the router/switch port?

.         
What does it cost you for the cross-connect or IX port?

.         
What does it cost you in engineering man hours to maintain peers.

.         
What does it cost to maintain a backbone and backhaul traffic?

.         
What does it cost in lost 95th percentile aggregation?

.         
What does it cost in lost traffic with a transit provider, which you could use to leverage a lower price per meg?



.
Consider this small example:

.
You privately peer with 10 networks using 10x 1GE ports.

. Say each cross-connect costs you $300/mo.

.
You publicly peer at an IX with a 10GE port.


. Say the IX port costs you $3000/mo.

.
What is the cost of peering?

.
The simple answer would be $0.30/Mbps, right?

.
But what is the actual utilization on those ports?

. Lets say that it’s actually 50%, now your cost is up to $0.60/Mbps.

.
What would the 95th percentile be if this traffic was combined?






. It’s not uncommon to see a 50% difference from muxing traffic, especially
to peers in different time zones (i.e. a European, US, and Asian peer).
.
What did it cost you for each of those router ports?

.
What did it cost in engineering time to configure and maintain peers?




.         Peering often starts out easy, but gets harder as you go.
.         
Your first peers will be with open networks, and you will see transit traffic reductions with very little overhead for you.

.         
But after you knock off the low hanging fruit, more peering often requires building to new locations, investing in a backbone network, buying more IX ports and more cross-connects, etc.

.         
People often fail to take these costs into account until after they’ve


already built a peering network that costs them more than transit.
.         Peering is also the cause of a lot of bad routing/congestion.
.         If another network is willing to cause disruption to their customers until
you pay them money, and you aren’t, then you “lose” and must pay.
.         
Often peering becomes a game of “chicken”, to see who will blink first.

.         
While people are busy creating poor routing and buying from networks with restrictive peering policies to try and motivate peering, they may


not realize that they’re actually hurting their own customers.


Richard A Steenbergen <ras@nlayer.net>