North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

NANOG 17 Peering BOF Meeting Notes

  • From: William B. Norton
  • Date: Mon Oct 18 01:07:31 1999

Hi all -

I want to thank all of you who participated in the Peering BOF Monday night
at the last NANOG meeting in Montreal. It was lively and very productive.
To document the BOF, and for the benefit of those who could not attend
NANOG, I am sending out my BOF meeting notes. Comments/Corrections welcome.

As an aside, I'd like to thank my old colleagues at Merit for inviting me
to return to the stage and MC the Montreal NANOG. It was a lot of fun -

Hope these notes help. Cheers -


			Peering BOF - NANOG 17 - Montreal
			Monday October 4, 1999 7:30 PM EST

Moderator: William B. Norton, Equinix, <[email protected]>
Attendees: About 150? from NANOG meeting

The agenda consisted of three items:
1) Ground Rules,
2) Presentation and Discussion of the Peering Decision Tree white paper, and
3) Populating the initial Peering Contact Database to facilitate peering

1) The Ground Rules were
	A) Not focus on about Peering politics, who gains, etc.
	B) Not about settlement, nor valuation of peering, etc.
	C) Instead, 
		Focus on positive things we can do to facilitate peering
		Capture (document) the essence of peering process

2) Peering Contact Database
The Peering Decision Tree interviews (about a dozen) turned up a key
challenge ISP Peering Coordinators face: identifying the right staff to
speak with at the other ISP. 

To this end, we helped with this process by having participating Peering
Coordinators toss their business cards into the hat. Information they *did
not* want in the peering contact database was crossed out. I assembled the
cards and e-mailed an excel spreadsheet of all the Peering Coordinators to
all the Peering Coordinators that tossed in their cards.

<Note: About 50 ISPs tossed in their card and received a copy of the
Peering Contact Database.  I'm updating the database : if you are a
involved in peering and would like to be listed in the Peering Contact
Database (and receive a copy of the current Peering Contact Database),
please send the relevant contact information (Contact Name & Title, Company
Name, address, AS #, [email protected]<ispdomain>.net address for peering, phone
numbers, etc.) to [email protected]  I will not send the database to those who
do not contribute info to the database, nor folks who are not peering
coordinators.  I'll send out periodic updates as appropriate.>

Naturally, the value of the Peering Contact Database to the community is
proportional to the number and type of peering coordinators listed.  To
help maximize this, I'll try to grow the population of this database by
repeating the peering BOF at a variety of forums domestic and
international. Suggestions welcome.

3) The Peering Decision Tree Discussion
The majority of the meeting was focused on the Peering Decision Tree. I
interviewed about a dozen ISPs and documented the peering process in the
Peering Decision Tree white paper. We walked through the paper in the BOF
and validated the decision model; this roughly matches the peering
coordinator logic.

I started to write up a summary of this part of the meeting and found
myself rewriting the paper!  To save time, I'm simply including a raw text
draft of the peering white paper. (Prettier version (pictures, etc.)
available as a word document for those who participate in the Peering
Contact Database or are willing to be interviewed/add to the document.)

Please consider this a work in progress and send me comments! I'll try and
incorporate those comments back into the document for the community.
    ------------------------------- snip -----------------------------------

				Peering Decision Tree				       
			William B. Norton <[email protected]>
					DRAFT v 0.7

Internet Service Provider (ISP) peering is an interconnection business
relationship that decreases the cost and reliance on purchased Internet
transit. As the single greatest operating expense, ISPs seek to minimize
these telecommunications costs.
Interviews with Internet Service Providers have highlighted three distinct
decision phases of the peering process: 

	Identification (Traffic Engineering Data Collection and Analysis), 
	Contact & Qualification (Initial Peering Negotiation), and 
	Implementation Discussion (Peering Methodology). 

The first phases identifies the who and the why, while the last phase
focuses on the how. 

The appendix includes a diagram highlighting key questions asked when
identifying peering candidates and determining methods of peering.

I. Phase 1: Identification of Potential Peer: Traffic Engineering Data
Collection and Analysis
Choices made by Internet Service Providers (ISP) are often dominated by
telecommunications cost issues. Highest among these costs are Internet
transit costs that provide the ISP with connectivity to the global
Internet. Transit Prices for DS-3 transit for example vary by circuit mile
but can be as high as $50,000/month, and OC-3 transit can cost up to
$150,000/month . To reduce these costs, ISPs seek peering (-zero or reduced
cost) relationships with other ISPs that provide more direct traffic
exchange and reduce the load on these expensive transit services (as shown
To identify potential peers, ISPs use a variety of criteria.  There may be
existing business relationships that can be leveraged to include peering. 
Quantities of traffic distributed between networks often set the pace of
the negotiation; to quantify this, ISPs may systematically sample inbound
and outbound traffic flows. Flows then are mapped to originating AS, and
calculations are made to determine where peering (direct interconnections)
would most reduce the load on the expensive transit paths. There is
substantial work involved here, as this traffic sampling results in a large
number of data.  The end result of this first phase is list of the top ISP
candidates for peering.
The greatly simplified peer qualification decision tree looks something
like this:
Once the measurements have been made and analyzed, and it appears to be of
benefit to peer,, the ISP enters into Phase 2, Contact & Qualification,
Initial Peering Negotiation.

II. Phase 2: Contact & Qualification, Initial Peering Negotiation
Internet Service Providers typically have a person or group specifically
tasked with peering and traffic engineering issues. For example, UUNet has
a "Peering Steering Committee" to evaluate peering requests.  Some
variations of the following steps lead to the parties either leaving the
negotiation or proceeding to peering methodology discussions.

The first step is for one of the parties to initiate contact with the
potential peer. Today, discussion starts in one of the following ways:
a)	as part of a larger business transaction,
b)	via electronic mail, using the pseudo standard [email protected]<ispdomain>.net
or a personal contact, 
c)	from contacts listed on an exchange point participant list,
d)	with tech-c or admin-c from DNS or ASN registries,
e)	informal meeting in an engineering forum like NANOG, IETF, RIPE, etc., 
f)	at trade shows from introductions among speakers, or with booth staff, 
g)	from the target ISP sales force,
h)	from the target ISP NOC.

The interviews to date have highlight a key challenge for ISPs: finding the
right person to speak with is a difficult and time intensive process.
Mergers and acquisitions further cloud lines of communication. This is
where "people networking" helps a great deal, and hiring experience for
their contacts speeds this initial contact process up quite a bit. 

Second, mutual non-disclosures may be negotiated and signed, and a
discussion of peering policy and prerequisites follow. Traffic engineering
discussions and data disclosure may be used to justify the peering
relationship. Each ISP typically has a set of requirements for peering that
include peering at some number of geographically distributed locations,
sometimes at public exchange points. 
Traffic volume is usually a key determining factor. The decision rule
hinges upon whether or not there is sufficient savings from  peering to
justify spending capital on a port on a router and/or a portion of the
interconnection costs or augmenting existing capacity into an exchange
point. A Bilateral Peering Agreement (BLPA) is the legal form that details
each parties understanding of acceptable behavior, and defines the arms
length interactions that each would agreed to. 

Another motivation for peering to factor in includes lower latency and/or
more regional distribution of traffic than existing connections allow.
This process is diagrammed below.
After this initial discussion, either party may decide to walk away from
peering discussions until certain criteria are met . If both parties agree
that their requirements are met sufficiently to discuss methodology, and
they both benefit from the peering relationship, they move onto Phase 3:
Implementation Discussions. 

III. Phase 3: Implementation Discussions: Peering Methodology
Since peering is of mutual benefit, both parties next explore the
interconnection method(s) that most effectively exchange traffic to and
from each other's customers. The primary goal is to establish mutual
point(s) of interconnection, and secondarily detail optimal traffic
exchange behavior (MEDs).
For interconnections, ISPs face three  options: Direct Circuit
Interconnection, Exchange-Based Interconnection or some global combination

The "Interconnection Strategies for ISPs" white paper  quantifies the
economics and technical tradeoffs between the first two options. To
summarize this report, the preferred methodology depends on the number of
peers participating in the region and bandwidth required for its regional
interconnections. ISPs that expect to interconnect at high or rapidly
increasing bandwidth within the region, or expect interconnections with
more than five parties in the region often prefer the exchange-based
solution. Those that do not anticipate a large number of regional
interconnects prefer direct-circuits and typically decide to split the
costs of interconnection with the peer by region.  On occasion the costs
are covered in whole by one peer.

For direct-circuit interconnects, key issues center upon interconnection
location(s) and who pays for and manages the interconnection. This becomes
a material cost issue as traffic grows and circuits increase in size and
In either case, ISPs generally have the following goals for establishing
1.	fulfill obligations of larger business agreement,
2.	get peering set up as soon as possible,  
3.	minimize the cost of the interconnection and  their transit costs,
4.	maximize the benefits of a systematic approach to peering, and
5.	execute the regional operations plan as strategy dictates (may be
architecture/network development group goal).

Exchange Environment Selection Criteria
>From this point on the focus will be on the more scalable of the two
approaches (the exchange-based interconnection method) and describe the
selection criteria an ISP typically uses when selecting an exchange.  Note
that these issues are listed in no particular order.
Telecommunications Access Issues
How fast can circuits be bought into the interconnection environment? How
many carriers compete for my business for circuits back to my local Point
of Presence (POP)? For facilities-based ISPs, what is the cost of trenching
into the exchange (how far away and what obstacles present themselves)? Are
there nearby fiber providers that lease strands? These answers will answer
the most important question to ISPs: How fast can my peer and I get
connectivity into the exchange? Multiple carriers lead to speed and cost
efficiencies. Some ISPs have volume deals with certain carriers or
otherwise preferred carriers so prefer exchanges where these carriers can
quickly provision circuits.  These answers strongly impact the desirability
of the exchange environment.

Deployment Issues
How do I get my equipment into the exchange (assuming it supports
collocation)? Do I ship equipment in or do I have to bring it with me as I
fly in? Will someone act as remote hands and eyes to get the equipment into
the racks or do I do the installation myself? What are the costs associated
with deployment (travel, staff time, etc.)? Does the exchange have
sufficient space, power. The answers to these questions impact the
deployment schedule for the ISP engineers and the costs of the
interconnection method.

ISP Current Presences
The most inexpensive (and expedient) peering arrangements are the ones made
between ISPs that are already located in the same exchange with sufficient
capacity to interconnect. Cross-connects or switching fabrics can easily
establish peering within hours or at most days. ISPs will prefer to
interact where one of both ISP already has a presence.  Somewhat related,
is the exchange in the right geographic location?

Operations Issues
Once the ISP equipment and/or circuit into the exchange is installed, these
issues focus on the ongoing operations activities allowed within the
exchange. Does the exchange allow private network interconnections? Are
there requirements to connect to a central switch? How secure is the
facility? Is there sufficient power, HVAC, capacity at the switch, space
for additional racks, real time staff support?  Is it easy to upgrade my
presence over time? Upgrading in this context means the ability to increase
the speed of circuits into the exchange, the ability to purchase dark
fiber, the ability to increase the number of racks and cross connects in
the exchange, the ease of increasing the speed of interconnection. ISPs
will prefer bandwidth-rich, ISP-friendly exchanges over those with
restrictions over future operations.

Business Issues
Perhaps the most far-reaching issue is strategic: do we want to support
this exchange operator, and do their interests enhance or conflict with ours? 
According to Lauren Nowlin, Peering Coordinator at PGExpress, "Bandwidth,
strategic partner alliances, and corporate ties often override the
technical justification ." Will using this exchange support a competitor
(contribute to their net income, their credibility, their positioning)?
Does the exchange have requirements (require use of their carrier or ISP
services) that limit the market for services within the exchange? Since it
is difficult and disruptive to move equipment out of an exchange, ISPs will
prefer a neutrally operated exchange environment that doesn't suffer from
market distortions and limitations due to business conflicts of interest.

Cost Issues
What is the cost of using this exchange? What are the rack fees, cross
connect fees, port fees, installation fees? What are the future operating
fees going to be? What are the motivations and parameters surrounding these
fees? Cost issues shadow most of the other issues listed in this paper. All
else being equal, ISPs will seek to minimize the costs, particularly
upfront costs, associated with the interconnection for peering.

Credibility Issue
Does the exchange exist today and will it exist tomorrow? Who will be there
in the future? The benefit to the ISP grows (as pictured below) as the
number of interconnection possibilities grow, as the number of transit
customers grows, and as the bandwidth between the exchange participants
grow. During the early stages of the exchange, ISPs are asked to make a
leap of faith when committing, and therefore prefer an exchange with strong
backing and the credibility to ensure the ISP obtains value.
Does the exchange operator have the backing and credibility to attract the
more valuable peering candidates? Who is managing the exchange and what
technology is in use signals the credibility and survivability of the
exchange. ISPs will prefer an exchange with credibility - one that is
financially and technically well backed and likely to attract the most
desirably peering candidates.

Exchange Population
Are there other side benefits to using this exchange? Are there other ISPs
there that are peering candidates? Are there transit sales possible at the
exchange? In the context of the credibility issue discussed above, who will
likely be at the exchange in the future, and when will the cost of
participation equal the value of the interconnection (also known as the
Critical Mass Point)? ISPs will prefer established and well-populated
exchanges, particularly those with potential customers that can generate
revenue for the ISP.

Existing Exchange vs. New Exchange?
There are many regional "meet me" exchange points in each region of the
U.S. There are also emerging exchanges that may be considered. However,
given the pace of ISP expansion, it is unlikely that emerging exchange
offerings are differentiated or compelling enough to be preferred over
existing exchanges. Chronic traffic congestion can influence the decision
to plan to peer in an existing exchange or wait until a better exchange
opens. Customers with heavy flows of regional traffic can also influence
the decision.  Long term benefits (scalability) may lead to preferring a
next generation exchange.  However, all else considered equal, ISPs
generally prefer an existing exchange to an emerging one.

This third discussion phase can be summed up by the following graph.

IV. Summary
The paper provides a rough description of the decision process ISPs follow
to obtain peering relationships and reduce their transit costs. It focuses
on the elements of the decision that lead to the selection of a specific
exchange environment for peering. 

The results of the interviews with ISP Peering Coordinators can be
summarized with the following observations:
1) To reduce transit costs, peering goals for ISPs include a) to get
peering set up as soon as possible, b) to minimize the cost of peering, c)
to maximize the benefits of whatever architecture is selected, and d) to
leverage the architecture to accomplish their strategic regional goals. 
2) The selection of an exchange is made relatively late in the peering
process, and the selection of a soon-to-be-available exchange is a very
difficult sell. This is particularly true when there are existing exchanges
in the region that meet the ISPs requirements. 
3) Most critical to the ISP are issues surrounding business opportunities
presented at an exchange, telecommunications access, deployment,
population, operations, cost, and credibility. ISPs will prefer and
exchange environment that best suit these needs.
4) One major challenge facing Peering Coordinators is the identification of
potential peers and initiating discussions.

This paper was the result of interviews with key Internet peering
coordinators, interactions with ISP support staff, and side conversations
at various meetings and forums. Special thanks to a few folks who
contributed their time and ideas to help create this early draft report:
Ren Nowlin (PGExpress/Onyx), Joe Payne (IXC), Dave Diaz (Netrail), Jake
Khuon and Alan Hannan (Frontier GlobalCenter), Dan Gisi and Jeff Rizzo
(Equinix), Patricia Taylor-Dolan (Level 3 Communications), Sean Donelan
(AT&T Labs).  As interviews continue, this list will expand to identify
other key contributors to this paper.

[1] "Maturation in a Free Market: The Changing Dynamics of Peering in the
ISP Market" by Jennifer DePalma
[2] "NAPs, Exchange Points, and Interconnection of Internet Service
Providers: Recent Trends, Part I: 1997 Survey of Worldwide NAPs and
Exchange Points", Mark Knopper

Appendix A: Peering Decision Tree 
Appendix B: Detail Peering Decision Tree

Some interesting side notes from the BOF
+At least two countries including Israel have regulators that forced ISPs
to peer.
+Formal legal documents are not required for peering, although commonplace
+Keith Mitchel (LINX) has a prototype Bi_Lateral Peering Agreement (BLPA)
+The group wanted the Peering Contact Database NOT to be public, but rather
limited to the folks participating.
+Non-U.S. ISPs seem to have greater difficulty establishing peering here in
North America. Language, culture issues are mixed with technical issues.

William B. Norton	<[email protected]>     +1 650.298.0400 x2225 
Equinix Co-Founder, Director of Business Development