North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

2006.06.05 NANOG-NOTES Peering BOF notes

  • From: Matthew Petach
  • Date: Tue Jun 06 06:53:27 2006
  • Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta;; h=received:message-id:date:from:sender:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition:x-google-sender-auth; b=kHXYdUmWYOmWu3PqeBqRPntabSUoiDSmWaQU2by0J3lzMoi0gLj61e9Q7lV+V6BE+ntN7bn2376FCRGDfyZLM6i2kNQjbfx6K6c4zTwuh80DjJpptUXeOMnF9uRgY4RNU+T26u2s1w9oGbJGNHKz/hrZQNctjZTlzb+KjJGBwaw=

(This time around I opted to go to the peering BOF and take some notes.
It's the one downside to parallel tracks--wish I could be in two places at
once.  ^_^;; --MNP)

2006.06.05 Peering BOF

Bill Norton introduces the Agenda;
unfortunately, my laptop took so
long to boot, I missed the Agenda

Doug Toy?, Transit Cost Survey,
data collected at NANOG 36;
he's just here to present the collected
info, not really representing anyone.

At NANOG 36, people indicated their cost
per Mb and commit level.
length of contract was usually 1-2 years.

42 data samples collected
avg $25/Mb
$95/$10 were the extremes.

Avg commit level 1440 Mbps

Other observations
as expected, cost per Mbs tends to decrease
as the commit level increases.
Tier1's are more expensive
Cost tends to vary more with Tier 1 providers
than with others.
between 0-500Mb commit level, prices are all
over; at higher commits, prices level out at
the bottom.

Question: Mbps, is that the cap, the usage,
inbound plus outbound?
A: That's the general 95th percentile higher of
the two inbound and outbound.  Committed amount.

Graphs tend to approach a hard bottom; as
commit increases, doesn't change all that much.
Bottom is around $10/Mb, even though commit
levels increase.
Of samples collected, 2/3 were from Tier1
90% of contracts are 1-2 year in length,
so didn't cause much variance.
Tier 1 definition is based on Wikipedia

Questions from audience?
Q: Data looked pretty clean; were there samples
pulled out to make it look cleaner?
A: No, other than people who left fields blank on
the survey.

Q: was there a timestamp of when contract started?
A: much of it wasn't complete.
Mostly within the 1-2 year range for length as well
as start date, so nothing really ancient in there.

BillNorton; people had some concerns about violating
NDA or contract details when filling out the survey.
Where do we draw the line in doing these types of
SteveGibbard; NDA is agreement between transit
provider and customer, and this was anonymized
and voluntary.
Data is interesting, both for purchasers and
sellers of transit.

Q: 42 samples graphed, there were 80-100 people in
the room at the time; so the real comments from
the rest of people weren't counted?
A: No, there were less than 50 submissions total,
of which 42 were complete.
Q: Patrick; many people put more than one transit
provider on their form; how were those other
transit providers handled?
A: no clue, he just got a spreadsheet with data.

Back to Bill Norton

Peering Lists Issue -- make available  to customer
prospects?  15 mins.
Peering disclosure dilemma:
customers often asked for peering list,
sometimes peerings restricted under NDAs
Metric for determining connectedness, capacity,
Is there a better metric for customers?
IX capacity in/out
Peering pipe size?

ISPs are getting commonly asked about this, based
on hands raised in the room.
How many people lose business because the customer
doesn't get an answer?
Sylvie, VSNL notes they provide the info when they're
under an RFP; they won't give capacity, they give an
aggregate, they won't go peer by peer, that would be
a violation of NDA.
BillN: are the NDAs written to allow total numbers
like that?
Sylvie: you should not disclose capacity per location
or per circuit, but they don't forbid aggregate total

BillN: is there something else that could be given
to the customer that would satisfy their question
without revealing what
Chuck: A lot of ISPs lie about their peerings; he
runs AMSIX, people claim to have multiple gig to
the peering exchange, he knows they don't really
have that much.
Patrick: but he can look at the peering stats on
AMSIX--Chuck notes only members can.
Patrick: customers ask how many gigs they can send
to a provider; it's available headroom, so they
ask their upstreams how much available headroom
is left.  Most providers are having a lot of
trouble getting the right capacities to the
right networks.  The reason many don't
answer is they don't like the answer they have
to give.
Ted Seely, Sprint, how do you solve the problem?
There's lots of traffic that needs to be exchanged,
how do you fix it?
Patrick: how about everyone upgrade to 10gigE
in many places?  If you can't afford it, stop
selling bandwidth.
But most people can't go to all the different
providers, they have to buy from a small subset
of providers.
RAS: No technology problem with doing it; it's
the money.  Not charging enough to cover the costs
of the technology you need to install to cover
the bandwidth.
Ted notes you can't just link at one spot, you
have to connect at six places, and you need to
have links in and out of the site to support the
volume, etc.
Patrick: can you tell us how much they exchange?
40Gb times 6 providers in six locations is probably
more traffic that Sprint has in total.
Ted Seeley; it's a time scale issue--yes, it can be
solved, but in what timeframe?

BillN feels it's reasonable information for a
customer to request.  Is there a proxy that
customers could use that would give similar

Tristan Horn, Q: has anyone use skitter to try to
figure out
the answer on their own?
Ted: last skitter pass was March, 2005--data isn't
current enough to be useful.

Michael Abrams, tcp test, consumers can run to see
how much traffic they can pass along; it becomes
very quickly apparent who has problems, and consumers
tend to reach quickly to that, go elsewhere.
Shows packet loss and latency at different throughputs.

Sweden, size of CA, they have 4 exchange points, and
want more.  Still gamers are most vocal about any
problematic interconnects.

Video Peering
Video Stats and distribution models -- all
YouTube Peering Personal -- Colin

They're getting into the peering fray, and only a
year old.
This is gigs and gigs, has potential to dwarf
current peering traffic.  Current issues could be
tiny compared to the flood of potential issues when
hundreds of Gbs comes flooding towards customers.

Voxley? on the web; 320x240 pixel image, want to
do a one hour episode, will be about 210MB file.
currently, show is watched by 10,000,000 households.
So, that's 2.1PB to transfer all those episodes.
If you try to stream it, via say iTunes, over 3
days, would be about 64Gbps solid for 3 days.
That's a pretty big number.
That's one episode, from one show.  American Idol
is even bigger.
He claims a better way is using peer to peer.
2.5% of the agents could be propagation nodes,
or 256,000 PCs as transfer agents; that would fit
on existing broadband connections.

What about multicast?
problem at core, end distribution; pushing to 256,000
is much more doable.
Ted Seeley notes you still have to pass the unicast
streams eventually, so you still haven't solved the
issue.  If you don't do multicast or caching at the
edge, you've just shuffled the problem.

Swedish police, 100Gb of peer-to-peer traffic at
peak, AMSIX lost 10gig, LINX lost 5gigs, probably
lost about 40Gb weds/thurs last week when the swedish
police shut it down.  Which site?  Pirate Bay torrent

Peer to peer has similar replication pattern to
multicast?  But it's not localized properly,
unlike multicast.

RickW, the pirate bay site, that site was only
distributing meta information.  peer to peer
traffic can be 12-20% higher because of the
duplicated traffic.  Peer to peer networks
aren't as efficient as unicast streams.

Matt notes 64Gb, 256,000 subscribers isn't
that big a stretch anymore.  Peer to peer
is unlike multicast, because it doesn't have
any localization to the best of his understanding.

Do peer to peer systems use latency to control
who they share with?  Somewhat; faster peers are
selected over slower peers.

Comment from audience is that live events are
still going to be the challenge; HDTV is getting
gb/sec from cameras, needs to feed it out, no
chance to cache, so multicast ends up being the
only viable option for it.

Question about is the assumption that getting it
over the internet is the right model, vs using
settop boxes?  How do you get a whole country
to change their viewing habits?

Korea Telecom uses peer to peer to pass content
around; it is limited to a certain DSLAM or POP,
which answers Matt's localization/topology issue.

Ted Seeley; yes, people are watching material
online vs set top boxes, the paradigm is shifting

RAS, works through numbers; a DVD of MPEG2 will be
around 4GB, if you drop out the special bits,
audio tracks, etc; heavily compressed, 600MB, to
around 1GB mpeg4 for DVD quality, 2-4GB compressed
for HD quality.
if we take 1000MB filesize, cost per movie to deliver
Assume $10/Mb, you get 1Mbit/sec/month for $1,
Assume normal traffic curve,
150GB/month for your per Mb $10 point.
Assume it's mpeg4 compressed to 1GB, that's
150 movies per $10/month, that's less than the
cost of postage.

How does it best make sense to distribute these
files?  Compression reduces file sizes about 15%
per year., has article about it.

1Mb/sec will do 320GB/month.  So number of
movies would vary.
Peer to peer want immediately, and want quality,
so the model might be problematic.
Caching at the pop edge would help.
And live events, multicast would help, use unicast
to fix lost packets.

How do you compensate for lossy networks?  Forward
error correction, which means you lose the benefit
of compression you were hoping to get.
Larger buffer won't help for retransmits unless it's
Curtis points out live streaming vs Richard's
downloaded movies.
RAS was replicating Netflix model, you order in
advance, get it later that day.
If you want to stream live stuff, it needs to be
multicast, provider needs to do the replication.

How much of it is unavailable in the country where
the provider is located, that's why you can't
cache or work with the provider to distribute it.

Brokaw notes more streaming is coming; more video
stuff will be coming in the months and years to
come.  For the access networks, how many are
multicast enabled, and are ready for helping
this scale?  3 hands.  Does the community agree
we have a bit of a challenge here?

Matt Peterson, encryption to make sure the file is
different for each user, to make sure it's legal.
Bittorrent with RSS to get metadata,
weather channel model,
ultraDNS, Akamai model
multicast model

What about skycache?  One-to-many distribution?

Colin from YouTube will start off the peering
They've recieved an AS, they push about 20Gb sustained,
their first datacenter went live in the middle of
March.  It is definitely growing.  Not multicast
enabled, no demand for it.  The average show they
have is 4 minutes, hundreds of millions of different
videos, not a good multicast target.
20% growth month to month, 50 million videos viewed
per day.
20gbit/sec outbound?

"Ask a ninja net neutrality" video.
Product launched in December, that's when they saw
real traffic; been moving more and more in house
in March and April.
They're pushing 20gig in house, not from CDN
network.  This is is all short form video content,
hopefully no licensed content sneaking in.
San Jose Lundy on equinix fabric
PAIX on paix fabric
Los Angeles on equinix fabric
Linking all 3 locations together.
Metro, long-haul providers take a long time.
Lundy is poorly connected.
Getting connectivity at 10 gig (Fiber swaps, metro
 links, cross datacenter crossconnects)
Other items
Starting to see traffic from v6-v4 gateways
most hardware providers do v6 support in software

Seeing v6 traffic from v6-to-v4 gateways from
Asia.  Are people doing much successful v6 on
modern devices?
Force10 s50, 4500 from cisco, all seem to be
software based v6.
Should they have a native v6 presence, go dual
stack, what hardware would be good to use?
sup720 will be the cheapest hardware to do
hardware v6 at line rate 10 gig.  Can't do it
on the lower end.

Person from puerto rico notes that hardware
switches can do 200Gb/sec of IPv6 switching,
why can't routers do it?  Frustrated audience
reactions from all over about differences between
switching and routing.

Robert Seastrom--should you do v6 at all?
Should you be a pioneer, and make the v6 people
happy, sure, do it; if you want to make money,

Q: how do they make money, and will that
model support their bandwidth?  it's
advertising supported, hopefully it will
support their needs.
Q: how do you compete with Google?  No idea,
he's not a business person.

Q: Content providers often don't get peers
at public sites, much harder to get peers than
access providers; are they running into that?
A: Right now, peers are still coming to him; he
won't know about resistance until he goes out
asking for it.

Open Peering Policy
[email protected]
[email protected]
AS# 36561

How much of the 20gig is going across peering
links vs transit links?  It's all transit right
now--he's just now asking about peering.

Jeffrey notes that if there are people who pay
transit to get your content, they will peer
regardless of ratios.
DanGolding notes that the tier1's are stuck on
a treadmill; they can't peer with you even if
they desire it, for fear of messing with their
own ratios.

Josh asks if they will go to other locations
than just the west coast of US.
Colin notes that they're doing a datacenter
Chicago is next.

Dan Golding, network neutrality on the peering

One year old company, video content, already doing
20gig/sec.  This is a buttload of traffic, and they're
already getting into the peering mode; will this traffic
ultimately dwarf the rest of our traffic?

Network Neutrality, an open discussion.

What it means to peer.
network neutrality: an introduction
what is network neutrality
simply put, it's the idea that internet service providers
should treat all legitimate traffic the same way regardless
of origin and destination.

Why should I care?
a significant portion of the internet is based off

The Bell perspective
An unfair load
the bell companies-att and verizon-aren't big fans of
they feels they're carrying traffic for others
and are uncompensated or not compensated sufficiently
Google, Yahoo, making big profits from the sweat of
their brows.

The Bell Answer
$$$ needs to flow from content carriers to the Bells.

How would Bells make this work again?
everything old is new again.
in the PSTN world, the bells charge reciprocal compensation
for voice calls terminating on their networks.
Talk of prioritization...
[he's really whizzing through his slides!]

Content perspective
The internet works because of neutrality
non-neutrality would stifle innovation
The bells have a bad business model, they're asking
content to help prop it up.
Prioritization is a protection model;

The bells are in a power position
their control of the internet is at an all time
the conglomerate controls a huge amount of access,
they are tier 1 internet peers, SFI peering with each

Others are tempted to jump on.
MSOs see non-neutrality as a way to keep other's
VoIP off their networks
On the other hand, the Bells will squeeze them; the cable
companies lack peering.

His take; leans towards content.
Would be fair, except their networks were built under
monopoly auspices.
Consumers have little choice in many locations
some sympathy for bells; it's their network, after all.

How will this end?
possibility #1;
regulation of internet to enforce neutrality.
regulation makes the internet less flexible
probably vague law with lots of FCC regulations
Regulatory Affairs officer has to vet ACLs
Difficult to do filtering quickly
ability to provide tiered services will be more difficult.

Possibility #2
bells get their way
VZ and ATT become tier 0 internet carriers
depeer everyone else as soon as current restrictions expire
charging reciprocal compensation to all others
where are the MSOs?  other access providers?
why buy transit from anyone other than a bell?
potentially disastrous scenario
internet growth is stifled
forces you to always buy transit from the bell companies.

Possiblity #3
threat of regulation causes a continuation of status
best solution
loss of neutrality could destroy the internet
internet regulation could stifle it; and the bells are
VERY good at regulation.
perhaps the evolution of tiered consumer services could
come about by volume, not quality
consumers warming to this model
content providers sponsor some access models.

Patrick Gilmore: Tier 0, how does that work when
VZ and ATT don't make up the bulk of the internet
anymore.  What about the rest of the world?
The bells get regulations changed when the world
doesn't move in the direction they like; reciprocal
compensation got removed when the ISPs became CLECs
and got lots of compensation from the Bells.  Now,
regulations modified.

Sean Donelan; it's net neutrality, not internet
neutrality, so what about other players?
broadcasters, radio, etc.
For voice calls, not such a big issue.
broadcast networks don't have much of an idea of
what this is about; not sure they see the applicability
of this yet.  When they do, they have huge lobbying
power as well.
Bells have very sophisticated lobbying efforts, make
them very smart.  The newbies like Google and Yahoo
aren't that savvy in washington yet.

60% liklihood that the end result is regulation;
is *not* a happy outcome.

Is there some pivot point that may make people
change their models?
Patrick Gilmore asks for non-US peering coordinators;
who would care if ATT/VZ depeered you?
Not many respond

Curtis--not just a US problem; similar discussions
in European countries.  At what layer does the
competition come into play?  In European, there's
no text on the table yet, but there's discussions
taking place at least.  Somewhat of a reverse
problem, it's the new players looking at tax
AMSIX, much better competition on the last mile
and the infrastructure, the big players can't
play the game the same way.

Dan Golding notes that if we had many different
ways of getting local loop to your house, it
would be less of an issue.

Incent development of alternate methods; wifi,
3G/4G/5G networks, etc.

Mikhail Abramson, with high speed cellular,
mobile is making network neutrality less of an
issue, since you do have more options.  The GSM
providers are happy with the internet bandwidth
usage on mobile data, it's making them really good

Matt Peterson?  Last comment about cell phones;
but people are used to packet munging; most people
are used to a clampdown at this point; they may not
even notice anymore.  Until Grandma using YouTube
understands network neutrality, it'll be this
community only fighting/caring.

If we all went to a common $10 provider, we could
create a new tier0 and bypass the bellheads.

A few peering personals.

Jeffrey Papen, MySpace, AS 33739, well over 20gigs,
in LA and by august, SJ, CHI, ASH, all Equinix
locations.  Needs to be in!!

People are now talking about putting
in their peering contracts.

Stafford Rau, CLEC in Portland, ops, in WA, northwest,
currently acquiring ELI from frontier, will be
about twice as big.
SIX westin, PAIX, AS12003, 7845, will roll all of it
including ELI into 7385, they don't get 5650
2-3gig transit now.

Philip, MSO in Canada 5769, 13571, about 15gigs,

Matt Peterson, SIX seattle, copy it in SF, neutral,
non-profit, 365 Main, mostly content providers,
CNET, craigslist, others.  load up your blogs,
SFMIX, 365 Main, he or Tim Pozar from UnitedLayer.
MetaInterfaces, video, 5gigs, makes money, tells you
what kind of video it is.  ^_^;

Vish from Netflix, looking for potential peers
in SJC, CHI, ASH, [email protected]
really aiming for eyeballs.

Ann claiborne, prolexic technologies, also in
peeringDB, NOTA, LINX, [email protected]
hoping to come to west coast.

Dash to restrooms and then to the next BOF.