North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

INCOMING: IP Provider Metrics

  • From: Matt Mathis
  • Date: Wed Mar 15 12:23:41 1995

Attached is my initial cut at an agenda and some introductory discussion for
the upcoming IP Provider Metrics BOF at Danvers.  I plan to circulate this
to the main IETF list on Friday.

Even though I am going to work hard to prevent it, this is likely to become
quite political.  My intent is to provide tools and methods (standard
procedures) which will enable the market to reward providers who are true to
their mission, what ever that may be.

Although my historical point of view has always been in the operations
community, I have clearly become "just a customer".  Furthermore, there is a
large pool of "experts" who think that they can do the job better than you.
Therefor, it is crucial that current providers take an active role in this
effort, if nothing else to offset the "experts".

Comments are welcome.

Draft agenda of the "IP Provider Metrics" BOF
6:00 - 11:30 Wednesday, April 5, 1995

Consider a possible WG.		(15 minutes at opening)
*       - Is a new WG needed?
*       - Relationship to BMWG/GISD/others
        - Inventory of possible tasks
        - Requirements for IP Provider Metrics
*       - Define scope and priorities
*       - Chair issues, Editor/secretary
*       - Charter
(* introduce at the open, but postpone detailed discussion until the close)

General Requirements for IP provider metrics	(15 min)

IP bulk data transfer performance (technical issues, 30 min)

IP routing stability and robustness (technical issues, 30 min)

Consider a possible WG...., part 2 (30 min)

----------------  Rough Draft Charter
IP Provider Metrics (IPPM)

The IPPM WG will develop a set of standard metrics that can be applied to the
the quality, performance and reliability of an IP datagram service.  These
metrics will be designed such that they can be performed by the providers
themselves, customers, potential customers or independent testing groups.

The areas covered will include:
	I) Path Performance
		A) Bulk data transfer performance (ftp, etc)
		B) Interactive performance (Telnet, X11, etc)
		C) Real-time and multicast performance (delay statistics)
	II) Routing stability and robustness
		A) End to end availability
		B) Time to recover (e.g. switch to backup paths)
		C) Route stability (Spurious route transitions, etc)
	III) others TBD

The IPPM WG will select or adapt existing tools and develop standard
procedures for performing and documenting the measurements.

---------------- Notes on the Charter

o Services peripheral to basic IP, such as NOC/NIS services, DNS, etc. are
  expressly excluded because there is vehement anti-consensus among Internet
  providers.  Prior efforts to standardize these areas have failed because
  they impinge on the different business models held by the providers.  For
  example some providers include full, high quality DNS service bundled with
  their IP service.  Others provide inexpensive DNS "starter" services, but
  expect most customers to eventually run their own servers.

o It is also important that the language not be pejorative, to permit
  providers to strategicly balance their price and performance.  We want to
  encourage all market positions including "Inexpensive, good enough service"
  as well as "The best possible service".

o The milestones will be weighted to address I.A. (bulk performance) and
  possibly parts of II first.  The other goals will be addressed as an ongoing
  effort.  The scope will start very narrow, and will be extended only as we
  have assured closure on the initial tasks.

o I have a prototype bulk transfer diagnostic that can safely and accurately
  measure IP bulk performance.  It is a combination of traceroute and Reno
  TCP, and can provide traceroute like path information under exactly the
  conditions induced by TCP.  See the discussion below.

o The difficulty with routing stability metrics, is that the best ones require
  collusion with the Internet providers themselves.  This clashes with the
  provider-to-provider trust model that is a necessary for healthy operation
  of the Internet as a whole.  BGP can easily be used to collect engineering,
  business and other data on competitors.  I believe that there exists a good
  technological solution based on the global collection and archiving of
  sanitized BGP logs.  Clearly, there is a substantial piece of this
  technology that belongs with the RA and other global routing agents.
  However, there are also parts that belong to the community in the form of a
  working group.  There needs to be a discussion of the possible role for IPPM
  in this process.

o The IETF is not "poised" to address "standard measurement procedures".  Is
  this a problem?

o The final draft of the charter should be completed after the WG is convened.

o IPPM should have an editor other than the chair.

Comments on the bulk performance tool:

I am working on a prototype diagnostic tool ("treno") designed to address bulk
performance issues.  It is derived from Windowed ping (Matt Mathis, INET'94,
See, but designed to be safe for
typical network administrators at user sites.

It uses traceroute to emulate Reno TCP over any IP infrastructure.  This
approach has a number of strong advantages:
- Its bandwidth consumption is bound by the same congestion mechanism as TCP.
  (Although this is more than might be tolerated for ongoing monitoring).
- It measures the network under the same queue dynamics as inflicted by TCP.
- Differences in the end systems need not affect the results.
- Today it can identify the lowest performance provider in a long path if
  there is a suitable unix box near each interconnect.
- Someday it will be able to diagnose (to a specific hop) any path across the

Treno is not yet completed.  There are some open issues which can best be
addressed as part of an IETF WG, with open and public input from all interested
parties.   In the end this will make it a stronger tool for supporting the
growth of the Internet.

Incomplete Tasks:
- Certify that treno emulates Idealized Reno TCP.  Understand and evaluate the
  implications of differences between treno and Reno as it appears in the
  field.  (IPPM and end2end research community)
- Develop testing methodology for "data sheet" measurements:  How to select
  and specify endpoints, sampling schedule, etc for measuremets to be used on a
  "data sheet" or service level language in a contract.  (IPPM and the network
  operators community)
- Document the need and utility for all router vendors to support full rate
  ICMP TTL exceeded messages. (IPPM and RREQ?)