North American Network Operators Group Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical Re: Internet Backbone Index
Sean, Do you have a pointer to the raw data? I couldn't find it on the site. -scott At 04:25 PM 6/27/97 -0500, Sean Donelan wrote: >Ok, so I said I wasn't going to comment on the methodology, I lied. >I wouldn't say the Keynote study is the worst ever. There are some >really rotten studies in the fields of psychology and sociology. > >Since the Boardwatch/Keynote study didn't 'test' DRA Net, I guess I'm >one of the few independent, disinterested parties to comment on the >study's methods. > >A problem with the Keynote study is it seems very dependent on the >location, type and connections of the testing platforms. Keynote >mentions that connections from Dallas and Phoenix were slow to 'every' >backbone site. This would indicate some systematic problem with >the testing sites. Perhaps the results are even more dependent on >the testing systems than the systems under test. There are also >problems with outlier data points. For example, elsewhere on the >Keynote site, the MCI web site had very fast access from 28 test >sites (< 4secs), and very slow access from one test site in >philly (> 14secs). Mixing and matching data points, if you left >out the one outlier data point, MCI would have been faster than >Savvis. So I don't know if the rankings are very meaningful if a >single test site can have such a pronounced effect. > >Unlike a scientific study, there doesn't seem to be enough information >to independently reproduce the results, so I'm just going from the bits >and pieces I can glean from the Keynote pages, press release, and article. > >>Concerning Internet performance, there have always been a variety of ways >>of measuring it. It all depends on what you are really trying to measure. >>The Keynote study is attempting to measure something to which the average >>Internet user (not engineers) can relate. However, There are also clearly >>the possibility of artifacts in the data because of the testing machine's >>TCP stack or other issues (Vern Paxson has covered these issues at NANOG >>and IETF meetings over the last few years). Checking their web site, their >>software appears to run on top of the TCP stacks of many systems, so the >>known artifacts of some of these platforms could be an issue. >-- >Sean Donelan, Data Research Associates, Inc, St. Louis, MO > Affiliation given for identification not representation > >
|