> Being able to distill a single number for any of these standard
> metrics -- packet loss, availability, etc. etc. -- on a per-provider
> basis would be really, really powerful. However, it also seems (to me
> at least) to be really hard to do in a way which doesn't hold inherent
> bias toward particular providers or architectures.
Well, perhaps the way to garner this number and still maintain a semblance of objectivity is to offer target numbers for different classes of service or types of architectures.
For example, certain routers allow you to "prioritize" your traffic according to content [putting the different types into different queues]. For such an architecture, you might specify that the highest priority packet types should pass with less than 1% packet loss (which begs the question of how you would measure that packet loss... :-)]. Lower priority packet types might pass with less than 5% and then 10%, to specify an arbitrary three levels of detail. :-)
Another example would be the ISP which wants to provide different "levels" of service. For example, Bell Atlantic used to sell (still sells?) CIRs on Frame Relay circuits which amount to "guarantees" on the level of over-subscription into your particular switch. The cheapest level is a 0 bps CIR on a 56 Kbps FR link. Many people like the 28 Kbps CIR on the 56 Kbps FR link, which is a little more expensive. An ISP would be able to market this with T1/T3 circuits by guaranteeing levels of oversubscription into routers with a given bandwidth out of the router.
Each of these different architectures is targeted to a different audience and would, therefore, have different levels of acceptable packet loss. Unless we are going to attempt to specify a maximum acceptable level of packet loss [based on the idea that after N% packet loss, non-TCP services collapse, for example], we would probably need to content ourselves with providing the measurement tools, the measurement methodology, and the knowledge of how to interpret the results.
Various industry consortia have set minimum acceptable standards to get a sticker like "Multimedia PC - Level 2" (which my home PC has on it). The IETF has requirements for routers and for hosts that you have to follow in order to be consider RFC-compliant. We need to be cautious in exercising the authority that an IETF standard would be able to exert over manufacturers for sure.
By giving people the tools to accurately measure and understand the results of the measurements, we've taken a huge step toward improving the overall ability of IP providers to provide quality of service measurements and to distinguish themselves in the marketplace accordingly. Coming up with a single measurement per provider for packet loss sounds like it might be a magic bullet - one that kills the providers who try to market tiered service.
Personally, I would prefer that the University of Pennsylvania have the best connectivity to the Internet that money can buy. However, given that the mission of the University is still [at least to some degree] education, not selling Internet services, there is a point at which the folks who control the purse strings will deviate from my plan. As such, I would like to be able to at least systematically explain to them what they are getting for their $XX,XXX or $XXX,XXXX per year so that they can make a decision that both they and I can live with. Without these measurements, I am always subject to attempting to explain to people who don't understand the technical details why we have to spend 2 or 3 times the amount of money that Joe Bob is charging a "T3" connection to his back-bedroom ISP.
-------------------------------------------------- Jon Boone Operations Engineer ISC Networking University of Pennsylvania tex@isc.upenn.edu (215) 898-2477