Re: Network end users to pull down 2 gigabytes a day, continuously?

as part of their normal operation.  The key is getting users to contact
peers who are topologically closer, limiting the bits * distance
product.  It's ridiculous that I often get better transfer rates with
peers in Europe than with ones a few miles away.  The key to making
things more efficient is not to limit the bandwidth to/from the customer
premise, but limit it leaving the POP and between ISPs.  If I can
transfer at 100kB/s from my neighbors but only 10kB/s from another
continent, my opportunistic client will naturally do what my ISP wants
as a side effect.

The second step, after you've relocated the rate limiting points, is for
ISPs to add their own peers in each POP.  Edge devices would passively
detect when more than N customers have accessed the same torrent, and
they'd signal the ISP's peer to add them to its list.  That peer would
then download the content, and those N customers would get it from the
ISP's peer.  Creative use of rate limits and acess control could make it
even more efficient, but they're not strictly necessary.

Good thinking. Where do I sign? Regarding your first point, it's really surprising that existing P2P applications don't include topology awareness. After all, the underlying TCP already has mechanisms to perceive the relative nearness of a network entity - counting hops or round-trip latency. Imagine a BT-like client that searches for available torrents, and records the round-trip time to each host it contacts. These it places in a lookup table and picks the fastest responders to initiate the data transfer. Those are likely to be the closest, if not in distance then topologically, and the ones with the most bandwidth. Further, imagine that it caches the search - so when you next seek a file, it checks for it first on the hosts nearest to it in its "routing table", stepping down progressively if it's not there. It's a form of local-pref.

The third step is for content producers to directly add their torrents
to the ISP peers before releasing the torrent directly to the public.
This gets "official" content pre-positioned for efficient distribution,
making it perform better (from a user's perspective) than pirated
content.

The two great things about this are (a) it doesn't require _any_ changes
to existing clients or protocols since it exploits existing behavior,
and (b) it doesn't need to cover 100% of the content or be 100%
reliable, since if a local peer isn't found with the torrent, the
clients will fall back to their existing behavior (albeit with lower
performance).

Importantly, this option makes it perform better without making everyone else's perform worse, a big difference to a lot of proposed QOS schemes. This non-evilness is much to be preferred. Further, it also makes use of the Zipf behaviour discussed upthread - if 20 per cent of the content and 20 per cent of the users eat 80 per cent of the bandwidth, forward-deploying that 20 per cent of the content will save 80 per cent of the inter-provider bandwidth (which is what we care about, right, 'cos we're paying for it).

One thing that _does_ potentially break existing clients is forcing all
of the tracker (including DHT) requests through an ISP server. The ISP
could then collect torrent popularity data in one place, but more
importantly it could (a) forward the request upstream, replacing the IP
with its own peer, and (b) only inform clients of other peers (including
the ISP one) using the same intercept point. This looks a lot more like
a traditional transparent cache, with the attendant reliability and
capacity concerns, but I wouldn't be surprised if this were the first
mechanism to make it to market.

It's a nice idea to collect popularity data at the ISP level, because the decision on what to load into the local torrent servers could be automated. Once torrent X reaches a certain trigger level of popularity, the local server grabs it and begins serving, and the local-pref function on the clients finds it. Meanwhile, we drink coffee. However, it's a potential DOS magnet - after all, P2P is really a botnet with a badge. And the point of a topology-aware P2P client is that it seeks the nearest host, so if you constrain it to the ISP local server only, you're losing part of the point of P2P for no great saving in peering/transit.

However, it's going to be competing with a deeply-entrenched pirate
culture, so the key will be attractive new users who aren't technical
enough to use the existing tools via an easy-to-use interface. Not
surprisingly, the same folks are working on deals to integrate BT (the
protocol) into STBs, routers, etc. so that users won't even know what's
going on beneath the surface -- they'll just see a TiVo-like interface
and pay a monthly fee like with cable.

As long as they don't interfere with the user's right to choose someone else's content, fine.

Alex