I mean't this actually to be a reply, but it was too large to fit... what else was I to do.
In the following discussion I am assuming one is trying to create a rough replacement for YouTube, just without the giant bandwith costs. This means, yes, there is a central server with a repository of all content and client information.
In the traditional usage of bittorrent, moving around teh warez (and maybe legitimate linux cd images), consumers demand error-free transmission of the entire file, with no preference for any specific part of it -- this is built right into the protocol. For moving trashy video content around, 'tube-styles, your consumers have a different set of interests. They want to watch their chunk of data front to back, and many will lose interest after seeing just a small introductory piece and abandon the download. Additionally, in the interest of quick loading, users are willing to put up with some degree of glitchiness and general variability of the content's quality. I claim we need a different kind of p2p protocol to pull off what you are asking -- something that builds in the sequential access aspect.
Futher complication, people like to go on to something else as soon as a video ends. With bittorrent there is that pressure to seed after you have finished, and that is just important here. A way to keep people around without making them too angry would be to have a single client that would play several videos (possibly including the searching tools) so that all of your hard downloading work isn't tossed as soon as you finish watching.
I'm thinking of (the p2p architecture that everyone loves to hate) a supernode-with-reservations architecture. High-powered, well-connected, un-firewalled, idle clients would give out soft IOUs for a few seconds of content within a certain time window -- "I'll do my best to get you the high quality stream of video from 1:10 to 1:15 so long as you confirm your request for it within the between 30 to 40 seconds from now." Clients should try to arrange for supernodes to promise them both a high and low quality version of so the viewer doesn't get stuck with NO content in case a node can't keep its promise. In a sequential design, you'd never want anything from a client who hadn't downlaoded as much video from you, so a lot of the cool two-peers-cooperating-on-a-single-file stuff that you see in bittorrent isn't possible here. Cooperation would have to work at a little higher scale with either a very fancy cryptographic system or a central server providing trustable ul/dl ratio information to provide an incentive to stay online to seed. Bittorrent swarms get unhealty as the number of participants dwindle. For the likes of trashy 'tube videos your effective swarm for a given video is probably pretty small at any given time. If a set of videos is commonly viewed together hopefully there would be some mechanism to make it quicker to view related content -- maybe its easier to do business with peers you've already done business with, who, having watched the related content, could serve it to you faster than a random buddy. Though there wouldn't be specific, independent swarms, clusters of peers would temporarily form (~30 minutes) that improve the user experience for that sector of content -- search results could even be sorted by their estimated "distance" to you (some measure of how easy it would be for you to view that stream).
I haven't gotten to the quick-order-of-magnitude-estimate stage to guess how big the various windows and promise intervals would have to be, or how big a swarm would have to be for it to make sense. One cool thing, however, is that in the case where there is no one else around to help out, the server simply keeps making IOUs to any client connects for whatver they want (given some throttle) and the client software wouldn't have to care that the content was actually coming from the original provider. The "tracker" thingy would simply tell them there was a promising supernode at the same IP address.
To keep things pretty simple, the mast3r server would have knowledge of:
- what clients were active
- what thier IPs are
- if they are connectable
- how many pieces of which files they had cached
- a count referrals the server has made to this node
- some ratio stats
- some more numbers to be used with the number of referrals so far to help the master server decide how often to send referrals to this client
(much of this could be reported to other clients on demand to establish a reputation and serving-priority for other clients that try to talk to them)
One important new thing I introduced there was the idea of pieces within files (obviously stolen from bt, yes). A piece would be sized so it represented a decent chunk of video, maybe a minute or 5 minutes. Clients would request IOUs for some smaller unit of video (a block, ala bt). Since blocks and pieces are always downloaded sequentially it would only be necesary for a client to report "how many" not "which" pieces it had. Because the First piece would always be the one in highest demand, the master server can use piece status to have a large list of peers who had completed the first bit of video without waiting for clients to report having finished caching the whole file. "Proof of posession" could be provided by sending the master server a hash of a random range challenge.
I'm making a big mess of the words "server" "client" "peer" and "node", but I hope its still pretty clear.
Anyways, this entails a lot of more thinking to keep the network fair and efficient. Beyond that, it requires (I think) a completely new video format (at least a compatible container format for other video streams) that is piece and block aware.
Thinking thinking thinking.
I do like where this is going.
Lets keep talking.
- employ a cryptographically secure virtual currency to represent credit
- central server still requires an immense number of concurrent connections
- reputation between peers (from tit-for-tat) encourages clustering
- most consumery connections are asymmetrical (dsl/cable), many users can't usefully contribute
- bt has a sequential mode
- this is a new slot between the worlds of p2p, lossless, any-order transmission and c/s lossy sequential front-loaded streaming
- watch out for a group of rogue clients banding together to generate cash/reputation
- the focus should be on taking any shortcuts possible knowing the access patterns for the data, generality not important (other good solutions for other access patterns)
- fair and efficient are often opposing goals
- low quality stream is used in last-chance, clients should start unless they think they can maintain full quality the whole time (only call in low-bw promise when high-bw promise falls through)
- scheduling is important, this is a soft-realtime problem
- "just-in-time streaming"
- i cant think of p2p systems with bandwidth promises
- easy to show status in "loading bar"
- if you cant start watching any video in under a minute, its useless (other alternatives more promising)
This is great material for a whiteboard at devhaus.