I'm suspecting that Comcast is doing more than just throttling connections, but they may be outright interrupting data streams they don't like. No clear-cut evidence, but we use a Comcast link to rsync files from a local server to/from our main web server out on the internet every night, in the wee hours of the AM (so I don't think we'd be dealing with node congestion, or inconveniencing any significant number of other users). Total transfers vary, but can get moderately large: A really big xfer might be 2-3 GB, and that amount might move both upstream and downstream over the course of a night's syncs.

After years of pretty much flawless operation, in the last few months we've seen the rsyncs fail fairly frequently. It's clearly not been an issue with the server on either end of the connection, and the connection itself seems to stay active. - Just relaunching the rsync a little while later usually gets everything across that we need to. Running sample rsyncs manually shows that what seems to be happening is that just the particular rsync link seems to go dead for a few minutes at a time, while the rest of the connection stays alive. (Eg, we can still browse, check email, etc over our Comcast link.) Rsync is pretty robust, but if the connection is interrupted long enough, it'll time out. Simply relaunching the rsync once it times out lets everything start back up again just fine.

I'm no network engineer, but it sure looks to me like Comcast is deliberately glitching specific traffic streams it doesn't like, in this case the rsync stream, after "x" amount of data has transferred. It's not a matter of throttling, because (a) the link just goes dead, rather than simply slowing down, and (b) other services over the same Comcast connection seem to continue working just fine.

For this particular application, I'd be perfectly fine if they cut our bandwidth to 1/2 or 1/3 of normal, as the syncs aren't time-critical; we just need them to be done by the next AM. But I'd *really* like them to tell us in detail what the actual terms of service are.

I could just switch our servers' connections over to the redundant AT&T DSL connection we maintain, but Comcast's higher speed (in bursts) is very handy during the day when we want to move smaller chunks of data quickly. So I do have an alternative available, it's just less attractive for various reasons. I do think they should disclose fully what it is their customers are buying, though, and should be required to disclose changes in their bandwidth-management practices. We didn't actually lose any data here, but the consequences could have been severe if we'd had a device failure and needed to call on our "backup".

Has anyone else seen what looks like deliberate interruption of specific data streams like this?

The thing that is particually noticible is that manufacturers are now being actively co-opted into sharing this information under NDA with MS to allow the hardware to work seamlessly with Longhorn.

Actually, that's incorrect. The Longhorn interface is binary-only (no source code or format information is communicated to Microsoft or to the OS). Basically, the manufacturer (or third-party developer) writes a driver with an API that makes processed RGB data available to the OS. This is the same basic mode of operation as Canon and Nikon (and probably others) have implemented already in their free SDKs. Here's a brief interview with a Microsoft exec about the Longhorn interface and the shortly forthcoming "powertoy" RAW thumbnailer/viewer that's coming for XP. -- Not likely the level of detail /. people would want, but more than I've seen elsewhere, may help dispel some of the misconceptions.

Of course, this means that the proprietary RAW formats remain entirely proprietary in the Longhorn era.

For the record, I personally think that some level of open documentation of RAW formats makes a whole lot more sense than trying to come up with a common standard. A number of people (Adobe prominent among them, of course) have proposed Adobe DNG as a "universal" format. This sounds like a wonderful idea until you look at the assumptions underlying the format: It assumes a rectilinear pixel array, with a Bayer color filter array pattern (a checkerboard of RGB color filters on the pixels, with twice as many green pixels as red or blue). This is indeed the format used by the majority of cameras out there, but it completely misses innovations such as Foveon's full-RGB-in-every-pixel sensor, Fuji's hexagonal-pixel/diagonal-array "SuperCCD", and Fuji's latest "SR" sensors, which combine low- and high-sensitivity sensors in each pixel.

While a "universal" RAW format would help with the issue of access to the underlying data, so would simple documentation of the structure of various proprietary RAW formats, and the latter wouldn't have the negative effect of stifling innovation in sensor technology.