Usenet Encoding: yEnc 431
Motor writes "Anyone remotely interested in usenet binary newsgroups must have noticed the spread of yEnc. yEnc is an encoding scheme for usenet binaries which avoids the enormous (30-40%) bloat associated with the schemes currently in use - which all have to produce 7-bit data to stop ancient newsservers from choking. A good thing, surely? Well, not according to some people. The guy has some good points about yEnc and standards, but I can't help thinking that "standards" people have endlessly discussed better encoding schemes, and nothing has come out of it. yEnc may not be perfect, but it works and it's here - hence the rapid adoption. What do you think?"
Intertia vs. Good Ideas (Score:3, Insightful)
Re:Intertia vs. Good Ideas (Score:5, Informative)
Breaking MIME is not something I would (do) lose sleep over. People in the MIME community screamed at us when we had the temerity to introduce the text/html content type, rather than use application/binary. They were completely obstructionist when it came to insisting on 8-bit clean transport for HTTP. In the end we treated them as damage and routed around them. HTTP uses several headers that the MIME people villified.
The functional issues raised are significant and it would be good to see them addressed. In particular using the subject line is pretty lame. Either you want the encoding format to be completely independent of MIME or you don't. I think that MIME independence would be the better route since then it would be easier to move to a more modern protocol such as BEEP. But using magic numbers and MD5 inside the encoding does not seem like a bad move.
The more interesting 'meta-point' however is that tweaking the encoding format is only scratching the surface when it comes to fixing UseNet. The main problem with USEnet is that it still has to route every single article to every single node whether it is going to be read or not. While the flood fill routing was a good scheme when NNTP was developed and the number of nodes was small it is needlessly wasteful now that we have hundreds of thousands of NNTP servers, it is just not necessary to have that level of redundancy to route arround censorship.
Re:Intertia vs. Good Ideas (Score:5, Insightful)
> servers, it is just not necessary to have that level of redundancy to route
> arround censorship.
I disagree entirely. Never underestimate the government's ability to stretch censorship to new levels.
Unless the very way NNTP servers operate is to gulp down and pass on each article for each newsgroup, the government would easily target those servers that spcifically carried groups or posts it doesn't like.
Pressure for news providers to drop certain groups began several years ago when the Vacco busts of people trading in child pornography led a news service to be criminally charged for the content of some groups and led other news servers in that state and elsewhere to drop gcertain groups thanks to their content. The charged news service took a plea even though they clearly would have won at trial or on appeal by claiming common carrier status, but hey, nobody wants to be the expensive test case.
Some may not see the problem with news servers being coerced by the government to drop those particular groups thanks to their contents, but the principle it sets is horrid. Certain "content owners" have of late been threatening to use the DMCA as a club to get news servers to drop groups which share TV shows and other such copyrighted material. If groups were more "localized" to a set of specific servers, or articles were localized to their originating servers, that would make it exceptionally easy for the DMCA to be used to require the "closure" of groups or removal of articles from USENET.
Furthermore, in this time of anti-terrrorist hysteria, the government has gotten away with the USA/PATRIOT mess already and is continually making some questionable choices. If it finds a newsgroup dedicated to dissent, or more spcifically dedicated to anti-globalism, for example, it cannot easily dstroy such a group because of the nature of USENET--the damage would be routed around by servers in other countries, even if every U.S. server could be forced to remove a group or article (not that they could be).
However, if the architecture of USENET were redesigned to localize groups or articles to subsets of servers--the likelihood of a government censoring USENET speech is magnified considerably.
It is the redundant architecture of USENET which will keep it free of censorship long after the WWW has been tamed--as it will be. Just look at the broiling mess within ICANN over officials trying to hand control of the WWW over to government-appointed reps. Eventually something like that will happen, and governments will cooperate with each other to make censorship in their mutual interests easier. Thanks to the architcture and nature of USENET, it will remain free and uncensored long after the WWW has fallen to censorship.
Just my 2 pence, though...
Re:Intertia vs. Good Ideas (Score:3, Insightful)
Umm, that's the point.
He who controls DNS exercises immense influence on the Net, particularly the WWW. Giving control of the DNS system to delegates directly appointed by governments is a recipe for fostering censorship in the interests of those respective governments.
I.e., if DNS were in the hands of representatives appointed by governments, and some given websites were to be deemed undesirable--delete their DNS entries, and they go away. Poof. The instant DNS is controlled by governments, is the instant they begin implementing a system to instantly pull the DNS entries of websites which are "dangerous" and "patently offensive." Governments hardly ever agree on anything, but you can bet they'd agree to some deal-brokering--you support the "delisting" of our seditious sites, I'll support the delisting of yours.
USENET isn't as vulnerable because of its architecture. You can't just "delist" one newsgroup the same way you can delist one website.
If the ICANN higher-ups have their way and hand over the reigns to government-appointed reps, I guarantee you governments will take the opportunity to at least consider using their control of DNS to enable censorship.
Re:Intertia vs. Good Ideas (Score:5, Insightful)
Re:Intertia vs. Good Ideas (Score:2)
Yes. A shipping product beats theoretical vapor every time, or, as previous generations would have said it, "A bird in the hand is better than two in the bush." But, that doesn't mean the product couldn't have been done right the first time (particularly with the apparently large number of folks who had made recommendations that would have improved the spec, which the yENC author is only now hacking into it). I believe this is the article author's point.
< tofuhead >
Re:Intertia vs. Good Ideas (Score:3, Insightful)
Not really comparable to the betamax vs. VHS debate. I've not seen anyone arguing that the alternatives, uuencode or base64, are better than yEnc, just that yEnc has serious deficiencies.
Perhaps Mr. Nixon is arguing that yEnc is worse than some wholly theoretical alternative.
Some of Mr. Nixon's points do seem interesting, but if he is convinced that there is a better alternative to be put forward, he should get the code out there. Anything else is just sniping.
Re:Intertia vs. Good Ideas (Score:3, Interesting)
Now, yEnc looks like it was created by Microsoft. It's not a standard, it's a hack. The only way it can become a standard is by pushing it down people's throats and then using "public pressure" to force applications to support it.
To use a videotape analogy, it's like releasing magnetic tape reels after people had been using cassettes for years, just because the reels use slighlty lighter tape.
I hope yEnc in its current form is *not* supported by the industry. I think a company such as Forté could create a real standard using an encoding method similar to yEnc (it wasn't "invented" by yEnc's author, anyway). I think Agent's programmers, of all people, should know how hard it is to deal with these (non)standards, and could save themselves a lof of work in the future by making sure it's done right.
As it stands, yEnc is the same as UUEncode, only in smaller portions (actually it's worse, because you can sort of wrap UUE in MIME; you can't with yEnc).
Yenc is great! (Score:4, Informative)
In fact, Yenc will help pay-per-gigabyte Usenet users achieve a greater bang for their buck. Anything that saves money is a good thing!!
Re:Yenc is great! (Score:2)
Did you read the article? One of its major points was that traditional binary encoding sucks, and instead of yEnc, people should come up with something better.
Re:Yenc is great! (Score:2)
This is what I interpreted from the article, A rant about the use of a new type of encoding because it is a new take on the old. So what if it is kludgy. It seems to be working. I don't see any new type of encoding coming down the pipe that would be considered a next-generation-USENET-type-of-thingy.
Hell, maybe USENET has lived beyond it's usefulness and does need an overhaul, but I seem to remember a time when people were bitching "Stop posting in base64! The rest of us can't read these files!" or even "Stop posting in JPG format! It takes too long for my poor 286 to decode JPGs instead of GIFs"
All this appears to be is more of the same. The haves versus the have-nots.
the point of the article (Score:2, Insightful)
DIME? (Score:3, Informative)
One, called DIME, is a MIME-like system that handles binary content, chunks, etc.
http://www.oasis-open.org/cover/dime.html
yEnc is terrible. (Score:2, Insightful)
BTW, I work for a pr0n site
Agent (Score:4, Informative)
Pan (Score:2, Informative)
I wonder if it's in CPAN yet...
Module Convert::yEnc (P/PN/PNE/Convert-yEnc-0.03.tar.gz)
Yep. Works for me!
Don't bother downloading now (Score:2)
Re:Don't bother downloading now (Score:2, Informative)
Subject: As Requested a UUencode version of Agent 1.91 with yEnc [1/2] - a32-191.exe
Message-ID: <q7Pm8.81615$rr6.1172226@news.webusenet.com>
Is usenet dead? (Score:2)
Does anybody really use usenet anymore? everytime i've poked around on my ISP's NNTP server, it seems to be filled with 90% spam, and non-spam posts seem to always be grossly offtopic. And no, i'm not just talking about the alt.binaries groups either.
Re:Is usenet dead? (Score:2, Funny)
Re:Is usenet dead? (Score:2)
The binaries groups I have seen have been pretty much noise free. If the users are vigilant and like to use physical means to squash spammers, the forum is void of abuse.
Re:Is usenet dead? (Score:2)
I do. Some groups are mostly noise, but others are still pretty good.
As for the spam problem, maybe you can suggest to your ISP that they install Cleanfeed [exit109.com] or similar. (Yes, that's the same site as the anti-yEnc page, which BTW I agree with.)
Re:Is usenet dead? (Score:2)
Re:Is usenet dead? (Score:2)
Yes.
That's why folks care about the yEnc issue. If it were a fight over, say, a Gopher implementation do you imagine there'd be much discussion?
Usenet is in trouble, it may be mortally wounded, but it'll be around for awhile yet and in that time lots of folks use it.
Re:Is usenet dead? (Score:2)
As someone else said, there are plenty of idiots posting the same questions over and over (groups.google.com anyone?), and topics tend to degrade, but for the most part, the groups that are actually dedicated to something are pretty decent. The Tolkien groups, the linux groups, some of rec.arts groups.
Re:Is usenet dead? (Score:3, Interesting)
Of course, the existence of alternate web based bulletin board systems like this one decreases its relevance for search purposes. And it's suffocating under the weight of all that spam. But USENET is still the biggest forum out there, and it's still the one that's the most easily searched.
What newsgroups do you read, anyway?? (Score:2)
-russ
no no no (Score:3, Insightful)
There was a market for this thing, it spread like wild fire. It's too bad that no one made a better spec and program (the author aludes that there was planty of time to do this). yenc meets the "GOOD ENOUGH" criteria, thus it will be used, shitty, non-robust standard or not.
Re:no no no (Score:2)
yENC (Score:4, Insightful)
Personally I can't see why we can't just send the data as 8-bit binary. uuencode and similar encoding formats should have died out with UUCP years ago, since there is no physical reason why 8bits can't be sent over the wire anymore.
Re:yENC (Score:2)
Ogg Vorbis has a specification for its file format that doesn't break any previously specified standard. Therefore, Ogg Vorbis can be called a standard.
yEnc, on the other hand, breaks the NNTP standards, and will likely break the MIME standard. That's the difference. yEnc must fit within the standards already specified for the transmition methods it uses, and does not.
Re:yENC (Score:3, Insightful)
When you create an encoding method that is going to be used to transfer data across a network, you need to ensure that this method is compatible with everything along the way. When you send an e-mail with an Ogg file attached, this file is encoded in a way that enables the servers and the client at the other end to identify it, check its integrity, reconstruct it, process it, delete it independently from the rest of the message, etc. It doesn't matter what the file itself is (Ogg, MP3, TIFF, Doc, XYZ, etc.); these methods work with any file, regarldess of its type or contents.
8 bits can be sent over the wire, and in fact are sent over the wire. But to make sure the servers (and the clients) can tell where one file (or part of the file) ends and the next one begins, you need to "wrap" the data in a package that programs can understand. That's what MIME does. It says "this part is the message text in HTML", "this part is the message text in plain text", "that part is an image", "that part is an executable file".
Instead of using this "universal" packaging system, yEnc forces programs to look for specific strings and try to guess where things begin and end. And it has no mechanism for identifying individual parts in a multi-part post (again, programs must look at the text or message subject and try to guess).
Doing something right is usually not much harder than doing something wrong. And when people get used to something that's broken, they won't want it fixed.
Re:yENC (Score:3, Insightful)
One approach to short-circuit standards is to take the rogue approach that Netscape did, which is very different from the one folks keep accusing Microsoft of. That approach is to take an almost-suitable standard and extend it, in the same spirit, with the intent that your well-thought-out extension will be adopted later. Of course they did a less than perfect job, but the idea is sound: don't wait for the spec, lead it, while making it possible for it to remain open. That's different from the Microsoft approach of "Embrace, Extend [and Exterminate]" wherein you add undocumented proprietary features that lock you into a single vendor's solution. A hell of a lot of what's in the HTML 4 standard was released as a non-standard but documented extension by Netscape in 1.1 and 2.0. Some of that was bad (blink, and arguably frames), but when you judge Netscape, try to separate the new HTML tags from the bugs, because it's the bugs (and having to code around them) that make the web such tower of Babel.
So basically what I'm saying is, if MIME is almost there, CAREFULLY THINK THROUGH and implement the extensions you want, and implement it already. Or, pick something other than MIME. But, don't start from scratch and make mistakes that have already been learned from and solved. Build on the good decisions of the past. The real crime is solving the same problem, badly, over and over; getting ahead of a standards body while keeping intentions pure and decisions wise is not nearly as bad.
Re:yENC (Score:2)
there is no real bloat associated with uuencode (Score:3, Interesting)
Transfer speed isn't the only reason for it. (Score:2)
- have megabyte quotas, both upload and download
- pay by the megabyte for their downloads
Also, this saves space on the server hard drive. NO WAY are usenet servers compressing data on their hard drives. It's one of the most challenging situations for a hard drive, they're not going to wreck their performance by using compression. Having less data means more retention on the server.
I personally have a Newsguy extra account, grandfathered at 1GB per day. I EXHAUST MY QUOTA by about 10AM, most days. I still do, but by then I've gotten 30% more data.
It's not all about transfer speed. yEnc means people with download quotas get 30% more stuff per day.
Re:there is no real bloat associated with uuencode (Score:3, Interesting)
I have yet to see any DSL or cable modem with compression. So for most of the heavy binary users, uuencode data will not be compressed. And on regular modems it won't be smaller than the the yEnc, since if, as you say, the binary can be compressed, then the yEnc will be compressed..
Do you run a heavily used news servers to provide proof that the CPU overhead is 'negligible'.
Re:there is no real bloat associated with uuencode (Score:3, Interesting)
For 99% of the people, getting the tunneling software on your system will be easy, getting your news server to install software on their software will be alot harder...
Adoption of yEnc (Score:4, Interesting)
As I'm a lifetime lurker (well eight years, but it seems a lifetime!) I can only choose not to download yEnc encoded binaries. And no one will know! (my news server doesn't log downloads) It's all up to the posters to adopt or not.
Re:Adoption of yEnc (Score:2, Informative)
Newspro - http://www.usenetopia.com
XNews - http://xnews.3dnews.net/
Newspro is shareware, XNews is freeware.. They're both good, for different things.
Cheers,
Backov
Does it compress? (Score:2)
From a news server standpoint, smaller physical size files of Yenc would allow more binaries and longer times on the server before being pushed off.
Re:Does it compress? (Score:2)
Traditional encodings first inflate the binary by 30% or so. Then your compression scheme compresses that expansion right back down.
yEnc inflates by much less, and your compression scheme deflates it right back down.
Upshot: In real terms, you're probably looking at the same amount of time to download 300KB of real, decoded data. The old system looks faster because it has a bigger bytes/second number, but in real data per second, after compression, they're roughly equal.
This will probably hold for nearly any simple encoding; encode in Base64 and watch your compression achieve 3-4x speed increase.
Consider this next time you are looking at marketing materials; numbers can say anything, but it's real performance that matters. yEnc gains by having less data sit on the news server, and costs nothing to those using compressed connections.
Re:Does it compress? (Score:2)
Re:Does it compress? (Score:2)
I agree and thats why I brought this point up. The article linked in the story written by Jeremy states:
yEnc creates significantly smaller encoded binaries than either uuencode,
base64, or binhex. That means faster downloads and faster uploads.
He uses that as the only real advantage of using yEnc. I tend to disgree that it is faster.
Re:Does it compress? (Score:2)
Re:Does it compress? (Score:3, Interesting)
Download an ISO vs a ZIP of an ISO from the same server on the same connexion, and you can see this at work with modem compression too. Chances are fair that the ISO download will complete first. (Well, unless you have a WinModem
I have to agree, the only real benefit is longer server retention. I don't see any benefit in having to juggle newsreaders every time yEnc gets updated. Maybe once it's a mature standard, then it'll be worth the effort to find another newsreader (I *loathe* Agent, and unfortunately my preferred NewsXpress is an abandoned project with no source available); meanwhile, I choose to ignore yEnc'd posts. And I'm thankful that they're marked in the subject line.
yEnc = XMODEM part deux (Score:4, Insightful)
Despite its problems, XMODEM took off because it filled a need, just as yEnc does. Nixon's complaint that shrinking files by 35% won't make Usenet any smaller because people will just post more files is besides the point; it's like saying getting a 35% salary increase won't help your finances because you'll just buy more stuff with the extra money. Most people want that extra 35%, and Jürgen stepped up to the plate and delivered it.
Thankfully, as far as I know, nobody railed against Ward Christiansen the way Nixon does against Helbing. XMODEM's problems became obvious and the solution was to introduce YMODEM and then ZMODEM. XMODEM is still around, but its successors (and of course serial IP) have pretty much supplanted it. Ward's initial efforts are still deeply appreciated.
Yes there's the problem of legacy software, but a protocol that's only been around for a few weeks or months can't have that much of a legacy. The only programs that currently support yEnc are the ones whose maintainers react pretty fast to new developments, and those maintainers are likely to also quickly pick up any revisions/fixes to yEnc.
So the solution Nixon should be calling for is not a years-long bureaucratic standardization process that will get yEnc 1.3 entrenched while the standardization is happening. The solution is to fix yEnc's problems and release a new version as fast as possible, before the old version gets spread around too widely.
Re:yEnc = XMODEM part deux (Score:2)
I still feel that moving to an eight-bit encoding system is fine, but let's be clear about what the issues are, okay?
Re:yEnc = XMODEM part deux (Score:3, Insightful)
That's the point--right now, the only yEnc implementations are in programs maintained by people who jumped on things quickly. So by fixing yEnc right now instead of waiting a year, you avoid trapping users of the slower moving programs. The slower moving programs simply aren't using yEnc yet.
I agree yEnc should have been more carefully thought out before being released. But its problems are not obscure. They are obvious to people who have dealt with these issues before. I don't think years of study and a lot of iterations are needed to figure out how to fix yEnc's problems. It should be possible to fix them quickly and get the fixed version out there before too many people are using the old version.
Save /. Money... (Score:3, Funny)
-Sean
He can't be actually using binaries groups (Score:2, Interesting)
Obviously, if the rest of the problems were as big as he's trying to claim, yEnc would only be a minor setback for a new and more comprehensive standard, but the fact is that the 35-40% overhead of current standards is by far what's most annoying to usenet users. After we got PAR (parchive.sourceforge.net), reposts have been reduced drastically (except for pr0n and partly warez groups, where the dumb people with shitty servers rules).
Also, he's trying to say that because the increase in volume will outgrow the savings, there really is no savings. What kind of logic is that? Let's stop making processors faster, we'll just find bigger problems for them to be too slow for anyway, so what's the point?
After the introduction of PAR and yEnc, as a long time binaries downloader, I'll say the actual content of multimedia groups has more than doubled, and probably tripled, the last 6-9 months. That's progress to me.
Standards ARE important... here's why. (Score:5, Insightful)
I work in an industry that relies heavily on standards, and my job deals specifically with standards. Making sure that WE follow standards, and making sure that other vendors follow standards.
Sure, they're slow to develop. But they're the best for interoperability, and that's crucial. In my line of work (for a major Mobile Phone System NSS provider), I have to deal with other providers that have to follow the same standars we do. That allows both of our products to communicate. This gives the end consumer (i.e., Cingular, Sprint, etc.,) the option to buy from different vendors. This forces us to make better products. This forces us to be more efficient. This forces our competitors to do the same thing. In the end, everybody wins.
The other alternative is what I see as the Micro$oft approach: Standards be dammed, I'm going to do it this way, and f*ck everybody else. It's the same approach that gives you security holes in your browser, because, well, who needs the standards?
I can't believe I'm reading comments like "well, it's here and it works so what's the problem?"
The problem is the future.
The problem is the inability to send an SMS from a CDMA service like Sprint to a GSM one like Voicestream. That's what happens when you blow off standards.
The problem is the inability to read an M$ Word doc that was sent to a Linux user.
Ignoring standards and going off on your own (especially, going off BADLY on your own) just divides us.
Good standards help us all. They give us better products. The lower costs.
CD-Rs. FireWire. PCI. countless others.
Besides, as the article begins by asking: Just what problem were they trying to solve?
Humor - "standards" (Score:2)
Sig: What Happened To The Censorware Project (censorware.org) [sethf.com]
yENC isn't even important (Score:2, Insightful)
If you don't know what they are, then you haven't been on usenet for a while.
But essentially, it allows you to stripe you sets with parity so that you can lose up to "n" posts and the PAR programs can rebuild the missing pieces.
I believe this has helped the backbone tremendousl.
Counterpoints to all of Jeremy Nixon's main points (Score:2, Flamebait)
There is no reason to despise magic strings. They work, and cannot ever occur in the user data. All yEnc magic strings start with =y, = being the escape character. Ctrl-Y does not need to be encoded, so yEnc is free to use =y for it's own purposes (e.g. =ybegin, =yend). Jeremy Nixon continues his misled rant...
No, using the subject line is not obviously a terrible way to determine filenames, segments, and anything else. I find it very convienent to know exactly what my yEnc files will be saved as, how big they are, and how many parts they are in inside the subject line. Nixon says "Sure, it works out most of the time, but it is imprecise and error prone (especially when spaces are used in filenames)" This is blatently false nonsense. Quotes reliabily allow clients to discern the filename. It's not "imprecise and error prone" by any stretch of imagination.
I give them that. Non-USASCII data in headers is a pain, and a large powerful organizational bodies needs to agree on a character encoding standard. Oh wait, they already did - Unicode!
False again. I've never had a filename containing quotes on my Windows box. If we expect newsgroups standards to reach everyone, we must use the lowest common denominator. Similar to how ISO9660 used 8.3 filenames, but on a higher level.
Which is exactly what the creators of yEnc intended.
They mean "AOL users" of course. Usenet hasn't had a new encoding format in 6 years, it's about time. Adopting this format should be as easy as switching from Napster to OpenNap to Morpheus to Grokster to Blubster and so on.
I don't blame him. Jurgen is a coder, not a politician. I would have done the same thing.
In short, yes I agree yEnc needs to be more polished. But the point is it works right now, and it's working great. It filled a gap in Usenet, itched a stratch to borrow an ESRism. Once yEnc is standardized as Y.32049 Annex D or whatever those standard organizations call it, we will use it. Until then, yEnc forever!
Technical arguments against yENC, blah (Score:4, Insightful)
It should be pointed out that this site [faerber.muc.de], linked from yENC's own website, goes into more technical detail regarding the technical flaws of yENC. The fact that it's linked from yENC's own site is proof that the author is at least familiar with the concerns that people have with his implementation.
I personally still find it difficult to argue against the article author's point that THERE WAS NO RUSH to force yENC out the door in such an unpolished form. After so many years of waiting for something better, why ignore the recommendations of those you are trying to help?
< tofuhead >
A few words from the original author (Score:5, Informative)
In my essay, I state that what Usenet needs is "a better way to post Binaries". The next piece of the puzzle, of course, is to answer the question, "What IS a better way to post binaries?" I was thinking about finishing that page up tonight, but I am writing code at the moment instead.
So, when reading my comments, just keep in mind that, yes, I DO have some answers to that question, too. It's just that it's a bit of a more time-consuming question, so that page isn't done yet.
This time around, though, I will make sure to include a prominent warning to NOT run off and implement the ideas as quickly as possible, and to please not use all of Usenet as beta-testers. The idea that whatever gets done fastest is best just doesn't work for me. There were good reasons I didn't go and get people to implement my smaller encoding ideas when I first wrote the code. If only the yEnc implementor had continued where I left off rather than going down his rather misguided path...
All the comments are welcome. I've been getting some interesting email, too, of course. Many programmers of Usenet client software absolutely despise the thing and are quite annoyed at the amount of their time it is wasting. I guess it's just more of that never-ending divide between the users and the techies. So it goes.
yEnc is here, that's for sure. Now we just have to try to deal with it.
Jeremy
Re:A few words from the original author (Score:3, Insightful)
Your essay is the best summary I've seen so far of the reasons not to use yEnc. You have done a service to those of us who have been annoyed with yEnc -- now we don't have to explain it to anyone, we can just point them to your essay.
So, be it resolved that yEnc leaves much to be desired.
However, if yEnc is the impetus which actually gets the community moving toward implementing a good, solid standard, then it will have served its purpose. Perhaps if we had had yEnc 5 years ago, we would have a standard already. But we didn't, and now we must pay the piper.
Since people aren't going to give up the advantages of yEnc without a substitute, the priority going forward is clear: to develop a better standard. If it truly is better (and not simply another hack) then ensuring its wide adoption shouldn't be too much of a problem. If, however, people can't be persuaded to switch, so much the worse for Usenet -- but no point in dwelling on doomsday scenarios. As you say, the cat is out of the bag, and all we can do is damage control.
Consider the source (Score:2, Flamebait)
Re:Consider the source (Score:3, Insightful)
The "he's against it because it saves bandwidth" argument makes no sense. If it saves users a little bandwidth, it saves Supernews many many times that much bandwidth, lowering their costs (which means they don't have to charge users as much to provide the same service). It also saves disk space, meaning Supernews doesn't have to buy new disks quite as soon. And a good bit of Supernews' business is in the corporate (outsourced ISP) service, which they don't charge by the gigabyte (they have speed caps, not monthly download quotas).
The problem is that any savings are just an illusion; this is just a momentary blip in the growth of Usenet. Since yEnc doesn't have the 100% market penetration that uuencode and MIME have, people are more likely to post binaries in multiple formats, causing storage and bandwidth needs to increase, not decrease.
Re:Consider the source (Score:2, Informative)
Smaller encoding helps us (the providers), it doesn't hurt us. (Besides, one look at the Supernews price list versus the competition will reveal that metered individual accounts are hardly the main focus.) With the customers the actual money comes from, less downloads would increase the margins, not decrease them. In fact, to a lesser extent, it would do so with the individual accounts as well -- the pricing on individual Usenet accounts (at almost all of the top providers) is such that the margins are lower at higher download levels. At the highest levels, it's nearly break-even. So less downloading would even be somewhat better there.
If you really think that I'm against a smaller binary encoding scheme, then you completely missed the point of my essay -- and you also must have missed the part about how it was my first experimental code, implementing smaller encoding, from which yEnc was hatched. If I truly wanted to avoid smaller encoding, why would I have implemented it in the first place? Why would I have done it in public and then sent the code to several people, explicitely releasing it into the public domain? You think I would have done that to prevent the spread of smaller encoding?
Had Jürgen picked up where I left off and did the thing right, I would be singing an entirely different tune.
Jeremy
Encoding efficiency isn't a BIG deal (Score:4, Insightful)
If, by any chance, you're transferring things over a modem (v42bis' lzw) or ssh vpn (zlib's deflate) or possibly other types of links, then you're probably not going to notice a difference anyway. The systematic encoding inefficiency that goes with base64 and uuencoding, results in a substantial lack of entropy that will be picked up on and exploited by good compression algorithms. Then end result won't be quite as good as having efficient encoding to begin with, of course, but it will be in the same ballpark. There's no way it'll be anywhere near a 33% difference.
This sounds like something that would have been useful 15 years ago before compression was widely used, and when people were still writing newsreaders. Now it looks like a waste of time and an excuse to get people to "upgrade" their software.
Standards for Usenet encoding??? (Score:3, Insightful)
There are no Usenet binary transmission standards, just a few different hacks to make it work. If this guy's new hack makes it work better, good for him.
What a miserable bugger (Score:2, Insightful)
A smaller encoding scheme gives us exactly one benefit: faster downloads and uploads for the users. It is not going to make Usenet smaller. It is not going to allow servers to increase retention. Do you really think people aren't going to post more, if they can do it faster? Of course they are. They're always going to post more, with or without yEnc [...] big deal.
So effectively, what he's saying is, in effect: "this system changes nothing, and is of no benefit, except that it makes more data available on the Usenet and gives users faster uploads and downloads. So it's worthless."
This guy obviously hasn't had to use a metered dial-up account for a while. A 33% saving on transfer times is an enormous benefit. I feel quite insulted by the way he seems to think it's of no importance, as if my time and money aren't worth anything. "What's the rush" indeed! I'd happily tear up MIME and MD5 tomorrow if it would speed up my transfers by a third.
If yEnc is so widespread, it can only be because there's a demand for it. And if there's a demand for it, why the hell shouldn't programmers support it? Last time I checked, RFC's weren't enforced by law. The Net has seen a million non-standard hacks, and has, for the most part, assimilated the good ones and outlived the bad. yEnc is by no means the worst, and it brings real benefits to tens of thousands of people every day. I say leave it alone - or if you have to oppose it, at least oppose it constructively, for Christ's sake!
Re:What a miserable bugger (Score:2, Informative)
FYI, I am using a metered dialup account (actually ISDN, which isn't much faster, just more reliable). I cannot get unlimited, high-speed access (yet, I keep hoping). I pay well over $100 per month for Internet access at a fraction of the speed of peoples' cable modems and DSL lines, and it would be quite a lot more if I left it nailed up 24x7.
You, like several others here, seem to have somehow gotten the impression that I am opposed to smaller encoding. I can only conclude that someone is spoofing my web host's DNS and some of you are reading a different page than the one I wrote.
Jeremy
Screw Elegance (Score:2, Troll)
Personally, I will adopt anything that will make my life (not the developer's life) easier. yenc allows me to download 30-40% less, so I use it. Why? because if I don't use it, I won't be able to obtain the mp3s, the games, the warez, the pr0n, etc, etc, etc that I wanted
Good idea, but... (Score:3, Informative)
... the implementation sucked.
I leech regularily from alt.binaries.anime [animeusenet.org] and the related newsgroups. When the yEnc posts started coming in, I simply upgraded my newsreader [3dnews.net] to the newest version. But a LOT of people out there use Agent [forteinc.com], and it was absolute pain to combine/decode all the yEnc posts that started popping up all over the place. The worst of it is that the yEnc posters were basically saying, "Start living in the present and upgrade". Nevermind that at the time that only yEnc-capable newsreaders were for Windows...
I mean, I don't know, but this sounds a lot like the OS wars that have been going on for quite some time. Some people simply don't want to have to switch newsreaders. Some people don't want to have to switch OSes. And that's fine, because it's a free world out there. I like the idea of yEnc (I get more out of my Easynews account), but I really don't think it should have been introduced so quickly.
~ Firecaster ~Two Problems: (Score:5, Interesting)
Two: Loss in transmission... I've been downloading yENC attachments for the last month, and out of them, found over 50% loss/corruption in posting... Not due to retention/propagation either... Just files missing large chunks... Now this *could* be due to some problems on the senders' end, but it seems just a little *too* coincidental that almost all of the losses have occured with yENC uploads...
Reminds me of Napster data format (Score:3, Insightful)
yEnc sounds like a good idea, and a horribly bad implementation.
Jeremy's right, but it's too late now. (Score:5, Interesting)
But yEnc's bandwidth savings are real, which is a huge win for alt.binaries users. yEnc has been the most-requested feature for Pan over the last month. (0.11.2.90 [rebelbase.com] supports it.) IMO yEnc is the format to use for multiparts right now.
Hopefully yEnc will motivate others to come up with a mime-friendly alternative encoding for Usenet. yEnc Considered Harmful [faerber.muc.de] is another yEnc opposition page that suggests mzip compression [faerber.muc.de], but I haven't seen any public discussion of it yet.
If/when such a replacment comes along, Pan will support it too and add an are-you-sure dialog for yEnc postings.
More like it used to be (Score:3, Insightful)
I, for one, am happy to see a useful format publically available.
Yeah, and off the web too! (Score:2)
I second this. Also, let's make it so that non-validating and just generally malformed XHTML documents are rejected by browsers, and can be filtered out of google. plain HTML =
I'd just love to turn on a "[ ] Reject non-validating pages" option in google and see the world wide web with new eyes :-)
Re:This is why it is bad (Score:2)
Yes, because everyone who doesn't speak English should just shut up. You did realize that MIME is the only standard on how to send non-ASCII plain text across email, didn't you.
Re:Why this is bad (Score:2)
Is it good is it bad? To late its here.
James
Re:concretely: this is why you don't need it (Score:2)
When you know it's a uuEncoded file, it should be very easy to write a speedy compression algorithm. Even just use a standard Huffman algorithm - it was originally designed to be implemented by hardware for storage devices (think tape drives) and wouldn't impose any real overhead on a modern server. I just can't see why we have to implement a new standard before it's ready.
Willy
Re:concretely: this is why you don't need it (Score:2)
The new modem standards apparently even have special compression modes for HTML, and if uuencode and base64 matter, modems and other compressors could recognize them as well.
That's great ... if you use a dial-up modem. Personally, I think any more incentive we can give to stamp out dial-up modems, the better.
Re:concretely: this is why you don't need it (Score:3, Informative)
Re: (Score:2)
Re:Screw luddites (Score:2)
So if straight ASCII is so great, why do you use HTML in your Slashdot posts?
Grumble mutter bah humbug
Your sig is particularly appropriate for your post.
Re: (Score:2)
Re:Screw luddites (Score:2)
Stop setting up straw men.
No, it's not a straw man. Look past your "it's always been done that way" attitude, and think about it. Why is a Slashdot HTML forum "good" and a Usenet HTML forum "bad"? History is not a reason. Standards are meant to be enhanced.
Sheesh, some geeks are worse than PHBs on this issue. It reminds me of that shipping company commercial where the boss drones on about the fact that "we've always shipped this way. We will always ship this way" while the young employee stands there mocking him.
Why do I have a feeling that all the old farts are going to have to die off before this issue goes away?
Re: (Score:2)
Re:Screw luddites (Score:2)
You propose a method that many people can't read,
and while it's supposed to look better if you can read it, it frequently just looks like over-elaborate crap.
Geeks believe in content over packaging, at least more than your average person. HTML offers very little in an email or newsgroup environment, where messages are quickly written, but often come with huge overdone packaging. Geeks also like to use their computers via text links, and HTML is not friendly to that. Somehow it should not be surprising that they aren't happy with HTML in those enviroments.
Look ma, no mouse! (Score:2)
Oh no! slashdot is beautifully rendered in a textual browser such as lynx and lightning quick over a 9600 baud cellular modem. If you never tried viewing this site under lynx, you may be surprised at the artistic detail of the formating.
Over to the left where the links take the shape of an ascii candle flame followed by more links presented in an intuitable format. Rob was generous leaving this site accessable to the more mature historical browsers.
Problems, Solutions, and Change (Score:2)
In the case of yEnc, someone found a problem and (Freely, as in public domain even) offered a solution if people wish to adopt it. People are adopting it. Progress is happening.
And then I hit:
And I can't help but think "wow - what a jerk." Why? Not because I don't have newsreaders and email clients capable of handing HTML based messages. Not because I dislike HTML itself (open standard - yay). But because HTML seems to be completely unneccisary in these environments.
Now... sure. There are some forums that might bennifit from it. Times where typefaces and bulletlists, etc. really add value to information. Maybe even times where some forms of information NEED this technology or it becomes very difficult to portray. But I rarely see it.
Instead, HTML based messages in Email and Usenet tend to increase the overhead with no real added value to the content. In some cases, they are used to attempt various shennanigans such as web-bugs (not to mention worms, etc). Little wonder tech-heads dislike it.
Change must happen. But when you run around trying to force change for change's sake rather than to solve real problems, then you simply become a problem yourself.
Re:Screw luddites (Score:2)
Re:Screw luddites (Score:2)
Re:Screw luddites (Score:2)
Not only isn't HTML postings an important "technical advancement", not everyone thinks HTML postings look better, either - surprise.
I, for one, deliberately killfile HTML postings and filter HTML emails because I don't need the funny colours that distract contents, and the security concerns associated with scripts, images, "runnable" postings and cookies.
Plus, every single piece of HTML posting and email I've seen are spam anyway. Why bother?
If you want to post HTML contents, there is a place it should be done and it is called "the web". I don't complain HTML postings in web pages because it is what the HTTP was designed for - and there are already many decent methods built in or around an application called "browser" to protect yourself from security hazards and spams. I cannot say so for newsreaders, at least yet.
Re:Screw luddites (Score:2)
Not only isn't HTML postings an important "technical advancement", not everyone thinks HTML postings look better, either - surprise.
So should the web return to the days of pure ASCII? Should Slashdot disallow any HTML postings? HTML is useful. Look at how it used on Slashdot -- italics, boldface, links, etc. Not to mention that proportional fonts are infinitely easier to read.
I don't complain HTML postings in web pages because it is what the HTTP was designed for...
How can you possible be in favor of HTML on the web, yet not in favor of HTML in Usenet? Put it this way: If Usenet were invented today and they included HTML, could you honestly say it would occur to you to say, "you know what would make this better? ASCII only!"
I cannot say so for newsreaders, at least yet.
Depends on your newsreader. If you're using a Windows newsreader, it typically uses the IE COM component to display the HTML.
Re:Screw luddites (Score:2)
And when was that? You do know that HTML is an intregal part of the WWW, don't you?
How can you possible be in favor of HTML on the web, yet not in favor of HTML in Usenet?
Because the newsreader I use doesn't know what to do with it and the way a lot of the people who use HTML on Usenet do it, it doubles the length of the post.
Put it this way: If Usenet were invented today and they included HTML, could you honestly say it would occur to you to say, "you know what would make this better? ASCII only!"
For a medium that outside of the binaries newsgroups was designed to post text messages? Why wouldn't ASCII only be better for plain text? I actually think a lot of web pages have gone overboard and have worsened the bandwidth/content ratio considerably. There's nothing wrong with lean and mean.
If you're using a Windows newsreader, it typically uses the IE COM component to display the HTML.
While allowing other possibly malicious components to wreak havoc with your system - a problem that never happens with plain text messages, does it?
Go ahead. Post HTML on Usenet. But you should know if I read it, it's going to look like a hard to read text message with a lot of bracketed garbage and I'll be more likely to skip over your posts in the future.
Re:Screw luddites (Score:2)
How do you reconcile putting in this assertion: [etc]
Because it's absurd to argue about features in an application (i.e., Slashdot, Usenet) based on what protocol it uses. Either a feature is useful, or it is not. You can't have it both ways -- either HTML is good for discussion groups (Slashdot OR Usenet) or it is not.
Re:Screw luddites (Score:2)
One huge difference between Slashdot and Usenet is that Slashdot only permits a subset of HTML tags in posts. If Usenet HTML would limit itself to a similar subset, then it would be much more platable.
Also, Slashdot has already limited itself to those who can handle HTML, so why not permit HTML in posts? Usenet is and always has been available by plain text, so HTML gets much more scrutiny.
Re:Usenet? Please go away! (Score:3, Insightful)
On Usenet...sure - you don't get to search and finding a file involves posting a request and hoping someone fulfills, but you get bandwidth - assuming you want to pay for it...you get files that are there (assuming you have decent retention.) and not dependant upon someone being online. And unless you have a crappy server, you don't get halfway through a download and someone decides to kick you off. And 99% of the time, what something is posted as is what it is.
As for the replication - well, there is no one point of failure. As well, you don't have one site getting the shit hammered out of it either. I pay $9/mo for usenet...I get three fast servers to choose from and some high GB limit.
If I had to go to the same server as everyone else, you'd have the same problem that moviefone had when star wars tickets went onsale online - DOA - all with nice corporate control of content.
Re:Usenet? Please go away! (Score:2, Interesting)
Discovery, that's why. Napster (Napster?, talk about dead!) and the like are great for finding a name you already know. By design, that's all it can do. Do a search for "early '80's Boston punk" for example and see what you get. This is where Usenet excels. Enthusiasts congregate into groups and upload files they enjoy, meaning a far better chance that I'll enjoy them too. I'm continually discovering great bands through Usenet that would remain unknown had I relied on name-search based file sharing.
Re:Change we must (Score:2, Insightful)
- A.P.
Re:There's a lot of irony (Score:2, Interesting)
Re:On a related note... (Score:2)
16 so-1-3-0-0.SVCS-RTR1.RES.verizon-gni.net (141.156.255.46) 97.166 ms !A 97.214 ms !A 97.336 ms !A
Where the !A means access to the host is prohibited. (I'm going to have one hell of a time trying to convey this to a tech support moron.)
Re:Why yEnv is good for the software companies (Score:2)
Re:Seems silly to be complaining.... (Score:4, Insightful)
In particular, there is no yenc RFC and yenc does not use MIME which is the agreed upon standard for encoding binary attachments. Yes, uuencode is a gross grandfathered format, but it is still 7 bit clean.
Releasing problematic improperly specified encodings that break internet protocols is not being a good citizen. "it works" is a poor justification. it does not work, and breaks compliant software.
-Kevin
Re:All of Mr. Nixon's points are easily refuted (Score:3, Informative)
Well people do now have to think of yENC's longer lines. Where before my ISP newserver let me post 5000 lines of uue material now I only get 2280 with yENC. Longer lines under yENC. That's why Agent 1.91 has such a small line length as the default. Yes the payload is the same but you have to think a little different or just rediscover your servers limits.