Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
×

Comment "Their devices" (Score 1, Troll) 41

I'm so glad I got off the Apple bandwagon when they started ignoring the Mac for iPhone.

I actually had linux running on my MBP for a while before buying a PC laptop to when it was time to upgrade.

Every convenience feature is potential spyware when they could have done it cryptographically secure from the beginning.

I actually had a good chat with on-staff cryptographers back in the 90's. This one gal was a genius at elliptic curves

Those were the days.

Comment Remote School (Score 0) 47

Some of these comments make no sense in the reality that remote schooling is now mandatary and enforced by law.

Kids in poor families who need to be on Zoom 6 hours a day do not have other options claimed here.

Unemployment meetings, seniors' telehealth, etc. all fit similar patterns. There is often no choice given.

Fix those maybe before claiming that a 1GB cap is plenty.

Comment "Thanks, Americans!" (Score 1) 222

Sure, Americans pay taxes and inflated costs of goods to provide military security for Europe, drug development, technology development, etc. so they have more money for healthcare, arts, and leisure.

Everybody should need to work less as technology and productivity increases while maintaining the same standard of living, but nooooo, we can't have nice things.

Materialism plays a small part but GenZ has correctly realized that they can never live as well as their grandparents under this regime.

Boomers are happy to shout "avacado toast!" at them from their country-club golf carts.

I wrote here about coming Age Wars a decade ago. Let's hope somebody comes to thier senses.

Comment Re:Oh for crying out loud. (Score 1) 94

Unfortunately this is the logical conclusion of letting Microsoft dominate desktop computing for a generation and investing too much in software that relies on proprietary data formats. There isn't much effective competition for many of the users who are still using a Windows desktop, because they're locked in by whatever Windows-only software they still rely on. Those who could jump ship to mobile devices, games consoles, Apple desktop or Linux probably already have.

Comment Re:Flash is costly? (Score 5, Informative) 37

Creating the training dataset is the *last* step. I have dozens of TB of raw data which I use to create training datasets that are only a few GB in size. Of which I'll have a large number sitting around at any point in time.

Take a translation task. I start with several hundred gigs of raw data. This inflates to a couple terabytes after I preprocess it into indexed matching pair datasets (for example, if you have an article that's published in N different languages, it becomes (N * N-1) language pairs - so, say, UN, World Bank, EU, etc multilingual document sets greatly inflate). I may have a couple different versions of this preprocessed data sitting around at any point in time. But once I have my indexed matching pair datasets, I'll weighted-sample only a relatively small subset of it - stressing higher-quality data over lower quality and trying to ensure a desired mix of languages.

But what I do is nothing compared to what these companies do. They're working with common crawl. It grows at a rate of 200-300 TB per month. But the vast majority of that isn't going to go into their dataset. It's going to be markup. Inapplicable file types. Duplicates. Junk. On and on. You have to whittle it down to the things that are actually relevant. And in your various processing stages you'll have significant duplication. Indeed, even the raw training files... I don't know about them, but I'm used to working with jsons, and that adds overhead on its own. Then during training there's various duplications created for the various processing stages - tokenization, patching with flash attention, and whatnot.

You also use a lot of disk space for your models. It's not just every version of the foundation you train (and your backups thereof) - and remember that enterprise models are hundreds of billions to trillions of FP16 parameters in their raw states - but especially the finetune. You can make a finetune in like a day or so; these can really add up.

Certainly disk space isn't as big of a cost as your GPUs and power. But it is a meaningful cost. As a hobbyist I use a RAID of 6 20TB drives and one of 2 4TB SSDs. But that's peanuts compared to what people working with common crawl and having hundreds of employees each working on their own training projects will be eating up in an enterprise environment.

Slashdot Top Deals

People will buy anything that's one to a customer.

Working...