Follow Slashdot stories on Twitter

 



Forgot your password?
typodupeerror

Comment Re:2 million gigabytes of porn (Score 1) 233

Sure thing: You need lots of RAM when running deduplication as the hashes are stored in the ARC. Estimates run from 1 to 5 GB of RAM per TB of storage you have deduplication enabled on. I'm running it on a NAS4Free box (FreeBSD) with 16 GB of memory. The 2 TB I have deduped works fine, but if memory gets right for the ARC, it will go to spinning disks; you don't want that. When working with ZFS, it's my own personal rule-of-thumb to use an SSD for the ZIL and L2ARC. Your memory might get tight, but at least you'll falling back to an SSD and not any of the conventional drives in your system.

My sysctls are tweaked with:
vfs.zfs.txg.timeout = 2 (helps with burstiness)
vfs.zfs.l2arc_noprefetch = 0 (enables read ahead to the L2ARC)
vfs.zfs.l2arc_write_boost and vfs.zfs.l2arc_write_max are also tweaked well above the default (ymmv with this, test on your own rig)
vfs.zfs.arc_max is set in loader.conf to keep memory free. Even though ZFS is pretty good at handing back memory, this helps for my situation.

FWIW I tried ZFS on Linux and found it very slow and it would choke under high load. Even a SATA expander to a small 4 disk chassis running RAIDz1 would have the pool to detach under high load. I've been sticking with FreeBSD for any ZFS work for a few years now. ZFS is pretty good at running stock with defaults, but when you play with deduplication, it can benefit from some gentle hand-holding.

Good luck!

Comment Re:2 million gigabytes of porn (Score 1) 233

My only real-world experience with deduping is ZFS and NetApp, so it's limited. From my own experience/testing, the overhead of deduping 4-8 GB video files was high and gained nothing. (granted I don't keep multiple copies of videos, but this was testing)

Deduping my local copies of all my personal data has netted decent results, but that's mainly document-type files. Like anything: YMMV :)

Comment Re: Box (Score 2) 63

On our Apple TV 4th gen (and iPhones and iPads), we use a streaming app called Infuse. It scours our server (many TBs), fetches metadata, and presents a Netflix-like interface for all our stuff.

It has replaced our old PopcornHour boxes.

Comment Re:2 million gigabytes of porn (Score 1) 233

Deduping correctly is a fine balance of resource use. At such a grand scale, the overhead might well sink it.

At work I found things Just Ran Better when I applied lz4 compression on large ZFS pools. Space is cheap, if more storage is needed, just buy it. At home I've experimented on a relatively small 2 TB mirror with lz4 and ZFS' dedupe enabled. It took some tweaking to eliminate most of the 'bursty-ness' associated with it. The other ~24 TB is simply lz4 with no dedupe and it has never needed any real special attention.

tl;dr: dedupe is voodoo that can offer wonderful things, but can also bite you right in the anus. You'll be wearing Depends to soak up all that blood oozing from your sphincter, but hey, you'll have a dedupe hit of1.39!

Comment Re:"terms of" considered harmful (Score 1) 158

We have tried several media players over the years (PopcornHour hardware, Kodi, etc.) and the best we've found is Infuse Pro for iOS. Works on AppleTV 4th gen beautifully. Presents a Netflix-like interface with fetched metadata. Ours mounts a RO NFS share on a NAS4Free server.

No relation other than a very satisfied customer.

Slashdot Top Deals

The intelligence of any discussion diminishes with the square of the number of participants. -- Adam Walinsky

Working...