Clues Left By Brad? (Re: "smart migrator") (Score 1)

Hi. Perhaps it's more than just the power failure? Perhaps BradFitz's last cached LJ entries may hold some additional clues as to why it's taking so long after the power failure to bring LJ back up? Yes, perhaps they do. Enjoy: Jan. 12th, 2005 @ 02:31 am *yawn* It's one of those database nights. Watching 3 sets of progress bars move way too slowly. Wish there were something on Tivo at least to kill some time. Jan. 11th, 2005 @ 09:15 pm smart migrator I just wrote a database migration script which, after each chunk of data moved, asks the load balancer (Perlbal) the free user queue depth. If more than 10 (less is just noise), it sleeps a second and asks again. Only once the queues are empty does it migrate more data. End result: it moves data as fast as it can, without affecting page response times. I've been meaning to make a generic wrapper for any utility, where the wrapper parent watches the load balancers, and the child does its work at full speed, but the parent will occasionally SIGSTOP/SIGCONT it...... haven't got around to that yet. The generic wrapper could even have another pluggable-child as its rate-limit determiner, so anybody could use it. Jan. 11th, 2005 @ 01:24 pm Parallel compression Are there any multi-threaded compression algorithms, or at least wrappers/formats for interleaved compression, with variable interleave size? It'd be nice to take advantage of multiple processors when gzipping 380 GB, while still doing sequential reads, even if the resultant file was non-standard.