Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror

Comment Start Small (Score 1) 1

I'm in Computational Biology, and I'd say that the most valuable skills you should learn (and the ones most often seen in this field) are more mathematical and/or statistical than "big data." Understanding how to properly normalize your data or calculate a p-value will take you much further than being able to spin up a 100-node Hadoop instance in most labs.

I think you should spend the first year on your home PC. Download RStudio and work through a few R Tutorials, then find some data/questions that interest you and poke around. Post your results to a blog so that you'll have something to show for the time you spend and release the code on GitHub so that it's open to future employers.

I'd say get comfortable with a data analysis language (R will probably serve you best currently), and a data manipulation language (Python, Perl, etc.) and start asking questions of data that's around you (your email archives, a log of Internet sites you visit, your spending records, etc.). Once you've found that well-designed algorithms can't handle some dataset you're looking at, then look at Hadoop and other "big-data" projects.

When you're ready, I'd steer your towards Next-Generation Sequencing data. Most of the bioinformatics questions being asked (and funded) now have at least some interaction with NGS, and analysts capable of working with that data are highly valuable. Check out the 1,000 Genomes project when you're ready to start playing with free Sequencing data.

Comment On-The-Job Training (Score 1) 1

I think most people pick these skills up at work -- seeing these practices employed by others. I guess the short answer would be: go work for a company which is already employing people who understand these topics and learn from them for a couple of years.

Of course, that doesn't help you if you're not looking to change careers...

Comment Re:Moore's law is too slow (Score 1) 126

Interesting. I view this from a completely different perspective: if DNA sequencing really is outpacing Moore's Law, that just means that the results become disposable. You use them for your initial analysis and store whatever summarized results you want from this sequence, then delete the original data.

If you need the raw data again, you can just resequence the sample.

The only problem with this approach, of course, is that samples are consumable; eventually there wouldn't be any more material left to sequence. So this wouldn't be appropriate in every situation.

Comment Re:Moore's law is too slow (Score 1) 126

I assume you're talking about incoming data, not the final DNA sequence. As I understand it the final result is 2 bits/base pair and about 3 billion base pairs so about a CD's worth of data per human. And if you were talking about a genetic database I guess 99%+ is common so you could just store a "reference human" and diffs against that. So at 750 MB for the first person and 7.5 MB for each additional person I guess you could store 2-300.000 full genetic profiles on a 2 TB disk. Probably the whole human race in less than 100 TB.

The incoming data is image-based, so yes, it will be huge. Regarding the sequence data: yes; in its most condensed format it could be stored in 750MB. There are a couple of issues that you're overlooking, however:
1. The reads aren't uniform quality -- and methods of analysis that don't consider the quality score of a read are quickly being viewed as antiquated. So each two bit "call" also has a few more bits representing the confidence in that call.
2. This technology is based on redundant reads. In order to get to an acceptable level of quality, you want at least ~20 (+/- 10) reads at each exonic loci.
So that 750MB you mention for a human genome grows by a factor of 20, then by another factor of 2 or 3, depending on how you store the quality scores.

Your suggestion of deduplicating the experiments could work, but definitely not as well as you think because of all the "noise" that's inherent in the above two steps.

If you really just wanted to unique portions of a sample, you could use a SNP array which just reads the samples at specific locations which are known to differ between individuals. Even with the advances in the technology, the cost of sequencing a genome still isn't negligible. For most labs, it's still cheaper to store the original data for reanalysis later.

Comment Working on Commission? (Score 1) 608

I often stumble across some product on Wikipedia that I'm interested in buying (album, book, etc.). I actually would find it very convenient if such pages had a "Purchase this Item" link. I'm sure Amazon would kick in a few million for that privilege, or you could use their pre-existing referral program. I think most users would view those links as added value to Wikipedia.

Comment Re:I think there is... (Score 1) 2

I think there is a software for that. But it may not be out there iin public. Maybe developers are just keeping them private. Nevertheless, there's a high probability that the software you will need already exists. what is more important is your ETA. So you may find other means. Laptop Troubleshoot Tips

Of course! I'll just fix their laptops! Why hadn't I thought of that?

Comment Re:NoSQL? Waittaminute (Score 2, Insightful) 280

A big-ass Oracle or IBM-DB2 can do the job if you pay enough for tuning.

Why is it that, ever since Key-Value DBs came into vogue, that relational databases instantly got perceived as so neanderthal?

A normal-ass Oracle database would surely be just fine for storing a no-fly list which, by necessity, has magnitudes of order less than 6.whatever billion names; I'm guessing it would do so without much tuning, also.

Comment Re:"Sue fucking everyone" (Score 1) 949

I'm not familiar with the intricacies of the Torrent protocols, but it seems like this group would either need to be in one of two groups:
A.) Connect to a swarm as a "spectator" not uploading or downloading any data.
B.) Connect to the swarm and actively upload/download.

If A., it seems like it would be hard to prove that any IP logged as participating in the swarm is actively engaged in any malicious behavior. If B., aren't they (the group) guilty of the same crimes of which they're accusing these other people?

I guess I just don't see how they could assure the courts of a crime being committed without having to participate in the exact same action in order to prove it.

Slashdot Top Deals

Competence, like truth, beauty, and contact lenses, is in the eye of the beholder. -- Dr. Laurence J. Peter

Working...