Submission + - New tools help neuroscientists analyze 'big data' (esciencenews.com)
mspohr writes: "New technologies for monitoring brain activity are generating unprecedented quantities of information. That data may hold new insights into how the brain works — but only if researchers can interpret it. To help make sense of the data, neuroscientists can now harness the power of distributed computing with Thunder, a library of tools developed at the Howard Hughes Medical Institute's Janelia Research Campus.
"Freeman chose to build on a new platform called Spark. Developed at the University of California, Berkeley's AMPLab, Spark is rapidly becoming a favored tool for large-scale computing across industry, Freeman says. Spark's capabilities for data caching eliminates the bottleneck of loading a complete data set for all but the initial step, making it well-suited for interactive, exploratory analysis, and for complex algorithms requiring repeated operations on the same data. And Spark's elegant and versatile application programming interfaces (APIs) help simplify development. Thunder uses the Python API, which Freeman hopes will make it particularly easy for others to adopt, given Python's increasing use in neuroscience and data science.
Researchers can find everything they need to begin using the open source library of tools at http://freeman-lab.github.io/t...
To make Spark suitable for analyzing a broad range of neuroscience data — information about connectivity and activity collected from different organisms and with different techniques — Freeman first developed standardized representations of data that were amenable to distributed computing. He then worked to express typical neuroscience workflows into the computational language of Spark."
"Freeman chose to build on a new platform called Spark. Developed at the University of California, Berkeley's AMPLab, Spark is rapidly becoming a favored tool for large-scale computing across industry, Freeman says. Spark's capabilities for data caching eliminates the bottleneck of loading a complete data set for all but the initial step, making it well-suited for interactive, exploratory analysis, and for complex algorithms requiring repeated operations on the same data. And Spark's elegant and versatile application programming interfaces (APIs) help simplify development. Thunder uses the Python API, which Freeman hopes will make it particularly easy for others to adopt, given Python's increasing use in neuroscience and data science.
Researchers can find everything they need to begin using the open source library of tools at http://freeman-lab.github.io/t...
To make Spark suitable for analyzing a broad range of neuroscience data — information about connectivity and activity collected from different organisms and with different techniques — Freeman first developed standardized representations of data that were amenable to distributed computing. He then worked to express typical neuroscience workflows into the computational language of Spark."
