Submission + - New tools help neuroscientists analyze 'big data' (esciencenews.com)
"Freeman chose to build on a new platform called Spark. Developed at the University of California, Berkeley's AMPLab, Spark is rapidly becoming a favored tool for large-scale computing across industry, Freeman says. Spark's capabilities for data caching eliminates the bottleneck of loading a complete data set for all but the initial step, making it well-suited for interactive, exploratory analysis, and for complex algorithms requiring repeated operations on the same data. And Spark's elegant and versatile application programming interfaces (APIs) help simplify development. Thunder uses the Python API, which Freeman hopes will make it particularly easy for others to adopt, given Python's increasing use in neuroscience and data science.
Researchers can find everything they need to begin using the open source library of tools at http://freeman-lab.github.io/t...
To make Spark suitable for analyzing a broad range of neuroscience data — information about connectivity and activity collected from different organisms and with different techniques — Freeman first developed standardized representations of data that were amenable to distributed computing. He then worked to express typical neuroscience workflows into the computational language of Spark."