i hadn't seen 4 before. interesting. i agree completely, but it is fun to speculate!
no nodes came ready - we assembled/installed all the motherboards, cpu's, memory, disks, power etc.. on custom made shelves. they are arranged on four sides of a cube and we cool everything by directly blowing cold air into the centre of the frame. a custom insert distributes even airflow over the shelves. its a compact design and easy to maintain/upgrade. grounding straps essential? what is your source for that??
because its hard to train the human brain to make 10^13 floating point calculations each second and solve the poisson and navier-stokes equations..
surprisingly few - a couple of bad motherboards (or static
;). its only been up for a week or so and we are still testing/installing stuff before making user queues live.
see my reply to the post about software. we have some codes that run on GPUs but the main astrophysics codes are too complex to re-write or to fit onto a GPU. some subroutines can use the GPU as an accelerator and we hope to find the extra ~100,000 money units, to put GPUs on all the motherboards at some point.
exactly. gravity is a long range force, so all parts of the simulation volume are constantly transferring information. we need high bandwidth and low latency networks.
we benchmarked various configurations with our codes. then we made a tender for the motherboards, memory, cpu's. we got a pretty good price
i posted the original zBox1 specs/website here about eight years ago. our server was immediately slashdotted & crashed. quickly fixed it and we had over a million page views in 2 days
its the same design - shelves of motherboards arranged in a cube. cold air is blown into the center of the machine which has a custom design to allow even air flow over all the shelves. we could possibly run the machine without the cpu coolers which would double the density. its a little over one cubic metre and needs about 40kW of power.
we use various astrophysics simulation codes, i.e. GASOLINE, PKDGRAV, RAMSES etc. some are developed by us. they are all MPI and solve the coupled gravitational and hydrodynamic equations that can describe the dark matter and baryons evolving in the expanding universe. memory and speed of the computer limit the resolution that can be attained, so various "sub-grid" physical processes have to be treated carefully. for cosmological simulations we know the initial conditions - those are the fluctuations that we can read off the microwave background. they show the universe was hot, dense and smooth early on. the codes follow the perturbations into the non-linear regime when dark matter haloes, stars and galaxies form. we can then compare the properties of simulated structures with observational data etc.
your assumptions are close to mine when i estimated the ~1% compute capability of the brain. individual neurons send an outgoing signal depending on the amount and rate of incoming signals), but i am an astrophysicist, not a neuroscientist
;) the zBox4 can calculate at over 10 petaflops.
it took ~year to acquire the funds, benchmark tests, fix the design, make the tender for the parts etc, but all the construction was done in 3x8hr shifts
Hardware: CPUs: 384 Intel Xeon E5-2660 (8 cores @ 2.2 GHz, 95 W), 3072 cores in total Main Boards: 192 Supermicro X9DRT-IBQF (2 CPUs per node) on-board QDR Infiniband RAM: Hynix DDR3-1600, 4 GB/core, 64 GB/node, 12.3 TB in total SSD: 192 OCZ 128 GB high performance Vertex 4 drives, 24.6 TB in Total HPC Network: QLogic/Intel QDR Infiniband in 2:1 fat tree (9 leaf and 3 core switches) Gbit Ethernet and seerate dedicated 100 Mbit management networks Power usage (full load): 44 kW Dimensions (L x W x H): 1.5m x 1.5m x 1.7m Number of Cables: Power: 112 IB: 300 Ethernet: 388 Cost: under 750'000 CHF System Configuration OS: Scientific Linux version 6.3 Queue System: Slurm Swap: 8 GB on node-local SSD drive Temp Files: 110 GB on node-local SSD drive Booting: from node-local SSD, or over Ethernet Storage System (existing): Capacity: 684 TB formatted Raid-6 Lustre file system with 50 OSTs 342 x 1.5 TB HDD and 171 x 2.0 TB HDD Physical dimensions: 48 standard rack units 10 Gb Ethernet and 40 Gb (QDR) Infiniband 3 Controllers using Intel E5645 2.4 GHz CPUs Tape Robot: Capacity: 800 TB, 437 tape slots 4 x LTO-5 drives and 2 x LTO-3 drives 54 TB high speed tape cache (108 TB raw storage) 40 Gb (QDR) Infiniband connected
An anonymous reader writes "Time lapse video of students and postdocs at the University of Zurich constructing the zBox4 supercomputer. The machine has a theoretical compute capacity of ~1% of the human brain and will be used for simulating the formation of stars, planets and galaxies."
Link to Original Source
Link to Original Source