A team of scientists from the U.S.
Department of Energy’s Lawrence Berkeley National Laboratory (Berkeley
Lab) has won a prestigious Gordon Bell Prize, sponsored by the Association
for Computing Machinery (ACM), for special achievement in high performance computing
for their research into the energy harnessing potential of nanostructures. Their
method, which was used to predict the efficiency of a new solar cell material,
achieved impressive performance and scalability.
A test run of LS3DF, which took one hour on 17,000 processors of the Franklin supercomputer at NERSC), performed electronic structure calculations for a 3500-atom ZnTeO alloy. Isosurface plots (yellow) show the electron wavefunction squares for the bottom of the conduction band (left) and the top of the oxygen-induced band (right). The small grey dots are Zn atoms, the blue dots are Te atoms, and the red dots are oxygen atoms. (Image courtesy of Lin-Wang Wang)
The ACM Gordon Bell Prize annually recognizes the best performance of scientific
applications on supercomputers. This year’s prize, presented in a special
category for algorithm innovation, was announced Thursday, Nov. 20, at the awards
session of the SC08 conference in Austin.
The Berkeley Lab researchers used three of the most advanced scientific computing
facilities of the Department of Energy (DOE) Office of Science for this award-winning
work: the National Energy Research Scientific Computing Center (NERSC) at Berkeley
Lab, the Argonne Leadership Computing Facilities (ALCF) at Argonne National
Laboratory and the National Center of Computational Sciences (NCCS) at Oak Ridge
National Laboratory. Their study was titled: “Linearly Scaling 3D Fragment
Method for Large-Scale Electronic Structure Calculations.”
Nanostructures, tiny materials 100,000 times finer than a human hair, may hold
the key to energy independence. Scientists believe that a fundamental understanding
of nanostructure behaviors and properties could provide solutions for curbing
our dependence on petroleum, coal and other fossil fuels.
To better understand and demonstrate the potential of nanostructures, the Berkeley
Lab researchers simulated their behavior through development of the Linearly
Scaling Three Dimensional Fragment (LS3DF) method. These computer algorithms
use a novel “divide-and-conquer” technique to efficiently gain insights
into how nanostructures function in systems with 10,000 or more atoms.
The LS3DF team consisted of Berkeley Lab’s Lin-Wang Wang, Byounghak Lee,
Hongzhang Shan, Zhengji Zhao, Juan Meza, Erich Strohmaier and David Bailey,
an agregate of materials scientists, mathematicians and computer scientists
contributing their own special expertise to solve this problem.
The LS3DF application ultimately achieved a speed of 442 teraflop/s (442 trillion
calculations per second) on a Cray XT5 system with 147,146 cores at the NCCS.
The Berkeley Lab researchers were also able to run the code on the IBM BlueGene/P
system at Argonne, reaching 224 teraflop/s on 163,840 cores, or 40.5 percent
of the system’s peak performance capability.
The team first ran the LS3DF application on 36,864 cores of the Cray XT4 (Franklin)
at NERSC, achieving 135 Tflop/s. These initial results at NERSC provided the
key scientific insights from the application.
“By incorporating the correct chemical formulas into efficient computer
programs, scientists can learn a lot about the structures and properties of
molecules and solid,” said. Lin-Wang Wang, a computational material scientist
who led the Berkeley Lab team. “I like to think of computers as chemistry’s
third pillar. In most cases, computer simulations complement information obtained
by chemical experiments, but in some cases they can also predict unobserved
A science run using LS3DF, which took one hour on 17,280 cores of the NERSC
Franklin system, computed the electronic structure of a 3,500-atom ZnTeO alloy.
This run verified that the code could be used to compute properties of the ZnTeO
alloy that previously had been experimentally observed. The simulation led to
a prediction for the efficiency of this alloy as a new solar cell material.
LS3DF offers a more efficient way for calculating energy potential because
it is based on the observation that the total energy of a large nanostructure
system can be broken down into small pieces, and each piece can be calculated
separately. More traditional methods calculate the entire structure as a whole
system and are much more time consuming and resource intensive. Because LS3DF
scales almost perfectly with the number of compute cores, it is the first electronic
structure code that runs efficiently on computer systems with tens to hundreds
of thousands of cores.
“We are excited by the results we are seeing,” said LS3DF team
member Meza, who heads Berkeley Lab’s High Performance Computing Research.
“The efficiency of LS3DF on these large computer systems is impressive,
but the real story is the power of algorithms. Using a linear scaling algorithm,
we can now study systems that would otherwise take over 1,000 times longer on
even the biggest machines today. Instead of hours, we would be talking about
months of computer time for a single study.”
Getting codes to run with such high efficiencies on massively parallel machines
is not a trivial task. Bailey, Shan and Strohmaier of the DOE Office of Science’s
Scientific Discovery through Advanced Computing (SciDAC) Performance Engineering
Research Institute (PERI) worked hand-in-hand with Wang and his colleagues to
analyze the performance of LS3DF and to identify potential performance improvements.
Responding to this analysis, Berkeley Lab researchers assisted with a major
revision of the code, which led to the prize-winning submission.
“The computational power we have is staggering and it is important to
make sure that each research project can effectively harness the power of Argonne’s
Intrepid and optimize their calculations”, said Katherine Riley, the ALCF
computational scientist who worked with the Berkeley Lab team. “Not only
can we drastically reduce the time it takes to generate results, we can help
scientists ask different questions and develop new insights in order to accelerate
Once the LS3DF code had been optimized it was a matter of days before it was
running at each of the DOE supercomputing facilities. Oak Ridge National Laboratory
invited Wang and other Gordon Bell finalists to carry out runs on ORNL’s
leadership Cray supercomputer, Jaguar. In Wang’s case, the winning simulation
was achieved after only two runs over a two-day period, demonstrating the ease
of porting - and running - high-performance applications on the Cray XT architecture.
The project had previously been awarded time on Jaguar under DOE’s Innovative
and Novel Computational Impact on Theory and Experiment (INCITE) program.
“We still don’t quite understand how the electron moves around
in a nanostructure, and how such properties depend on the size, geometry, composition
and surface passivations,” said Wang. “Understanding this dependence
will allow us to design nanostructures for desired applications. Using our improved
LS3DF method will help us to understand and predict these properties.”
The ALCF, NCCS and NERSC are funded by the Office of Advanced Scientific Computing
research in the DOE Office of Science, providing some of the world’s most
powerful computing resources and support to thousands of researchers around