A feature story posted by the University of Texas at Austin Texas Advanced Computing Center’s Makeda Easter notes that many phenomena we encounter in everyday life, including sound, heat, and fluid flow, can be explained by a class of mathematics called partial differential equations; problems that can be solved using numerical simulations to give researchers insight into a broad range of science.
The report notes that for certain applications, because the number of variables required in these numerical simulations can be in the billions, solving large-scale problems of this magnitude requires not only massive computing power, but also algorithms that can harness this power to take full advantage of large-scale supercomputing systems. Now, researchers at The University of Texas at Austin (UT Austin’s) Institute for Computational Engineering and Sciences (ICES) are making this a reality.
George Biros, director of the specialized ICES group, Parallel Algorithms for Data Analysis and Simulation or PADAS, and a long-time user of advanced computing resources at the The University of Texas at Austin’s Texas Advanced Computing Center (TACC). One of the leading advanced computing centers in the nation, TACC and its staff assist researchers like Biros and his team to enable new discoveries that advance science and society.
“The focus of my group’s research is integrating mathematics, computer science and applications, and then scaling them to supercomputing platforms,” said Biros “Our goal is to design and deploy algorithms that capture the phenomena explained by partial differential equations. Without TACC, this work would be nearly impossible…. For us, using the Xeon Phi coprocessors is like using the turbocharger on a fast car. It enables us to be able to solve more complicated science problems, get higher resolutions (and thus, better science), and have more confidence in our simulations.”
Based on numerical methods, Biros and his team of researchers and students develop scientific computing libraries that serve as the basic building blocks for researchers creating various technologies. The team’s efforts are funded in part by the National Science Foundation, the Department of Energy, the Department of Defense, the National Institutes of Health, and UT Austin.
The article reports that one PADAS student, Amir Gholami, is developing software to help doctors treat gliomas, a type of malignant brain tumor, noting that currently there are about 60,000 annual cases of gliomas in the U.S., and despite advances in medical technology the survival rate has not changed in the past 30 years.
“One of the main challenges for doctors treating this disease is that MRIs do not provide the whole picture,” Gholami is cited observing. “We’re trying to use mathematics to treat gliomas and give neurosurgeons and radiation oncologists greater confidence in removing the tumor while sparing healthy tissue.”
Gholami’s software allows doctors to better quantify infiltration of gliomas in healthy tissue. Combined with MRIs, this software could help improve the current diagnostic skill and better guide surgeries and radiation treatments to remove gliomas. In a submitted journal paper entitled “Image-driven parameter estimation for low grade gliomas,” coauthors Amir Gholami, Andreas Mang, and George Biros present a numerical scheme for solving a parameter estimation problem for a model of low-grade glioma growth, noting that their goal is to estimate tumor infiltration into the brain parenchyma for a reaction-diffusion tumor growth model. The researchers use a constrained optimization formulation that results in a system of nonlinear partial differential equations (PDEs), and in their formulation, they estimate the parameters using the data from segmented images at two different time instances, along with white matter fiber directions derived from diffusion tensor imaging (DTI). The parameters they seek to estimate are the spatial tumor concentration and the extent of anisotropic tumor diffusion. The optimization problem is solved with a Gauss-Newton reduced space algorithm. They present the formulation, outline the numerical algorithms and conclude with numerical experiments on synthetic datasets, reporting that the results show the feasibility of the proposed methodology.
Another graduate student, Dhairya Malhotra, is developing software that can solve N-body problems. N-body problems were initially developed in the computational astrophysics community for the simulation of gravitational interactions. They are also applicable to many other fields of science and engineering. The PADAS group has created N-body software that finds applications to fluid mechanics.
“The solver we are working on is open source and has applications in a variety of fields from measuring blood flow to porous media flow. For instance, when you want to study how ground water flows or how petroleum flows through the ground,” Malhotra said. “We’ve received interest from researchers across the country.”
Malhotra, Gholami, and Biros’ paper on the solver entitled “A volume integral equation Stokes solver for problems with variable coefficients” was also selected as a finalist in the Best Student Paper category at the upcoming supercomputing (SC14) conference to run November 16-21 in New Orleans and providing a forum for new ideas and spotlighting the most original and fascinating scientific and technical applications from around the world, SC14 will bring together the HPC community — an unprecedented array of scientists, engineers, researchers, educators, students, programmers, system administrators, and developers — for an exceptional program of technical papers, tutorials, timely research posters, and Birds-of-a-Feather (BOF) sessions.
In their paper, the TACC researchers present a novel numerical scheme for solving the Stokes equation with variable coefficients in the unit box. They report that their scheme is based on a volume integral equation formulation. Compared to finite element methods, their formulation decouples the velocity and pressure, generates velocity fields that are by construction divergence free to high accuracy and its performance does not depend on the order of the basis used for discretization. In addition, they employ a novel adaptive fast multipole method for volume integrals to obtain a scheme that is algorithmically optimal, supports non-uniform discretizations, and is spectrally accurate. To increase per node performance, they have integrated their code with both NVIDIA and Intel accelerators. In their largest scalability test, they report that they solved a problem with 20 billion unknowns, using a 14-order approximation for the velocity, on 2048 nodes of the Stampede system at the Texas Advanced Computing Center. They achieved 0.656 petaFLOPS for the overall code (23% efficiency) and one petaFLOPS for the volume integrals (33% efficiency). As an application example, they simulate Stokes flow in a porous medium with highly complex pore structure using a penalty formulation to enforce the no slip condition.
Easter notes that while the PADAS group’s software allows researchers to accomplish a broad range of scientific research, one major challenge is scaling the software to advanced computing technologies.
“Even though supercomputers are very fast at performing calculations, they can be significantly slower if the data required for the calculations is not readily available,” Biros is cited explaining.
“Supercomputers organize storage and retrieval of data in a hierarchical manner. Small pieces of data are fast to access. But larger pieces of data require more time. When data is not available, no calculations can be performed. This decreases the efficiency of running code and slows the time to solutions… we have to be able to orchestrate how data is brought into the processor (where the computation takes place) to ensure efficient utilization of computing. If we aren’t doing so, then the processors are spinning idle consuming energy, but performing no work. In other words, we could be wasting taxpayer dollars.”
The PADAS group works closely with TACC to test and improve the scalability and efficiency of their software. Using many of TACC’s advanced computing systems including Stampede (the sixth most powerful supercomputer in the world), as well as the center’s Lonestar, and Maverick supercomputers. Stampede is a Dell PowerEdge C8220 Cluster with Intel Xeon Phi coprocessors, and as one of the largest computing systems in the world for open science research, the system provides unprecedented computational capabilities to the national research community, enabling breakthrough science that has never before been possible.
To obtain even greater efficiency, the researchers rely on Stampede’s Intel Xeon Phi coprocessors. The Xeon Phi coprocessors are the innovative component of the Stampede system and provide more than seven petaflops of performance to an already powerful supercomputer.
With a theoretical peak performance of nearly 10 petaflops, Stampede is powered by two different Intel processor architectures: the base cluster, which comprises 6,400 nodes with two Intel Xeon E5 processors each, providing 2.2 petaflops of computing power; and the innovative component, which comprises 6,880 Intel Xeon Phi coprocessors that add more than seven petaflops of performance. The scale of Stampede delivers opportunities in computational science and technology research, from highly parallel algorithms to high-throughput computing, and from scalable visualization to next generation programming languages.
Any researcher at a U.S. institution can submit a proposal to request an allocation of cycles on the system. The request must describe the research, justify the need for such a powerful system to achieve new scientific discoveries, and demonstrate that the proposer’s team has the expertise to utilize the resource effectively.
“For us, using the Xeon Phi coprocessors is like using the turbocharger on a fast car,” Biros tells Easter. “It enables us to be able to solve more complicated science problems, get higher resolutions (and thus, better science), and have more confidence in our simulations.”
Working through these scalability challenges is what makes the PADAS group unique. It also makes their impact widespread.
Adds Biros: “The big picture for us is developing scientific software that can be used in an efficient and reliable way. The more efficient they are, the lower the cost of clinical applications and the more likely they are to be used to design devices in the future that positively impact many areas of health and science.”
The University of Texas at Austin
Texas Advanced Computing Center
Institute for Computational Engineering and Sciences (ICES)
Parallel Algorithms for Data Analysis and Simulation (PADAS)
Supercomputing (SC14) conference
Cornell University Library
Texas Advanced Computing Center