2015

B. D. Goodwin, C. R. Butson.
**“Subject-Specific Multiscale Modeling to Investigate Effects of Transcranial Magnetic Stimulation,”** In *Neuromodulation: Technology at the Neural Interface*, Vol. 18, No. 8, Wiley-Blackwell, pp. 694--704. May, 2015.

DOI: 10.1111/ner.12296

**OBJECTIVE:**

Transcranial magnetic stimulation (TMS) is an effective intervention in noninvasive neuromodulation used to treat a number of neurophysiological disorders. Predicting the spatial extent to which neural tissue is affected by TMS remains a challenge. The goal of this study was to develop a computational model to predict specific locations of neural tissue that are activated during TMS. Using this approach, we assessed the effects of changing TMS coil orientation and waveform.**MATERIALS AND METHODS:**

We integrated novel techniques to develop a subject-specific computational model, which contains three main components: 1) a figure-8 coil (Magstim, Magstim Company Limited, Carmarthenshire, UK); 2) an electromagnetic, time-dependent, nonhomogeneous, finite element model of the whole head; and 3) an adaptation of a previously published pyramidal cell neuron model. We then used our modeling approach to quantify the spatial extent of affected neural tissue for changes in TMS coil rotation and waveform.**RESULTS:**

We found that our model shows more detailed predictions than previously published models, which underestimate the spatial extent of neural activation. Our results suggest that fortuitous sites of neural activation occur for all tested coil orientations. Additionally, our model predictions show that excitability of individual neural elements changes with a coil rotation of ±15°.**CONCLUSIONS:**

Our results indicate that the extent of neuromodulation is more widespread than previous published models suggest. Additionally, both specific locations in cortex and the extent of stimulation in cortex depend on coil orientation to within ±15° at a minimum. Lastly, through computational means, we are able to provide insight into the effects of TMS at a cellular level, which is currently unachievable by imaging modalities.

C. Gritton, M. Berzins, R. M. Kirby.
**“Improving Accuracy In Particle Methods Using Null Spaces and Filters,”** In *Proceedings of the IV International Conference on Particle-Based Methods - Fundamentals and Applications*, Barcelona, Spain, Edited by E. Onate and M. Bischoff and D.R.J. Owen and P. Wriggers and T. Zohdi, CIMNE, pp. 202-213. September, 2015.

ISBN: 978-84-944244-7-2

While particle-in-cell type methods, such as MPM, have been very successful in providing solutions to many challenging problems there are some important issues that remain to be resolved with regard to their analysis. One such challenge relates to the difference in dimensionality between the particles and the grid points to which they are mapped. There exists a non-trivial null space of the linear operator that maps particles values onto nodal values. In other words, there are non-zero particle values values that when mapped to the nodes are zero there. Given positive mapping weights such null space values are oscillatory in nature. The null space may be viewed as a more general form of the ringing instability identified by Brackbill for PIC methods. It will be shown that it is possible to remove these null-space values from the solution and so to improve the accuracy of PIC methods, using a matrix SVD approach. The expense of doing this is prohibitive for real problems and so a local method is developed for doing this.

A. V. P. Grosset, M. Prasad, C. Christensen, A. Knoll, C. Hansen.
**“TOD-Tree: Task-Overlapped Direct send Tree Image Compositing for Hybrid MPI Parallelism,”** In *Eurographics Symposium on Parallel Graphics and Visualization (2015)*, Edited by C. Dachsbacher, P. Navrátil, 2015.

Modern supercomputers have very powerful multi-core CPUs. The programming model on these supercomputer is switching from pure MPI to MPI for inter-node communication, and shared memory and threads for intra-node communication. Consequently the bottleneck in most systems is no longer computation but communication between nodes. In this paper, we present a new compositing algorithm for hybrid MPI parallelism that focuses on communication avoidance and overlapping communication with computation at the expense of evenly balancing the workload. The algorithm has three stages: a direct send stage where nodes are arranged in groups and exchange regions of an image, followed by a tree compositing stage and a gather stage. We compare our algorithm with radix-k and binary-swap from the IceT library in a hybrid OpenMP/MPI setting, show strong scaling results and explain how we generally achieve better performance than these two algorithms.

A. Gunduz, H. Morita, P. J. Rossi, W. L. Allen, R. L. Alterman, H. Bronte-Stewart, C. R. Butson, D. Charles, S. Deckers, C. de Hemptinne, M. DeLong, D. Dougherty, J. Ellrich, K. D. Foote, J. Giordano, W. Goodman, B. D. Greenberg, D. Greene, R. Gross, J. W. Judy, E. Karst, A. Kent, B. Kopell, A. Lang, A. Lozano, C. Lungu, K. E. Lyons, A. Machado, H. Martens, C. McIntyre, H. Min, J. Neimat, J. Ostrem, S. Pannu, F. Ponce, N. Pouratian, D. Reymers, L. Schrock, S. Sheth, L. Shih, S. Stanslaski, G. K. Steinke, P. Stypulkowski, A. I. Tröster, L. Verhagen, H. Walker, M. S. Okun.
**“Proceedings of the Second Annual Deep Brain Stimulation Think Tank: What's in the Pipeline,”** In *International Journal of Neuroscience*, Vol. 125, No. 7, Taylor & Francis, pp. 475-485. 2015.

DOI: 10.3109/00207454.2014.999268

PubMed ID: 25526555

The proceedings of the 2nd Annual Deep Brain Stimulation Think Tank summarize the most contemporary clinical, electrophysiological, and computational work on DBS for the treatment of neurological and neuropsychiatric disease and represent the insights of a unique multidisciplinary ensemble of expert neurologists, neurosurgeons, neuropsychologists, psychiatrists, scientists, engineers and members of industry. Presentations and discussions covered a broad range of topics, including advocacy for DBS, improving clinical outcomes, innovations in computational models of DBS, understanding of the neurophysiology of Parkinson's disease (PD) and Tourette syndrome (TS) and evolving sensor and device technologies.

A. Gyulassy, A. Knoll, K. C. Lau, Bei Wang, P. T. Bremer, M. E. Papka, L. A. Curtiss, V. Pascucci.
**“Morse-Smale Analysis of Ion Diffusion for DFT Battery Materials Simulations,”** In *Topology-Based Methods in Visualization (TopoInVis)*, 2015.

*Ab initio* molecular dynamics (AIMD) simulations are increasingly useful in modeling, optimizing and synthesizing materials in energy sciences. In solving Schrodinger's equation, they generate the electronic structure of the simulated atoms as a scalar field. However, methods for analyzing these volume data are not yet common in molecular visualization. The Morse-Smale complex is a proven, versatile tool for topological analysis of scalar fields. In this paper, we apply the discrete Morse-Smale complex to analysis of first-principles battery materials simulations. We consider a carbon nanosphere structure used in battery materials research, and employ Morse-Smale decomposition to determine the possible lithium ion diffusion paths within that structure. Our approach is novel in that it uses the wavefunction itself as opposed distance fields, and that we analyze the 1-skeleton of the Morse-Smale complex to reconstruct our diffusion paths. Furthermore, it is the first application where specific motifs in the graph structure of the complete 1-skeleton define features, namely carbon rings with specific valence. We compare our analysis of DFT data with that of a distance field approximation, and discuss implications on larger classical molecular dynamics simulations.

A. Gyulassy, A. Knoll, K. C. Lau, Bei Wang, PT. Bremer, M.l E. Papka, L. A. Curtiss, V. Pascucci.
**“Interstitial and Interlayer Ion Diffusion Geometry Extraction in Graphitic Nanosphere Battery Materials,”** In *Proceedings IEEE Visualization Conference*, 2015.

Large-scale molecular dynamics (MD) simulations are commonly used for simulating the synthesis and ion diffusion of battery materials. A good battery anode material is determined by its capacity to store ion or other diffusers. However, modeling of ion diffusion dynamics and transport properties at large length and long time scales would be impossible with current MD codes. To analyze the fundamental properties of these materials, therefore, we turn to geometric and topological analysis of their structure. In this paper, we apply a novel technique inspired by discrete Morse theory to the Delaunay triangulation of the simulated geometry of a thermally annealed carbon nanosphere. We utilize our computed structures to drive further geometric analysis to extract the interstitial diffusion structure as a single mesh. Our results provide a new approach to analyze the geometry of the simulated carbon nanosphere, and new insights into the role of carbon defect size and distribution in determining the charge capacity and charge dynamics of these carbon based battery materials.

J. K. Holmen, A. Humphrey, M. Berzins.
**“Exploring Use of the Reserved Core,”** In *High Performance Parallelism Pearls*, Edited by J. Reinders and J. Jeffers, Elsevier, pp. 229-242. 2015.

DOI: 10.1016/b978-0-12-803819-2.00010-0

In this chapter, we illustrate benefits of thinking in terms of thread management techniques when using a centralized scheduler model along with interoperability of MPI and PThreads. This is facilitated through an exploration of thread placement strategies for an algorithm modeling radiative heat transfer with special attention to the 61^{st} core. This algorithm plays a key role within the Uintah Computational Framework (UCF) and current efforts taking place at the University of Utah to model next-generation, large-scale clean coal boilers. In such simulations, this algorithm models the dominant form of heat transfer and consumes a large portion of compute time. Exemplified by a real-world example, this chapter presents our early efforts in porting a key portion of a scalability-centric codebase to the Intel ® Xeon Phi^{TM} coprocessor. Specifically, this chapter presents results from our experiments profiling the native execution of a reverse Monte-Carlo ray tracing-based radiation model on a single coprocessor. These results demonstrate that our fastest run confiurations utilized the 61st core and that performance was not profoundly impacted when explicitly over-subscribing the coprocessor operating system thread. Additionally, this chapter presents a portion of radiation model source code, a MIC-centric UCF cross-compilation example, and less conventional thread management techniques for developers utilizing the PThreads threading model.

A. Humphrey, T. Harman, M. Berzins, P. Smith.
**“A Scalable Algorithm for Radiative Heat Transfer Using Reverse Monte Carlo Ray Tracing,”** In *High Performance Computing*, Lecture Notes in Computer Science, Vol. 9137, Edited by Kunkel, Julian M. and Ludwig, Thomas, Springer International Publishing, pp. 212-230. 2015.

ISBN: 978-3-319-20118-4

DOI: 10.1007/978-3-319-20119-1_16

Radiative heat transfer is an important mechanism in a class of challenging engineering and research problems. A direct all-to-all treatment of these problems is prohibitively expensive on large core counts due to pervasive all-to-all MPI communication. The massive heat transfer problem arising from the next generation of clean coal boilers being modeled by the Uintah framework has radiation as a dominant heat transfer mode. Reverse Monte Carlo ray tracing (RMCRT) can be used to solve for the radiative-flux divergence while accounting for the effects of participating media. The ray tracing approach used here replicates the geometry of the boiler on a multi-core node and then uses an all-to-all communication phase to distribute the results globally. The cost of this all-to-all is reduced by using an adaptive mesh approach in which a fine mesh is only used locally, and a coarse mesh is used elsewhere. A model for communication and computation complexity is used to predict performance of this new method. We show this model is consistent with observed results and demonstrate excellent strong scaling to 262K cores on the DOE Titan system on problem sizes that were previously computationally intractable.

**Keywords:** Uintah; Radiation modeling; Parallel; Scalability; Adaptive mesh refinement; Simulation science; Titan

CIBC.
Note: *ImageVis3D: An interactive visualization software system for large-scale volume data. Scientific Computing and Imaging Institute (SCI), Download from: http://www.imagevis3d.org*, 2015.

C.R. Johnson, K. Potter.
**“Visualization,”** In *The Princeton Companion to Applied Mathematics*, Edited by Nicholas J. Higham, Princeton University Press, pp. 843-846. September, 2015.

ISBN: 9780691150390

H. De Sterck, C.R. Johnson.
**“Data Science: What Is It and How Is It Taught?,”** In *SIAM News*, SIAM, July, 2015.

C.R. Johnson.
**“Computational Methods and Software for Bioelectric Field Problems,”** In *Biomedical Engineering Handbook*, 4, Vol. 1, Ch. 43, Edited by J.D. Bronzino and D.R. Peterson, CRC Press, pp. 1--28. 2015.

Computer modeling and simulation continue to become more important in the field of bioengineering. The reasons for this growing importance are manyfold. First, mathematical modeling has been shown to be a substantial tool for the investigation of complex biophysical phenomena. Second, since the level of complexity one can model parallels the existing hardware configurations, advances in computer architecture have made it feasible to apply the computational paradigm to complex biophysical systems. Hence, while biological complexity continues to outstrip the capabilities of even the largest computational systems, the computational methodology has taken hold in bioengineering and has been used successfully to suggest physiologically and clinically important scenarios and results.

This chapter provides an overview of numerical techniques that can be applied to a class of bioelectric field problems. Bioelectric field problems are found in a wide variety of biomedical applications, which range from single cells, to organs, up to models that incorporate partial to full human structures. We describe some general modeling techniques that will be applicable, in part, to all the aforementioned applications. We focus our study on a class of bioelectric volume conductor problems that arise in electrocardiography (ECG) and electroencephalography (EEG).

We begin by stating the mathematical formulation for a bioelectric volume conductor, continue by describing the model construction process, and follow with sections on numerical solutions and computational considerations. We continue with a section on error analysis coupled with a brief introduction to adaptive methods. We conclude with a section on software.

C.R. Johnson.
**“Visualization,”** In *Encyclopedia of Applied and Computational Mathematics*, Edited by Björn Engquist, Springer, pp. 1537-1546. 2015.

ISBN: 978-3-540-70528-4

DOI: 10.1007/978-3-540-70529-1_368

C. Jones, T. Liu, N.W. Cohan, M. Ellisman, T. Tasdizen.
**“Efficient semi-automatic 3D segmentation for neuron tracing in electron microscopy images,”** In *Journal of Neuroscience Methods*, Vol. 246, Elsevier BV, pp. 13--21. May, 2015.

DOI: 10.1016/j.jneumeth.2015.03.005

**Background**

In the area of connectomics, there is a significant gap between the time required for data acquisition and dense reconstruction of the neural processes contained in the same dataset. Automatic methods are able to eliminate this timing gap, but the state-of-the-art accuracy so far is insufficient for use without user corrections. If completed naively, this process of correction can be tedious and time consuming.**New method**

We present a new semi-automatic method that can be used to perform 3D segmentation of neurites in EM image stacks. It utilizes an automatic method that creates a hierarchical structure for recommended merges of superpixels. The user is then guided through each predicted region to quickly identify errors and establish correct links.**Results**

We tested our method on three datasets with both novice and expert users. Accuracy and timing were compared with published automatic, semi-automatic, and manual results.**Comparison with existing methods**

Post-automatic correction methods have also been used in Mishchenko et al. (2010) and Haehn et al. (2014). These methods do not provide navigation or suggestions in the manner we present. Other semi-automatic methods require user input prior to the automatic segmentation such as Jeong et al. (2009) and Cardona et al. (2010) and are inherently different than our method.**Conclusion**

Using this method on the three datasets, novice users achieved accuracy exceeding state-of-the-art automatic results, and expert users achieved accuracy on par with full manual labeling but with a 70% time improvement when compared with other examples in publication.

M. Kim, C.D. Hansen.
**“Surface Flow Visualization using the Closest Point Embedding,”** In *2015 IEEE Pacific Visualization Symposium*, April, 2015.

In this paper, we introduce a novel flow visualization technique for arbitrary surfaces. This new technique utilizes the closest point embedding to represent the surface, which allows for accurate particle advection on the surface as well as supports the unsteady flow line integral convolution (UFLIC) technique on the surface. This global approach is faster than previous parameterization techniques and prevents the visual artifacts associated with image-based approaches.

**Keywords:** vector field, flow visualization

M. Kim, C.D. Hansen.
**“GPU Surface Extraction with the Closest Point Embedding,”** In *Proceedings of IS&T/SPIE Visualization and Data Analysis, 2015*, February, 2015.

Isosurface extraction is a fundamental technique used for both surface reconstruction and mesh generation. One method to extract well-formed isosurfaces is a particle system; unfortunately, particle systems can be slow. In this paper, we introduce an enhanced parallel particle system that uses the closest point embedding as the surface representation to speedup the particle system for isosurface extraction. The closest point embedding is used in the Closest Point Method (CPM), a technique that uses a standard three dimensional numerical PDE solver on two dimensional embedded surfaces. To fully take advantage of the closest point embedding, it is coupled with a Barnes-Hut tree code on the GPU. This new technique produces well-formed, conformal unstructured triangular and tetrahedral meshes from labeled multi-material volume datasets. Further, this new parallel implementation of the particle system is faster than any known methods for conformal multi-material mesh extraction. The resulting speed-ups gained in this implementation can reduce the time from labeled data to mesh from hours to minutes and benefits users, such as bioengineers, who employ triangular and tetrahedral meshes.

**Keywords:** scalar field methods, GPGPU, curvature based, scientific visualization

R.M. Kirby, M. Berzins, J.S. Hesthaven (Editors).
**“Spectral and High Order Methods for Partial Differential Equations,”** Subtitled **“Selected Papers from the ICOSAHOM'14 Conference, June 23-27, 2014, Salt Lake City, UT, USA.,”** In *Lecture Notes in Computational Science and Engineering*, Springer, 2015.

O. A. von Lilienfeld, R. Ramakrishanan, M., A. Knoll.
**“Fourier Series of Atomic Radial Distribution Functions: A Molecular Fingerprint for Machine Learning Models of Quantum Chemical Properties,”** In *International Journal of Quantum Chemistry*, Wiley Online Library, 2015.

We introduce a fingerprint representation of molecules based on a Fourier series of atomic radial distribution functions. This fingerprint is unique (except for chirality), continuous, and differentiable with respect to atomic coordinates and nuclear charges. It is invariant with respect to translation, rotation, and nuclear permutation, and requires no pre-conceived knowledge about chemical bonding, topology, or electronic orbitals. As such it meets many important criteria for a good molecular representation, suggesting its usefulness for machine learning models of molecular properties trained across chemical compound space. To assess the performance of this new descriptor we have trained machine learning models of molecular enthalpies of atomization for training sets with up to 10 k organic molecules, drawn at random from a published set of 134 k organic molecules with an average atomization enthalpy of over 1770 kcal/mol. We validate the descriptor on all remaining molecules of the 134 k set. For a training set of 10k molecules the fingerprint descriptor achieves a mean absolute error of 8.0 kcal/mol, respectively. This is slightly worse than the performance attained using the Coulomb matrix, another popular alternative, reaching 6.2 kcal/mol for the same training and test sets.

S. Liu, D. Maljovec, Bei Wang, P. T. Bremer, V. Pascucci.
**“Visualizing High-Dimensional Data: Advances in the Past Decade,”** In *State of The Art Report*, *Eurographics Conference on Visualization (EuroVis)*, 2015.

Massive simulations and arrays of sensing devices, in combination with increasing computing resources, have generated large, complex, high-dimensional datasets used to study phenomena across numerous fields of study. Visualization plays an important role in exploring such datasets. We provide a comprehensive survey of advances in high-dimensional data visualization over the past 15 years. We aim at providing actionable guidance for data practitioners to navigate through a modular view of the recent advances, allowing the creation of new visualizations along the enriched information visualization pipeline and identifying future opportunities for visualization research.

S. Liu, Bei Wang, J. J. Thiagarajan, P. T. Bremer, V. Pascucci.
**“Visual Exploration of High-Dimensional Data through Subspace Analysis and Dynamic Projections,”** In *Computer Graphics Forum*, Vol. 34, No. 3, Wiley-Blackwell, pp. 271--280. June, 2015.

DOI: 10.1111/cgf.12639

We introduce a novel interactive framework for visualizing and exploring high-dimensional datasets based on subspace analysis and dynamic projections. We assume the high-dimensional dataset can be represented by a mixture of low-dimensional linear subspaces with mixed dimensions, and provide a method to reliably estimate the intrinsic dimension and linear basis of each subspace extracted from the subspace clustering. Subsequently, we use these bases to define unique 2D linear projections as viewpoints from which to visualize the data. To understand the relationships among the different projections and to discover hidden patterns, we connect these projections through dynamic projections that create smooth animated transitions between pairs of projections. We introduce the view transition graph, which provides flexible navigation among these projections to facilitate an intuitive exploration. Finally, we provide detailed comparisons with related systems, and use real-world examples to demonstrate the novelty and usability of our proposed framework.