intel_ml / ipcc-poster

\documentclass[portrait,a0,bindingoffset=0in]{a0poster}
%\documentclass[landscape]{a0poster}

\usepackage{hyperref}

\hypersetup{
  pdfauthor = {Guilherme Amadio},
  colorlinks, urlcolor=black,
  pdfkeywords = {particle physics, detector simulation},
  pdfsubject = {HPC}, pdfpagemode = UseNone,
  pdftitle = {GeantV - High-Performance Detector Simulation Software}
}

\usepackage[nohead, nomarginpar, margin=1in]{geometry}

\usepackage{setspace}
\setlength{\parindent}{0pt}
\setlength{\parskip}{0.7em}
\setlength{\columnsep}{1in}
\setlength{\marginparwidth}{0in}
\setlength{\evensidemargin}{0in}
\setlength{\oddsidemargin}{0in}
\setlength{\hoffset}{0in}
\setlength{\voffset}{0in}

\usepackage{color}
\usepackage[no-math]{fontspec}
\usepackage{mathspec}
\usepackage{microtype}
\usepackage{multicol}
\usepackage{cutwin}
\usepackage{minted}
\usepackage{sansmath}

\usepackage{tikz}
\usepackage{pgfplots}
\pgfplotsset{compat=newest}

\pgfplotsset{
  tick label style = {font=\sansmath\sffamily},
  every axis label = {font=\sansmath\sffamily},
  legend style = {font=\sansmath\sffamily},
  label style = {font=\sansmath\sffamily}
}  


\defaultfontfeatures{Mapping=tex-text}
\setmainfont{Myriad Pro}
\setsansfont{Myriad Pro}
\setmonofont[Scale=0.9]{Source Code Pro}
\setmathsfont[range=\mathup]{Myriad Pro}
\setmathsfont[range=\mathit]{Myriad Pro}

\definecolor{structure}{rgb}{0.122,0.455,0.722}

% section headings: blue and bold.
\makeatletter
\renewcommand\section{\@startsection {section}{1}{\z@}%
                                   {-1.5ex \@plus -1ex \@minus -.2ex}%
                                   {1ex \@plus.2ex}%
                                   {\Large\bfseries\color{structure}}}
\renewcommand\subsection{\@startsection {subsection}{2}{\z@}%
                                   {-1.5ex \@plus -1ex \@minus -.2ex}%
                                   {0.5ex \@plus.2ex}%
                                   {\large\bfseries}}
\makeatother

\newminted{cpp}{}
\newminted{asm}{}
\newminted{cmake}{}
\newmintedfile{cpp}{}
\newmintedfile{asm}{}
\newmintedfile{cmake}{}

\newcommand{\cpp}[1]{\mintinline[fontsize=\tiny]{cpp}{#1}}
\newcommand{\asm}[1]{\mintinline[fontsize=\small]{asm}{#1}}
\newcommand{\cmd}[1]{\mintinline{bash}{#1}}

\begin{document}
\large
\begin{minipage}{0.065\textwidth} \centering
  \includegraphics[width=\textwidth]{figures/geantv-logo.jpg}
\end{minipage}
%
\hspace{10mm}
\begin{minipage}{0.75\textwidth} \centering
{ \fontsize{56}{56} \fontspec{Myriad Pro}
  \textbf{GeantV -- Path Towards High-Performance Detector Simulation}
  \vspace{-0.3em} { \color{structure} \vrule height 3pt width \hsize \hfill }
}

% authors
{ \normalsize \input{authors} }
\end{minipage}
%
\hspace{10mm}
\begin{minipage}{0.15\textwidth} \centering
	\raisebox{23mm}{\includegraphics[width=\textwidth]{figures/LogoUNESP.png}}
\end{minipage}

%
%\addtolength{\hsize}{1.5in}
%\setlength{\textwidth}{\hsize}
%\setlength{\linewidth}{\hsize}
%
\normalsize
\vspace{20mm}
\begin{multicols*}{3}
\raggedcolumns
\section*{Introduction}

Geant4 is a toolkit for the simulation of the passage of particles through
matter. It is used at the LHC to simulate particle collisions, and played a key
role in the discovery of the Higgs boson, announced in 2012. Geant4 is,
therefore, an essential tool in high energy physics research. Moreover,
Geant4 is also used in space research, to study the effects of space
radiation in electronic devices, and in medical physics applications, to
simulate radiation doses in human tissue for cancer treatment, for example.

\begin{center}
\vspace{5mm}
\includegraphics[width=0.75\columnwidth]{figures/cms-event}\\
\vspace{2mm}
{\color{structure} CMS Event Display (Image: CERN)}
\vspace{5mm}
\end{center}

Experiments at CERN have an ever increasing demand for simulated
collisions. Further increases in beam luminosity planned for the next
runs will push this demand far beyond the limits of what is available at
the Worldwide LHC Computing Grid, which already spends more than half of
its CPU budget ($\approx$300,000 cores) on detector simulation. Funding for
hardware, nevertheless, is expected to remain constant, so in
order to bridge the widening gap between simulation needs and available
computing resources, it is necessary to innovate on the software side by
redesigning the simulation framework from the ground up to fully exploit
SIMD and multithreading parallelism. This is the aim of the GeantV
project, which has recently produced a toolkit prototype capable of
running large scale simulations in complex geometries.

\section*{The GeantV Simulation Framework}

The main idea in GeantV is exploiting SIMD operations. For that, particles
have to be collected in contiguous and aligned arrays (baskets) via filters. The
filters make sure that the grouped particles can use the same algorithm in a
vectorized form to maximize SIMD effectiveness. Unfortunately, this is usually
true for only a few \emph{transport steps}, after which particles move on to
different fates and have to be re-filtered to be processed again in SIMD mode.
In the general architecture of GeantV shown below, different worker threads
handle the transport of different baskets while separate threads orchestrate the
distribution of the baskets to workers. 

\begin{center}
\vspace{2mm}
\includegraphics[width=0.98\columnwidth]{figures/geantv-framework}
\vspace{2mm}
{\color{structure} GeantV Event Processing Scheme}
\end{center}

\begin{center}
\vspace{2mm}
\includegraphics[width=0.98\columnwidth]{figures/geantv-brokers}
\vspace{2mm}
{\color{structure} GeantV Coprocessor Broker and Multi-Node Architecture}
\end{center}

Throughput gain is expected to come from processing several tracks at once in
SIMD mode, while simultaneously improving cache coherency. To maximize vector
content for different locality criteria, multiple collision events are processed
in parallel.

\columnbreak
\section*{Portable SIMD Abstraction Layer}

Portable and efficient vectorization is one of the main challenges in a large
and diverse code base as GeantV. The solution adopted by the project was to
develop generic computational kernels using abstract types from the VecCore
library, which follow the architecture-agnostic interface for vectorization
shown below.

\begin{center}
\vspace{2mm}
\inputminted[bgcolor=structure!10!white, fontsize=\scriptsize]{cpp}{code/veccore-api.cc}\\
\vspace{7mm}
\includegraphics[width=\columnwidth]{figures/veccore-api3}\\
\vspace{3mm}
\end{center}

The API above is used to implement single-track, multi-track (SIMD), and GPU kernels.
VecCore has several backends based on scalars and on the libraries
Vc \mbox{(\url{https://github.com/VcDevel/Vc})} and
UME::SIMD (\url{https://github.com/edanor/umesimd})
for efficient vectorization on all platforms.

\section*{Geometry Algorithms}

One of the most compute-intensive components of simulations of large
LHC-scale detectors is navigating their geometry structure. The VecGeom
geometry library used in GeantV has introduced a new, vectorized
multi-particle API to perform ray casting, distance calculations and
navigation in sets of particles within the detector geometry.

\begin{center}
\vspace{2mm}
\includegraphics[width=\columnwidth]{figures/veccore}\\
\vspace{2mm}
\end{center}

The figure below shows performance gains for several volume
types compared to previous implementations and the scalar API.

\begin{center}
\vspace{3mm}
{\color{structure} Vectorization Speedup of Selected Shapes on Intel® Xeon Phi™ (KNL)}\\
\vspace{3mm}
\hspace{0.19\columnwidth}
\includegraphics[width=0.8\columnwidth]{figures/Benchmarks/legend} \\
\vspace{3mm}
\begin{tabular}{cc}
\includegraphics[height=0.19\columnwidth]{figures/povray/box} &
\hspace{7mm}
\includegraphics[height=0.19\columnwidth]{figures/Benchmarks/Box} \\
{\color{structure} Box} & \\
\includegraphics[height=0.19\columnwidth]{figures/povray/cone} &
\hspace{7mm}
\includegraphics[height=0.19\columnwidth]{figures/Benchmarks/Cone} \\
{\color{structure} Cone} & \\
\includegraphics[height=0.19\columnwidth]{figures/povray/sphere} &
\hspace{7mm}
\includegraphics[height=0.19\columnwidth]{figures/Benchmarks/Orb} \\
{\color{structure} Sphere} & \\
\includegraphics[height=0.19\columnwidth]{figures/povray/trapezoid} &
\hspace{7mm}
\includegraphics[height=0.19\columnwidth]{figures/Benchmarks/Trd} \\
{\color{structure} Trapezoid} & \\
\includegraphics[height=0.19\columnwidth]{figures/povray/tube} &
\hspace{7mm}
\includegraphics[height=0.19\columnwidth]{figures/Benchmarks/Tube} \\
{\color{structure} Tube} & \\
\end{tabular}
\vspace{5mm}
\end{center}

\columnbreak

\setlength{\columnsep}{0in}
\section*{Electromagnetic Physics Models}

Energetic photons and electrons produce \emph{particle showers} in
parts of the detectors, which means that they are created most frequently
compared to other particle types during simulation. Therefore,
electromagnetic physics processes can take a large portion of computing
time and are the natural first targets for vectorization. However,
vectorization of physics models requires algorithm modifications, such
as replacing composition and rejection methods by something more easily
vectorizable, e.g., the alias sampling method. Vectorization gains for
the most common electromagnetic processes using this method are shown
below.

\begin{center}
%\vspace{-10mm}
{\color{structure} Electromagnetic Physics Processes}\\
\vspace{3mm}
\includegraphics[width=0.9\columnwidth]{figures/physics/em}\\
\end{center}

\begin{center}
{\color{structure} Vectorization Speedup of Physics Models on Intel® Xeon Phi™ (KNC)}\\
\vspace{2mm}
\begin{tikzpicture}
\pgfplotsset{ every non boxed x axis/.append style={x axis line style=-},
     every non boxed y axis/.append style={y axis line style=-}}
  \begin{semilogxaxis}[width=0.8\columnwidth, height=0.4\columnwidth,
                       axis x line=bottom, axis y line=left,
                       log ticks with fixed point, xtick=\empty, ymajorgrids,
                       extra x ticks={8, 16, 32, 64, 128, 256, 512, 1024, 2048},
                       extra x tick labels={8, 16, 32, 64, 128, 256, 512, 1024, 2048},
                       xlabel={Number of tracks}, ylabel={Vectorization speedup},
                       enlarge x limits=0.1, ymin=0, ymax=8, enlarge y limits=0,
                       ytick=\empty, extra y ticks={2,4,6,8}, extra y tick labels={2,4,6,8},
                       legend style={at={(0.98,0.02)}, font=\small,anchor=south east, draw=none}]
    \addplot[blue, ultra thick, mark=none] table {data/vecphys/bremsstrahlung.dat};
    \addplot[red, ultra thick, mark=none] table {data/vecphys/compton-scattering.dat};
    \addplot[orange, ultra thick, mark=none] table {data/vecphys/ionization.dat};
    \addplot[green!80!black, ultra thick, mark=none] table {data/vecphys/pair-production.dat};
    \addplot[yellow!60!black, ultra thick, mark=none] table {data/vecphys/photo-electric.dat};
    \legend{Bremsstrahlung,Compton Scattering,Ionization,Pair Production/Annihilation,Photo-Electric Effect};
  \end{semilogxaxis}
\end{tikzpicture}
\vspace{-1cm}
\end{center}

\section*{Geometry Navigation}

In addition to vectorized shape algorithms, VecGeom also introduces
vectorized navigation algorithms that allow for faster traversal of the
tree of volumes that form the detector geometry. This provides a
significant speedup in navigation by testing particles against multiple
bounding boxes at a time. The speedup is exemplified in the benchmark
below, where a detector geometry is used to create an \emph{X-Ray} image
by tracing particles through it without any physics interactions.

\begin{center}
\includegraphics[height=0.2\columnwidth]{figures/navigation-volume}
\hspace{1cm}
\includegraphics[height=0.2\columnwidth]{figures/navigation-xsec}\\
\vspace{2mm}
\end{center}

\begin{center}
  \begin{tikzpicture}[font=\sffamily]
		\begin{axis}[xbar, width=0.8\columnwidth, height=0.22\columnwidth, xmin=0, 
                 xlabel={Running Time}, axis x line*=bottom, axis y line*=left, xtick=\empty, 
								 ytick=data, enlarge y limits=0.4, bar width=1.05em,
								 symbolic y coords={VecGeom,Geant4,ROOT}]
                 
			\addplot[fill=structure!60!white] coordinates{(3.2,ROOT) (2.6,Geant4) (1.0,VecGeom)};
      \node at (axis cs:1.15,VecGeom) {1.0};
      \node at (axis cs:2.75,Geant4) {2.6};
      \node at (axis cs:3.35,ROOT) {3.2};
		\end{axis}
  \end{tikzpicture}
\vspace{-1cm}
\end{center}

\section*{Full Simulation Benchmarks}

More complex validation tests and benchmarks using the full geometry of
CMS and LHCb detectors and simplified physics models (tabulated physics)
have been used to estimate the speedup that GeantV could achieve
relative to Geant4 10.2. A single-threaded benchmark, when run on an
Intel® Xeon™ E5-2630 v3 (8cores, 2.4GHz) machine, yields a speedup of 3.5 
for the CMS geometry, and 2.5 for the LHCb geometry. Scalability of
GeantV was also measured on Intel® Xeon™ and Xeon Phi™ processors.

\begin{center}
\vspace{-5mm}
{\color{structure} GeantV Preliminary Full Simulation Speedups for CMS and LHCb}\\
\vspace{2mm}
\includegraphics[width=0.8\columnwidth]{figures/cms-lhcb}\\
{\large $\mathsf{T_{Geant4}/T_{GeantV} =}$ 3.5 \hspace{3cm} $\mathsf{T_{Geant4}/T_{GeantV} =}$ 2.5}\\
\vspace{5mm}
\end{center}

\begin{center}
{\color{structure} GeantV Full Transport Scalability on Intel® Xeon™ and Xeon Phi™}\\
\vspace{1mm}
\begin{tikzpicture}[font=\sffamily]
\pgfplotsset{ every non boxed x axis/.append style={x axis line style=-},
     every non boxed y axis/.append style={y axis line style=-}}
  \begin{loglogaxis}[width=0.8\columnwidth, height=0.35\columnwidth,
                       axis x line=bottom, axis y line=left, ymajorgrids,
                       log ticks with fixed point, xtick=\empty, ytick=\empty,
                       extra x ticks={1, 2, 4, 8, 16, 32, 64, 128, 256},
                       extra x tick labels={1, 2, 4, 8, 16, 32, 64, 128, 256},
                       extra y ticks={1, 2, 4, 8, 16, 32, 64, 128},
                       extra y tick labels={1, 2, 4, 8, 16, 32, 64, 128},
                       xlabel={Number of threads}, ylabel={Speedup vs single-thread},
                       enlarge x limits=0.03, log basis y={2},
                       ymin=1, ymax=128,
                       enlarge y limits=0, legend
                       style={at={(0.99,0.03)}, anchor=south east, draw=none}]
                       %style={at={(0.05,0.95)}, anchor=north west, draw=none}]
    \addplot[ red, ultra thick, mark=none] table[y index=2] {data/scalability.dat};
    \addplot[blue, ultra thick, mark=none] table[y index=1] {data/scalability.dat};
    \legend{Intel® Xeon™ E5-2630 v3,Intel® Xeon Phi™ 7210 (KNL)};
  \end{loglogaxis}
\end{tikzpicture}
\vspace{-15mm}
\end{center}

\section*{References}

\begin{enumerate}
  \item GeantV Website \url{https://geant.cern.ch}
  \item ``The GeantV project: preparing the future of simulation'',\\
        J. Phys.: Conf. Ser. 664 072006 (2015)\\\url{http://dx.doi.org/10.1088/1742-6596/664/7/072006}
  \item ``A concurrent vector-based steering framework for particle\\ transport'',
        J. Phys.: Conf. Ser. 523 012004 (2014)\\\url{http://dx.doi.org/10.1088/1742-6596/523/1/012004}
  %\item ``Adaptive track scheduling to optimize concurrency and vectorization in GeantV'',
  %      J. Phys.: Conf. Ser. 608 012003 (2015) \url{http://dx.doi.org/10.1088/1742-6596/608/1/012003}
\end{enumerate}
\end{multicols*}
\end{document}