如何进行HPCC测试
摘要
HPCC是一个测试集，包含RandomAcess，PTRANS，DGEMM，STREAM，FFT，Latency/Bandwidth，HPL。
本文简要介绍如何编译运行HPCC。
本文内容为荣裕大佬的测试记录，非原创内容
编译运行
HPCC编译主要依赖HPL内的Makefile，修改其中的makefile即可
cp ./hpl/setup/Make.xxxx ./hpl/一堆Makefile里随便CP一个
vi ./hpl/Make.Linux以下为使用ICC+MKL的Makefile
#
# -- High Performance Computing Linpack Benchmark (HPL)
# HPL - 2.0 - September 10, 2008
# Antoine P. Petitet
# University of Tennessee, Knoxville
# Innovative Computing Laboratory
# (C) Copyright 2000-2008 All Rights Reserved
#
# -- Copyright notice and Licensing terms:
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
#
# 3. All advertising materials mentioning features or use of this
# software must display the following acknowledgement:
# This product includes software developed at the University of
# Tennessee, Knoxville, Innovative Computing Laboratory.
#
# 4. The name of the University, the name of the Laboratory, or the
# names of its contributors may not be used to endorse or promote
# products derived from this software without specific written
# permission.
#
# -- Disclaimer:
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE UNIVERSITY
# OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
# ######################################################################
#
# ----------------------------------------------------------------------
# - shell --------------------------------------------------------------
# ----------------------------------------------------------------------
#
SHELL = /bin/sh
#
CD = cd
CP = cp
LN_S = ln -s
MKDIR = mkdir
RM = /bin/rm -f
TOUCH = touch
#
# ----------------------------------------------------------------------
# - Platform identifier ------------------------------------------------
# ----------------------------------------------------------------------
#
ARCH = $(arch)
#
# ----------------------------------------------------------------------
# - HPL Directory Structure / HPL library ------------------------------
# ----------------------------------------------------------------------
#
TOPdir = ../../..
INCdir = $(TOPdir)/include
BINdir = $(TOPdir)/bin/$(ARCH)
LIBdir = $(TOPdir)/lib/$(ARCH)
#
HPLlib = $(LIBdir)/libhpl.a
#
# ----------------------------------------------------------------------
# - Message Passing library (MPI) --------------------------------------
# ----------------------------------------------------------------------
# MPinc tells the C compiler where to find the Message Passing library
# header files, MPlib is defined to be the name of the library to be
# used. The variable MPdir is only used for defining MPinc and MPlib.
#
# Mlib 可以选择静态连接mpiicxx.a
MPdir = ${I_MPI_ROOT}
MPinc = -I$(MPdir)/intel64/include
Mlib =
#
# ----------------------------------------------------------------------
# - Linear Algebra library (BLAS or VSIPL) -----------------------------
# ----------------------------------------------------------------------
# LAinc tells the C compiler where to find the Linear Algebra library
# header files, LAlib is defined to be the name of the library to be
# used. The variable LAdir is only used for defining LAinc and LAlib.
# 关于MKL的fftw2编译之后提及
LAdir = ${MKLROOT}
LAinc = -I$(LAdir)/include -I$(LAdir)/include/fftw
LAlib = ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \
${MKLROOT}/lib/intel64/libmkl_intel_thread.a\
${MKLROOT}/lib/intel64/libmkl_core.a \
${MKLROOT}/lib/intel64/libmkl_cdft_core.a \
${MKLROOT}/lib/intel64/libfftw2xc_double_intel.a \
${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a \
${MKLROOT}/lib/intel64/libfftw2x_cdft_DOUBLE_ilp64.a \
-Wl,--end-group -liomp5 -lpthread -lm -ldl
#
# ----------------------------------------------------------------------
# - F77 / C interface --------------------------------------------------
# ----------------------------------------------------------------------
# You can skip this section if and only if you are not planning to use
# a BLAS library featuring a Fortran 77 interface. Otherwise, it is
# necessary to fill out the F2CDEFS variable with the appropriate
# options. **One and only one** option should be chosen in **each** of
# the 3 following categories:
#
# 1) name space (How C calls a Fortran 77 routine)
#
# -DAdd_ : all lower case and a suffixed underscore (Suns,
# Intel, ...), [default]
# -DNoChange : all lower case (IBM RS6000),
# -DUpCase : all upper case (Cray),
# -DAdd__ : the FORTRAN compiler in use is f2c.
#
# 2) C and Fortran 77 integer mapping
#
# -DF77_INTEGER=int : Fortran 77 INTEGER is a C int, [default]
# -DF77_INTEGER=long : Fortran 77 INTEGER is a C long,
# -DF77_INTEGER=short : Fortran 77 INTEGER is a C short.
#
# 3) Fortran 77 string handling
#
# -DStringSunStyle : The string address is passed at the string loca-
# tion on the stack, and the string length is then
# passed as an F77_INTEGER after all explicit
# stack arguments, [default]
# -DStringStructPtr : The address of a structure is passed by a
# Fortran 77 string, and the structure is of the
# form: struct {char *cp; F77_INTEGER len;},
# -DStringStructVal : A structure is passed by value for each Fortran
# 77 string, and the structure is of the form:
# struct {char *cp; F77_INTEGER len;},
# -DStringCrayStyle : Special option for Cray machines, which uses
# Cray fcd (fortran character descriptor) for
# interoperation.
#
F2CDEFS = -DAdd_ -DF77_INTEGER=long -DStringSunStyle
#
# ----------------------------------------------------------------------
# - HPL includes / libraries / specifics -------------------------------
# ----------------------------------------------------------------------
#
HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc)
HPL_LIBS = $(HPLlib) $(LAlib) $(MPlib) -lm
#
# - Compile time options -----------------------------------------------
#
# -DHPL_COPY_L force the copy of the panel L before bcast;
# -DHPL_CALL_CBLAS call the cblas interface;
# -DHPL_CALL_VSIPL call the vsip library;
# -DHPL_DETAILED_TIMING enable detailed timers;
#
# By default HPL will:
# *) not copy L before broadcast,
# *) call the BLAS Fortran 77 interface,
# *) not display detailed timing information.
#
HPL_OPTS = -DHPL_CALL_CBLAS -DUSING_FFTW -DRA_SANDIA_OPT2 -DLONG_IS_64BITS -DHPCC_MEMALLCTR -DMKL_INT=long -DHPCC_FFT_235
#
# ----------------------------------------------------------------------
#
HPL_DEFS = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)
#
# ----------------------------------------------------------------------
# - Compilers / linkers - Optimization flags ---------------------------
# ----------------------------------------------------------------------
#
CC = mpiicc
CCNOOPT = $(HPL_DEFS)
CCFLAGS = $(HPL_DEFS) -std=c99 -xHOST -O3 -ipo -no-prec-div -static-intel -fp-model fast=2 -ansi-alias -fno-alias
#
# On some platforms, it is necessary to use the Fortran linker to find
# the Fortran internals used in the BLAS library.
#
LINKER = mpiicc
LINKFLAGS = $(CCFLAGS) -z relro -z now -Wl,-R'$$ORIGIN/lib/intel64' -liomp5 -L$(MKL_LIB) -mkl
#
ARCHIVER = xiar
ARFLAGS = r
RANLIB = echo
#
# ----------------------------------------------------------------------之后编译MKL库中的fftw2
sudo su
module load nvme/intel/2018.4
cd $MKLROOT
cd interfaces/
cd fftw2xc
make libintel64 PRECISION=MKL_DOUBLE interface=ilp64 compiler=intel
cd ..
cd fftw2x_cdft
make libintel64 PRECISION=MKL_DOUBLE interface=ilp64 compiler=intel注意这里需要ilp64 主要问题是其接口为 int 不然有可能出现爆int, 但其他时候不需要ilp64,会使得HPL出现段错误
正式编译
make arch=Linux -j之后会生成_hpccinf.txt
cp _hpccinf.txt hpccinfo.txt运行
mpirun -n xxx ./hpcc
# 结果在 hpccout.txt中关于编译选项
-DHPCC_FFT_235` ：如没有此定义 运行`MPIFFT`时 N(进程数)会自动变为2的幂，当使用此定义时`N=2^x * 3^y * 5^z-DHPCC_FFTW_ESTIMATE：影响FFTW库的调用方式，会降低性能，但会加速FFT时间。
-DHPCC_MEMALLCTR： 可以很大的N
-DHPL_USE_GETPROCESSTIMES ： 将使用特定于Windows的GetProcessTimes（）函数来测量经过的CPU时间。
-DUSE_MULTIPLE_RECV： 同时多个非阻塞接收，默认只有一个非阻塞接收。
-DRA_SANDIA_NOPT：如果定义了此符号，则不会使用Global RandomAccess的HPC Challenge标准算法。 相反，将使用桑迪亚国家实验室的替代实施方案。 它通过MPI进程形成的虚拟超立方体拓扑在软件中路由消息。
-DRA_SANDIA_OPT2 ： 如果定义了此符号，则不会使用Global RandomAccess的HPC Challenge标准算法。 相反，将使用桑迪亚国家实验室的替代实施方案。 该实现针对作为2的幂的处理器的数量进行了优化。 优化是在发送和展开数据更新循环之前对数据进行排序。 如果进程数不是幂2，则代码与使用RA_SANDIA_NOPT设置执行的代码相同。
-DRA_TIME_BOUND_DISABLE ： 如果定义了此符号，则将使用标准的Global RandomAccess代码而不受时间限制。 对于大多数运行，这是不鼓励的，因为对于大型阵列大小，标准算法往往很慢，因为短MPI消息的开销很大。
-DUSING_FFTW ： 使用FFTW库
BUG
修改注释掉一处 fftw_destroy_plan(p);
位置：./FFT/tstfft.c
原因：N开大后段错误
影响：没有销毁释放内存，无其他影响
修改代码只运行部分测试
修改 src/hpcc.c
注释掉不要测试的注释即可
结果解读
前缀意义
MPI 全集群
Star 所有进程自己跑自己的
Single 所有进程按顺序跑
一般来说Single的性能会好于Star，因为多核一起跑往往会导致CPU降频还有共享内存带宽的问题
1. RandomAccess
RandomAccess measures the rate of random updates of memory.
1.1. MPIRandomAccess
1.2. StarRandomAccess
1.3. SingleRandomAccess
1.4. MPIRandomAccess LCG
LCG = Linear Congruential Generator 线性同余生成器
我的理解就是生成的数可能更随机了叭
1.5. StarRandomAccess LCG
1.6. SingleRandomAccess LCG
2. PTRANS
PTRANS measures the rate of transfer for large arrays of data from multiprocessor’s memory.
全集群运行，使用HPL中的参数变化而成
M=N=HPL(N/2)
NB=MB=HPL(NB)
P=HPL(Q)
Q=HPL(Q)
3. DGEMM
DGEMM measures the floating point execution rate for double precision real matrix-matrix multiplication.
3.1. StarDGEMM
3.2. SingleDGEMM
4.STREAM
STREAM is a benchmark that measures sustainable memory bandwidth (in GB/s).
4.1 StarSTREAM
4.2 SingleSTREAM
5. FFT
FFT measures the floating point rate of execution of double precision complex one-dimensional Discrete Fourier Tranform (DFT).
5.1. MPIFFT
在这个测试中，如果结果不稳定，可能是因为N开得不够大
5.2. StarFFT
5.3. SingleFFT
6. Latency/Bandwidth
7. HPL
不建议跑这个测试