Directory Core/Util/Timers/

Lines of Code

Developers

Most Recent Commits

ahumphrey 2017-08-02 17:23 Rev.: 56838

* Clean up several race conditions, true positives reported by TSAN.
* Catch exceptions by ref (should probably catch by const ref)
* Also clear out some compiler warnings, etc.

1 lines of code changed in 2 files:

• Core/Util/Timers: Timers.hpp (+1 -1)

ahumphrey 2017-04-21 12:06 Rev.: 56470

Fix and re-enable RuntimeStats class. Have co-opted SCI_DEBUG for MPIStats:+ to give detailed and accurate information about specific MPI calls within Uintah (see sample output below), e.g. COUNT, TIME, VOLUME. This also gives message volume by size class and a coarse histogram. This now makes uses of all the timers in UintahMPI.h (e.g. Impl::SendTimer, etc). We can now accurately track all MPI traffic very specifically (per MPI call) within Uintah. Timers::ThreadedTripTimer still needs some work.

Please note: stdout reporting is perfectly aligned for everything referenced in this commit log. The sample outputs provided here are slightly whacky, but serve to illustrate what one will see when exporting "SCI_DEBUG=MPIReport:+,ExecTimes:+,WaitTimes:+". This info is crucial for infrastructure debugging and quite useful to component developers.

Co-opted SCI_DEBUG for WaitTimes:+, moving away from the horeendously non-scalable, file-per-proc approach previously used. This reports (to stdout) per-task min, max, evg with a coarse histogram (see sample output below). Would like to deprecate the file-per-proc approach.

Co-opted SCI_DEBUG for ExecTimes:+, moving away from the horeendously non-scalable, file-per-proc approach previously used. This reports (to stdout) per-task min, max, evg with a coarse histogram (see sample output below). Would like to deprecate the file-per-proc approach.

Note that this is all thread-safe and maintains complete backward compatibility for those who use scripts to parse files generated with ExecTimes and WaitTimes env vars. Also note this does not impede or break Allen Sanderson's InfoMapper class in any way, and in fact works in unison with this.

Have also fixed the per-thread wait_time within Unified scheduler.

--------------------------------------------------------------------------------
MPIStats:+
--------------------------------------------------------------------------------
Name: Total: Avg: Min: Max: Max rank: Hist % [ Q1 | Q2 | Q3 | Q4 ]: %Load Imbalance
Count: Bcas: 4 1 1 1 3 100.0 0.0 0.0 0.0 0.0
Count: Recv: 96 24 24 24 3 100.0 0.0 0.0 0.0 0.0
Count: Redu: 12 3 3 3 3 100.0 0.0 0.0 0.0 0.0
Count: Send: 24 6 6 6 3 100.0 0.0 0.0 0.0 0.0
Count: Test: 886 222 81 468 1 50.0 25.0 0.0 25.0 52.7
Count: Wait: 7 1.75 0 3 1 25.0 0.0 50.0 25.0 41.7
Time: Bcas: 0.106 s 26.5 ms 4.11e+03 ns 44 ms 3 25.0 0.0 25.0 50.0 39.8
Time: Recv: 5.14 ms 1.29 ms 0.102 ms 4.19 ms 2 75.0 0.0 0.0 25.0 69.3
Time: Redu: 0.409 s 0.102 s 27.7 ms 0.307 s 0 75.0 0.0 0.0 25.0 66.7
Time: Send: 1.19 ms 0.298 ms 0.0916 ms 0.575 ms 2 50.0 0.0 25.0 25.0 48.1
Time: Test: 0.189 ms 0.0473 ms 8.22e+03 ns 0.154 ms 1 75.0 0.0 0.0 25.0 69.3
Time: Wait: 0.011 ms 2.74e+03 ns 0 ns 4.02e+03 ns 3 25.0 0.0 0.0 75.0 31.7
Volume Recv TOTAL : 1.75 MB 448 KB 446 KB 451 KB 0 75.0 0.0 0.0 25.0 0.8
Volume Recv0: <= 64B: 7 1.75 1 2 3 25.0 0.0 0.0 75.0 12.5
Volume Recv1: <= 4KB: 39 9.75 8 15 0 75.0 0.0 0.0 25.0 35.0
Volume Recv2: <= 2MB: 64 16 16 16 3 100.0 0.0 0.0 0.0 0.0
Volume Send TOTAL : 1.76 MB 451 KB 451 KB 451 KB 0 75.0 0.0 0.0 25.0 0.0
Volume Send0: <= 64B: 5 1.25 1 2 0 75.0 0.0 0.0 25.0 37.5
Volume Send1: <= 4KB: 60 15 15 15 3 100.0 0.0 0.0 0.0 0.0
Volume Send2: <= 2MB: 64 16 16 16 3 100.0 0.0 0.0 0.0 0.0


--------------------------------------------------------------------------------
WaitTimes:+
--------------------------------------------------------------------------------
Name: Total: Avg: Min: Max: Max rank: Hist % [ Q1 | Q2 | Q3 | Q4 ]: %Load Imbalance
Coarsen_Q_abskg: 0.0266 ms 6.64e+03 ns 4.42e+03 ns 9.6e+03 ns 0 50.0 25.0 0.0 25.0 30.8
Coarsen_Q_sigmaT4: 0.014 ms 3.5e+03 ns 1.82e+03 ns 5.78e+03 ns 2 50.0 0.0 25.0 25.0 39.5
carryForward_Var: abskg: 0.0296 ms 7.4e+03 ns 2.4e+03 ns 0.0105 ms 1 25.0 0.0 25.0 50.0 29.5
carryForward_Var: cellType: 0.0753 ms 0.0188 ms 0.0106 ms 0.0373 ms 3 75.0 0.0 0.0 25.0 49.5
carryForward_Var: color: 0.0413 ms 0.0103 ms 5.88e+03 ns 0.0168 ms 3 50.0 25.0 0.0 25.0 38.6
DataArchiver::outputReductionVars: 0.0128 ms 3.2e+03 ns 0 ns 0.0128 ms 0 75.0 0.0 0.0 25.0 75.0
DataArchiver::outputVariables: 0.029 ms 7.25e+03 ns 6.16e+03 ns 9.31e+03 ns 3 75.0 0.0 0.0 25.0 22.2
DataArchiver::outputVariables (CheckpointReduction): 4.11e+03 ns 1.03e+03 ns 0 ns 4.11e+03 ns 0 75.0 0.0 0.0 25.0 75.0
DataArchiver::outputVariables(checkpoint): 0.0323 ms 8.07e+03 ns 7.51e+03 ns 8.97e+03 ns 2 50.0 25.0 0.0 25.0 10.1
RMCRTCommon::sigmaT4: 0.019 ms 4.76e+03 ns 1.68e+03 ns 7.75e+03 ns 0 25.0 0.0 50.0 25.0 38.6
RMCRT_Test::computeStableTimestep: 0.178 ms 0.0444 ms 0.0294 ms 0.0708 ms 3 50.0 25.0 0.0 25.0 37.2
Ray::rayTrace_dataOnion: 15.2 ms 3.8 ms 0.0223 ms 7.92 ms 1 50.0 0.0 0.0 50.0 52.0
Ray::setBoundaryConditions: 0.0595 ms 0.0149 ms 0.0121 ms 0.0213 ms 0 75.0 0.0 0.0 25.0 30.2
reduceSysVar: 65.3 ms 16.3 ms 6.7 ms 32 ms 2 50.0 0.0 25.0 25.0 48.9


--------------------------------------------------------------------------------
ExecTimes:+
--------------------------------------------------------------------------------
Name: Total: Avg: Min: Max: Max rank: Hist % [ Q1 | Q2 | Q3 | Q4 ]: %Load Imbalance
Coarsen_Q_abskg: 2.83 ms 0.707 ms 0.55 ms 1.02 ms 3 75.0 0.0 0.0 25.0 30.5
Coarsen_Q_sigmaT4: 2.82 ms 0.705 ms 0.588 ms 0.993 ms 3 75.0 0.0 0.0 25.0 29.0
carryForward_Var: abskg: 5.48 ms 1.37 ms 0.0281 ms 5.26 ms 3 75.0 0.0 0.0 25.0 73.9
carryForward_Var: cellType: 4.64 ms 1.16 ms 0.887 ms 1.54 ms 3 50.0 0.0 25.0 25.0 24.7
carryForward_Var: color: 1.98 ms 0.495 ms 0.0327 ms 1.41 ms 0 50.0 25.0 0.0 25.0 64.9
DataArchiver::outputReductionVars: 0.0116 ms 2.91e+03 ns 0 ns 0.0116 ms 0 75.0 0.0 0.0 25.0 75.0
DataArchiver::outputVariables: 29 ms 7.24 ms 3.22 ms 14 ms 3 50.0 25.0 0.0 25.0 48.3
DataArchiver::outputVariables (CheckpointReduction): 0.159 ms 0.0398 ms 0 ns 0.159 ms 0 75.0 0.0 0.0 25.0 75.0
DataArchiver::outputVariables(checkpoint): 66.8 ms 16.7 ms 13.8 ms 20.6 ms 3 50.0 0.0 25.0 25.0 18.8
RMCRTCommon::sigmaT4: 1.86 ms 0.465 ms 0.395 ms 0.577 ms 1 50.0 25.0 0.0 25.0 19.4
RMCRT_Test::computeStableTimestep: 0.661 ms 0.165 ms 0.0678 ms 0.267 ms 1 25.0 25.0 25.0 25.0 38.2
Ray::rayTrace_dataOnion: 8.4 s 2.1 s 1.4 s 2.88 s 2 50.0 0.0 0.0 50.0 27.0
Ray::setBoundaryConditions: 6.96 ms 1.74 ms 1.46 ms 2.06 ms 0 25.0 25.0 25.0 25.0 15.5
reduceSysVar: 0.128 s 31.9 ms 15.9 ms 42 ms 0 25.0 25.0 0.0 50.0 24.1

5 lines of code changed in 1 file:

• Core/Util/Timers: Timers.hpp (+5 -5)

jas 2017-04-20 13:18 Rev.: 56452

Update copyright.

1 lines of code changed in 1 file:

• Core/Util/Timers: Timers.hpp (+1)

allen 2017-04-13 16:24 Rev.: 56421

Removed Time.h and replaced with Timers.hpp

1 lines of code changed in 1 file:

• Core/Util/Timers: Timers.hpp (+1 -1)