Disable compressible flow tests on the GPU for the time being. There is some interaction between the data archiver and wasatch on GPUs.
3 lines of code changed in 1 file:
Added Burns-Christion using the Arches Task Interface. Changes to Discrete-Ordinates to be compatible with this interface were also made.
314 lines of code changed in 8 files:
Added "ArenaSoil" option.
0 lines of code changed in 2 files:
Moving Arenisca3PartiallySaturated from Vaango to Uintah. The code needs to be verified to make sure that the deformation gradient and velocity gradient calculations in Uintah are used appropriately. New XML tags also need to be added to Uintah.
1 lines of code changed in 2 files:
Whitespace
5 lines of code changed in 1 file:
Divorce this model from the "DefaultHyperElasticEOS", which was just a copy of
this model's original functionality anyway. It seemed confusing, and since
there is no other EOS available or planned.
44 lines of code changed in 3 files:
Test commit from Ash. (whitespace)
3 lines of code changed in 1 file:
Remove SingleDevice (Unified_SingleDevice) debug stream in favor of masking visible devices with the prefered CUDA_VISIBLE_DEVICES env var.
see: http://www.acceleware.com/blog/cudavisibledevices-masking-gpus
1 lines of code changed in 6 files:
Added a function that removes arches intrusions that aren't relevant to the local patch/patches. This should not change the answers.
58 lines of code changed in 3 files:
Use std::atomic for DetatiledTask external dependency count. Removes CrowdMonitor in MPIScheduler.
Other minor cosmetics while refactoring.
142 lines of code changed in 8 files:
Turn off tab props GPU test until Babak sorts out why it is giving memory errors.
1 lines of code changed in 1 file:
Separate out DependencyBatch and DetailedDependency code. Refactor other related areas. This is a cleanup prior to work on the MPI engine to eliminate redundant all-to-all communication related to RMCRT.
Also add RuntimeStats header/source. Will be used in schedulers soon to provide more accurate reporting of MPI statistics.
1577 lines of code changed in 18 files:
activate TabProps GPU test
8 lines of code changed in 1 file:
Deactivate a test until I can figure out the syntax error in it.
2 lines of code changed in 1 file:
Support for GPU evaluation of tabular properties through TabProps.
Thanks to Babak for getting this ready.
33 lines of code changed in 7 files:
Fixing the functionality of the particle splitting, add splitting of plasticity
variables to UCNH. Have DamageModelFactory create a Null model if none is
specified, rather than issuing a warning. Remove tabs from MMS code.
575 lines of code changed in 10 files:
Add some comments to the particle scaling part of this code to help avoid
future misinterpretations in its use.
14 lines of code changed in 1 file:
Removed the running-sum averaging variables from the deposition model and wall heat transfer model. Removed t_ave_start flag. Modified the way the deposition rate is accounting for multiple environments. This commit is expected to change the answers, and the number of saved variables for the arches rt test: 1GW_RT.
7 lines of code changed in 7 files:
Reworked the verification dialog section. When the user replaces a large number of gold standards
a variable can exceed the system's word limit. Create a temp file that contains
the dialog options and use that instead. This requires the dialog "--file" option.
0 lines of code changed in 2 files:
added additional vars and streams to the in-situ
152 lines of code changed in 7 files:
Second try to clean up the SimState
138 lines of code changed in 12 files:
added short cut to simTime
13 lines of code changed in 1 file:
added debug streams and state vars
284 lines of code changed in 17 files:
Fixed issues that were breaking build.
3 lines of code changed in 1 file:
backed out sim state changes
754 lines of code changed in 166 files:
cleanup of the simulation state - most everything is now private
3 lines of code changed in 1 file:
Added new FVMMaterial which effectively is a SimpleMaterial until new features are added. Also changed class variable names to match the d_* naming convention
142 lines of code changed in 9 files:
Added code to extract the timings for the hypreSolve.
computing average mean time per timestep for 97 timesteps
#file MPIprocs threadsPerMPI totalThreads startTime endTime aveMean
noRad_small.2.32.out 32 0 0 60.37 1627.93 16.1604
noRad_small.4.64.out 64 0 0 33.35 842.55 8.34227
noRad_small.8.128.out 128 0 0 20.13 470.40 4.64196
noRad_small.16.256.out 256 0 0 15.54 322.97 3.16938
noRad_small.32.512.out 512 0 0 11.49 207.05 2.01608
noRad_small.64.1024.out 1024 0 0 9.20 139.06 1.33876
noRad_small.128.2048.out 2048 0 0 10.37 113.37 1.06186
noRad_small.256.4096.out 4096 0 0 12.11 129.25 1.20763
Now computing the averages for the SCI_DEBUG: ComponenTimings variables, if they exist.
#file MPIprocs threadsPerMPI totalThreads aveTaskExec aveTaskLocalComm aveTaskWaitComm
noRad_small.2.32.out 32 0 0 10.9606 0.0893772 1.46842
noRad_small.4.64.out 64 0 0 5.53971 0.0840534 0.88168
noRad_small.8.128.out 128 0 0 2.87376 0.0795455 0.366519
noRad_small.16.256.out 256 0 0 1.60823 0.0746436 0.433267
noRad_small.32.512.out 512 0 0 0.962228 0.0684513 0.342114
noRad_small.64.1024.out 1024 0 0 0.62706 0.0677906 0.187067
noRad_small.128.2048.out 2048 0 0 0.473545 0.0656119 0.157249
noRad_small.256.4096.out 4096 0 0 0.623889 0.0680324 0.146094
Now computing the averages for the hypre solve for the label: pressurePS, if they exist.
#file MPIprocs threadsPerMPI totalThreads aveHypreSolveTime
noRad_small.2.32.out 32 0 0 1.57329
noRad_small.4.64.out 64 0 0 0.782116
noRad_small.8.128.out 128 0 0 0.420424
noRad_small.16.256.out 256 0 0 0.343525
noRad_small.32.512.out 512 0 0 0.289876
noRad_small.64.1024.out 1024 0 0 0.228748
noRad_small.128.2048.out 2048 0 0 0.218502
noRad_small.256.4096.out 4096 0 0 0.449462
0 lines of code changed in 2 files:
cleanup of the simulation state - most everything is now private
10 lines of code changed in 1 file:
cleanup of the simulation state - most everything is now private
32 lines of code changed in 11 files:
white space clean up
1 lines of code changed in 1 file:
addded state var to the in-situ
222 lines of code changed in 8 files:
cleanup of the simulation state - most everything is now private
702 lines of code changed in 97 files:
compare_uda:
- changed cerr messages -> ostringstream and passing the stream into the abort_uncomparable();
abort_uncomparble(): slightly better reporting to the user why comparison aborted.
compare_sus_runs & runSusTests
- reduce the amount of noise pumped out by local/nightly RT when the input files differ.
- slightly better reporting to the user why a comparison failed.
193 lines of code changed in 4 files:
moved the in-situ state to be after creation
4 lines of code changed in 1 file:
added debug stream wrapper for the in-situ
3 lines of code changed in 1 file:
added stream vars
206 lines of code changed in 7 files:
added a file name, gave default stream, and removed unused var
0 lines of code changed in 4 files:
Added boundary conditions for the FVM component
228 lines of code changed in 5 files:
Fix a leak. Wrong variable type was being requested. Probably should
bullet-proof this for the future.
0 lines of code changed in 2 files:
Fix the calculation of number of particles in the angular direction
for the case of "constant_particle_volumes".
0 lines of code changed in 2 files:
Replace '@' with 'HEAD' as '@' gave errors on several machines.
4 lines of code changed in 1 file:
remove a cout statement
0 lines of code changed in 2 files:
fixed indexing
2 lines of code changed in 1 file:
Fix broken GPU build.
2 lines of code changed in 1 file:
Fix a bug in the surface normal calculation.
0 lines of code changed in 2 files:
removed MPI check
4 lines of code changed in 1 file:
added patch id
35 lines of code changed in 1 file:
added additional NaN info
30 lines of code changed in 3 files:
added header to exception message
0 lines of code changed in 2 files:
added check for stall patches
7 lines of code changed in 1 file:
removed all parallel dependency
0 lines of code changed in 2 files:
Dont output init timestep on coal_channel_hi_vel because this seems to
be causing problems for the RT.
Clean up some warnings.
12 lines of code changed in 7 files:
Missing include.
0 lines of code changed in 2 files:
Removing Operators.
579 lines of code changed in 106 files:
Removing spatialops includes.
Converting Lagrangian particle to standard Uintah loops.
208 lines of code changed in 36 files:
Actually remove the ArchesBCHelper.*
0 lines of code changed in 2 files:
Remove BCHelper.
Cleanup “using namespace” in BoundaryCond_new.
Silence a warning in DigitalFilterInlet (better way to do this?)
106 lines of code changed in 12 files:
Change relaxation_coe to relation_coef.
Change channel_WD_CO to have a shorter, symmetric geometry.
1 lines of code changed in 1 file:
Kokkos for RateDeposition.
Committed for Minmin (he did the hard work).
757 lines of code changed in 2 files:
Make sure coal density is initialized to zero in the extra cells.
Add a wall-deposition test (contribution from Minmin).
9 lines of code changed in 2 files:
Adding a new coal test.
2 lines of code changed in 1 file:
Turn off Yamamoto. This model isnt implemented correctly and is
producing garbage.
Change the branching in CoalDensity. This keeps answers consistent with
the SpatialOps implementation so as not to change answers.
43 lines of code changed in 2 files:
Turning off coal_channel_Yamamoto. This model appears to be out of date
and needs updating before turning this test back on.
2 lines of code changed in 1 file:
Kokkos -> ShaddixOxidation. This seems to have changed the answers for
CQMOM_coal_test. This isnt too surprising because the SO version was
iterating the entire patch with the same number of iterations while this
new version is iterating pointwise. The differences seem like fuzz.
310 lines of code changed in 3 files:
Kokkos -> ShaddixEnthalpy.
301 lines of code changed in 2 files:
Kokkos into FOWY (new version).
This model still needs to be updated to be consistent with the version
Ben was developing.
184 lines of code changed in 2 files:
Fixed the bug with partMassFlow.ups and cleaned up the files a little.
196 lines of code changed in 4 files:
Kokkos in variable stats. Answers arent changed.
**Note that it appears something affecting partMassFlow has snuck
through my set of commits. The next commit should fix this.
174 lines of code changed in 1 file:
Allow for the return of native ParticleVariables.
Update RandParticleLoc to meet new API.
Kokkos in WallHFVariable. This still needs work.
136 lines of code changed in 5 files:
Kokkos into CO and GridInfo.
57 lines of code changed in 4 files:
Removing old transport in favor of the Kokkos transport.
154 lines of code changed in 10 files:
Remove debugging print statements in SurfaceNormals.
Kokkos for DensityPredictor.
Added a helper KokkosTools header where we can place some commonly used
lines of code.
99 lines of code changed in 4 files:
Kokkos in SurfaceNormals. In the process removed an unused variable.
Also found a bug in GridTools when defining buffered patch ranges for
the y-direction specifically.
72 lines of code changed in 2 files:
More kokkos into Arches.
Injecting some parallel_for into DepositionVelocity. This still needs
work.
78 lines of code changed in 6 files:
Turns out that multipying your model by a random number is a bad idea.
(fix a missing initialization on m_pi)
80 lines of code changed in 1 file:
Missing initialization of a couple variables.
3 lines of code changed in 1 file:
Get the example cleaned up and working.
22 lines of code changed in 2 files:
Drag model -> Kokkos
Updated input files + spec.
114 lines of code changed in 5 files:
Kokkos for CoalTemperature.
Update OFC4_smith to use correct input.
19 lines of code changed in 1 file:
GridTools->ArchesCore namespace.
54 lines of code changed in 11 files:
Typos.
2 lines of code changed in 1 file:
Kokkos-ized CoalDensity.
56 lines of code changed in 1 file:
Kokkos-izing BodyForce.
56 lines of code changed in 2 files:
Constant Particle Lambda.
28 lines of code changed in 2 files:
Lambdas for CoalMassClip. This model wasn't under RT but isn\'t being
use. It may need to be pruned.
10 lines of code changed in 1 file:
Remove CoalTemperatureNebo.
Update SampleTask with lambdas.
42 lines of code changed in 9 files:
Use Output pointer member from SchedulerCommon only.
15 lines of code changed in 4 files:
Added new label class for the finite volume component
106 lines of code changed in 5 files:
Significant refactoring and clenaup in schedulers and loadbalancers prior to assesment of the required changes to the MPI engine to support Kokkos views. Also moving away from all mutex protected debug output, in favor of the light-weight, printf-based Dout class and DOUT macros. We now get fully coherent debug output, regardless of the proc/thread counts. Also moving to standardized naming conventions for class, static, thread-local and global variables.
1319 lines of code changed in 32 files:
Get rid of the SetCurrentTime expression. All Wasatch calculations of time related quantities (dt, t, tstep, rkstage...) take place in traditional Uintah tasks now. Access to these quantities is done via Expression Placeholders on SingleValueFields.
0 lines of code changed in 5 files:
Add a task to set the time at initialization. This brings us one step closer to removing the dependency on SetCurrentTime.
70 lines of code changed in 3 files:
Adds an additional preprocessor define to handle the case between cblas.h from macports and the native MacOSX native veclib.
4 lines of code changed in 3 files:
Don't source .cshrc file since it may contain module commands that confuse the batch scheduler
4 lines of code changed in 1 file:
fix memory leaks in Wasatch. This should fix the RT as well.
9 lines of code changed in 1 file:
A fix for an RMCRT/Arches scenario in which a single dependency source was needed for multiple patches at the destination.
169 lines of code changed in 1 file:
***This will change answers***
Fixing a bug that was introduced when attempting to set tangential
velocity BCs to be consistent on the face of the computational domain.
The index was moving in the wrong direction at the boundaries which
resulted in tangential velocities being set on the interior flow cell
next to the boundary rather than in the extra cell.
27 lines of code changed in 1 file:
Fix race condition in CharOxidationSmith.
Added copyright header and removed the opening up of std namespace.
Also removed the include of a SpatialOps header that wasn't being used.
86 lines of code changed in 3 files:
Loosen the tolerance on the GPU compressible tests.
3 lines of code changed in 1 file:
Fix bug in the build_wasatch_3p script. This should be bulletproof now and will update W3P as needed.
15 lines of code changed in 1 file:
Update Wasatch to use PerPatch<double> instead of PerPatch<double*>. This goes hand in hand with Brad's commit of yesterday as well as changes in ExprLib.
In addition, fix the updateCurrentTime task to avoid task dependency issues on GPU. This removes the dependence on SetCurrentTime.
100 lines of code changed in 5 files:
This patch has two parts. The first part changes the PerPatch variable so that uses C++11 shared_ptr so that it never makes a copy of the value, it instead preserves a pointer to a single value. This helps Wasatch who needs pointers to direct data. I've run this file against the full gold standard tests without issue. The others are some sizing fixes to the UnifiedScheduler knows how many PerPatch bytes to copy, before it was copying 0 bytes.
27 lines of code changed in 6 files:
Some fixes to ensure PerPatch and Reduction variables go into the GPU properly and with concurrency checks
100 lines of code changed in 1 file:
Fixed unnecessary limitation in charOxySmith requiring fields to be saved to disk as specified by the dataArchiver block.
28 lines of code changed in 1 file:
Fix incorrect extern decl for Dout objects.
6 lines of code changed in 2 files:
Small refactoring of UnifiedScheduler-related DebugStreams. Next step is replacement of these with Dout class and DOUT macros - removing all usage of cerr/cout Locks.
Update environmentalFlags.txt
8 lines of code changed in 5 files:
disable SSL server certification verification for wasatch3p.
10 lines of code changed in 1 file:
Remove CommRecMPI source/header files.
Begin incorporation of Dout class and DOUT macro for debug reporting - simple, printf-based and fully thread-safe, needing no locks for output. This also allows us to move away from DebugStream (which inherits from the standard library). DOUT fully co-opts the SCI_DEBUG flags, so the original names work as tehy alwasy have.
DOUT is wonderful and actually provides coherent output for multiple threads/ranks, etc and avoids the massive code clutter necessary for the same degree of safety in the DebugStreams.
Thanks to Dan S. for this great insight.
234 lines of code changed in 14 files:
Introduction of the Lockfree Pool data structure and the new CommunicationList. Using this now to replace CommRecMPI and its use of the problematic MPI_Testsome() and MPIWaitsome() calls. We now store individual requests in a lock/wait/contention free Pool and call MPI_Test() and MPI_Wait on individual MPI_Requests.
This fixes the MPI_Buffer memory leak seen in the threaded scheduler, in which multiple threads think they will recieve a message, allocate a buffer and then only one thread does the actual recieve and calls the after-communcation handler to clean up the buffer. This memory leak was most pronounced at large scale with RMCRT due to the global halo requirement.
Also backing out the support for non-uniform ghost cells across AMR levels for now, until the issue of some required messages not being generated from coarse radiation mesh to some ranks. This has to do within partial dependencybatches being created due to an incomplete processor neighborhood list across levels.
MISC:
* cleaned up some old TAU remnants in doc directory and build system
* refactor and cleanup in MPIScheduler
* removed unused source code, including the old ThreadedScheduler (only one threaded scheduler now - Unified)
* cleaned up non-existent entreis in environmentalFlags.txt
2433 lines of code changed in 35 files: