Lines of Code
| Author | Changes | Lines of Code | Lines per Change |
|---|---|---|---|
| Totals | 44 (100.0%) | 810 (100.0%) | 18.4 |
| dav | 14 (31.8%) | 386 (47.7%) | 27.5 |
| jas | 11 (25.0%) | 344 (42.5%) | 31.2 |
| ahumphrey | 14 (31.8%) | 48 (5.9%) | 3.4 |
| harman | 5 (11.4%) | 32 (4.0%) | 6.4 |
Reverting code to r52880 before my memory use fix commit. While most tests passed, the ones using AMR and the dynamic load balancer were not passing and the weeds are too deep within them to quickly figure out what is causing the problems. Next week I will attempt to figure out what is going on with them. My guess is that it will take quite a bit of effort to understand the inner workings of the dynamic loadbalancer in order to get everything to work, so it may be quite a while before this fix may go in.
112 lines of code changed in 6 files:
This is the first commit (of what may be several over the next month or so) dealing with Uintah's excessive use
of memory with respect to its XML data structures (that are used for I/O) when running on thousands of patches.
This commit specificly addresses the excessive use of memory when restarting a simulation. Before this commit,
Uintah would read in a timestep.xml file (from the checkpoints/t00000/) directory and create an huge DOM
structure in memory. (On 8K patches on Vulcan (BGQ - LLNL) memory per process would increase by 100% (by
20 MB per process on the advect sample problem). On 30K patches, memory use of 200 MB or more has been observed.)
To solve this problem, instead of reading the XML into memory, we parse it on the fly and extract the data we need,
but do not save it. Note, there are two parts of the timestep.xml file that cause the memory issue: the 1000s of
<Grid> and <DataItem> tags. Small XML nodes are not a problem, and thus we still use them for some things (for
example, the component section read from the timestep.xml file. A full DOM of this node is created (during the
streaming read of the rest of the file) and this node is passed down to the component as before.
A number of changes in this commit are cosmetic, dealing with white space and coding standards, and were made
while I was digging through the code to understand where the memory issue was and how to fix it. To better
see real changes in the code, I highly suggest that you use a diff that ignores generic white space changes,
such as:
svn diff --diff-cmd diff --extensions -w <file_name>
Note,asIgetolderInolongercanreadtextlikethis,soIamaddingspacesintothecodetomakeit easier to see. :)
M CCA/Components/SimulationController/AMRSimulationController.cc
- Number of levels in a Grid (grid->numLevels()) cannot be negative and thus should be an unsigned int... it is now.
- Costmetic: English sentences start with capital letters and end with punctuation... yeah yeah I know.
M CCA/Components/SimulationController/SimulationController.cc
- queryGrid: pass through a pointer, don't create a new copy.
- Unsigned int for grid level number.
- Cosmetic white space/coding standards/English updates. Fixed some strange indentation. Untabify.
M CCA/Components/Parent/Switcher.cc
- Unsigned int for grid level number.
- Cosmetic: Swatted some evil endl's. :) (Why write 8 more characters of code, when 2 will do? And makes it easier to add to strings later...)
M CCA/Components/LoadBalancers/ParticleLoadBalancer.cc
M CCA/Components/LoadBalancers/ParticleLoadBalancer.h
- Updates to consolidate code in load balancers... Still needs some direct attention as opposed to this
first pass while working on something else.
M CCA/Components/LoadBalancers/SingleProcessorLoadBalancer.h
- Add 'do-nothing' implementation of restartInitialize() to this class.
M CCA/Components/LoadBalancers/CostModelForecaster.cc
- Cosmetic: Untabbify, white space, {} around one line ifs, English.
M CCA/Components/LoadBalancers/DynamicLoadBalancer.h
M CCA/Components/LoadBalancers/DynamicLoadBalancer.cc
- Moved restartInitialize() (and a number of supporting class d_ variables) into LoadBalancerCommon.
- Cosmetic: Line up variables, untabbify, white space, {} around one line ifs, English.
M CCA/Components/LoadBalancers/ProfileDriver.cc
M CCA/Components/LoadBalancers/CostModeler.h
- grid->numLevels() is now unsigned updates.
- Coding standard cleanups / English / white space.
M CCA/Components/LoadBalancers/LoadBalancerCommon.cc
- Cosmetic: Alphabetize/sort #includes, line up variables, untabbify, white space, {} around one line ifs, English.
- Make global DebugStream variables be less global (made static to file).
- Unsigned int for grid level number.
- Move restartInitialize() from DynamicLoadBalancer and ParticleLoadBalancer (code was exact duplicate in both files) here.
M CCA/Components/Solvers/AMR/AMRSolver.cc
- Cosmetic: White space, line up variables, English.
- Unsigned int for grid level number.
M CCA/Components/DataArchiver/DataArchiver.cc
M CCA/Components/DataArchiver/DataArchiver.h
- Comment added for where next XML memory fix should take place.
- Cosmetic: Alphabetize/sort #includes, line up variables, untabbify, white space, {} around one line ifs, English.
M CCA/Ports/ProblemSpecInterface.h
M CCA/Components/ProblemSpecification/ProblemSpecReader.h
M CCA/Components/ProblemSpecification/ProblemSpecReader.cc
- Created new version of readInputFile() which takes a list of patches that are specific to this processor. However,
I don't think anyone (me?) is using it yet, and I'm actually not convinced that it is the right function to fix
(see comments in code). I'm committing it just in case it turns out that we need it as we fix the rest of the
XML memory bloat issues. However, at that point if it turns out not to be useful, it should be removed.
M CCA/Components/Schedulers/TaskGraph.h
M CCA/Components/Schedulers/TaskGraph.cc
- Add some consistency to the member variable naming in this file. Half the variables use d_, and the other half
have a trailing _, and the third half ;) didn't do either one... sigh. Some examples.
lb -> d_lb
dts_ -> d_dts
- A few cosmetic fixes - variable line up, white space, etc.
- Removed var/matl from lb->getOldProcessorAssignment()... need to verify that this is correct... see above.
M CCA/Components/Schedulers/OnDemandDataWarehouse.cc
- Removed unused parameters to lb->getOldProcessorAssignment(). However, the DynamicLoadBalancer still uses the
version with parameters (though it doesn't really need them either). It is unclear why getOldProcessorAssignment()
is not declared in LoadBalancerCommon common or several of the other Load Balancers, and also unclear which function
is actually being called. This is defined in the LoadBalancer.h in CCA/Ports/. This needs more investigation.
M CCA/Components/Schedulers/DetailedTasks.h
M CCA/Components/Schedulers/DetailedTasks.cc
- Almost all cosmetic, white space, untabbify, lining up vars, English, etc changes.
- Add trailing _ to numPendingInternalDependencies to denote member var.
M CCA/Components/Schedulers/MPIScheduler.cc
- Few cosmetic cleanups.
M CCA/Components/Schedulers/SchedulerCommon.cc
M StandAlone/tools/pfs/rawToUniqueGrains.cc
M StandAlone/tools/puda/PIC.cc
M StandAlone/tools/extractors/extractV.cc
M StandAlone/tools/extractors/extractF.cc
M StandAlone/tools/extractors/partextract.cc
M StandAlone/partvarRange.cc
M StandAlone/tools/puda/jim1.cc
M StandAlone/tools/puda/jim3.cc
M StandAlone/tools/puda/varsummary.cc
M StandAlone/tools/puda/puda.cc
M StandAlone/tools/puda/POL.cc
M StandAlone/slb.cc
- Cosmetic white space fixes.
- numLevels() casts to avoid warnings.
M CCA/Components/Arches/Radiation/RadHypreSolver.cc
- Few cosmetic changes.
M CCA/Components/Regridder/RegridderCommon.cc
M CCA/Components/Regridder/SingleLevelRegridder.cc
M CCA/Components/Regridder/TiledRegridder.cc
- Cosmetic white space fixes.
- numLevels() casts to avoid warnings.
M CCA/Ports/Output.h
- Use const vars when possible.
- White space, English.
M CCA/Ports/LoadBalancer.h
- Cosmetic: Alphabetize/sort #includes, line up variables, untabbify, white space, {} around one line ifs, English.
M Core/OS/Dir.h
M Core/OS/Dir.cc
M Core/OS/Dir_boost.cc
- Use 'const' when variables don't change.
- Cosmetic: line up variables, untabbify, white space, {} around one line ifs, English.
- Don't use "std::" in half the file and not in the other half.
M Core/Grid/Patch.h
M Core/Grid/Patch.cc
- Cosmetic: Line up variables, untabbify, white space, {} around one line ifs, English.
M Core/Grid/Grid.h
M Core/Grid/Grid.cc
- Added readLevelsFromFile(), parseLevelFromFile(), and parseGridFromFile() to parse XML without creating a full DOM.
This provides a 50% memory decrease (per process) on Vulcan (LLNL-BGQ) for 8K processors when restarting the
advect problem. At some point in the future it might be nice to replace my (brute force - straightfoward -
assumes that the input xml file is very well behaved) approach with the streaming reader functions from the
lib xml library. This is not done at this point as it is unclear how well that library will fit into our use
case, how much time it will take to convert to, and the fact that we want to get this pushed out to the users asap.
- Cosmetic: Line up variables, untabbify, white space, {} around one line ifs, English, coding standards.
- numLevels is now an unsigned int.
M Core/Grid/Task.cc
- Cosmetic: Alphabetize/sort #includes, line up variables, untabbify, white space, {} around one line ifs, English.
M Core/Grid/SimulationState.h
M Core/Grid/SimulationState.cc
- Cosmetic: Alphabetize/sort #includes.
M Core/Grid/Level.h
M Core/Grid/Level.cc
- Returned patch from addPatch() is not used anywhere so remove the return type.
- Cosmetic: line up variables, untabbify, white space, {} around one line ifs, English.
M Core/Geometry/Vector.h
M Core/Geometry/Vector.cc
- Added the fromString() function.
- Cosmetic: Alphabetize/sort #includes.
M Core/Geometry/IntVector.h
M Core/Geometry/IntVector.cc
- Added the fromString() function.
M Core/Containers/HashTable.h
- Moved the call to first() into the constructor... Not quite sure what the reason for this is,
but everywhere a HashTable is created, "first()" is immediately called, so it makes sense
to just do it in once place instead of having to do it each time it is used.
D Core/Containers/StringUtil.cc
D Core/Containers/StringUtil.h
A + Core/Util/StringUtil.h
A + Core/Util/StringUtil.cc
M Core/Grid/SimulationTime.cc
M Core/Tracker/TrackerServer.cc
M Core/Persistent/Pstreams.cc
M Core/Persistent/Persistent.cc
M Core/Util/FileUtils.cc
- Moved StringUtil.* to Core/Util/ to all its use in other locations (higher up the build tree).
M Core/Containers/sub.mk
- Untabbify.
M Core/DataArchive/DataArchive.h
M Core/DataArchive/DataArchive.cc
- Rename variables so they are readable, eg: d_tsurl becomes d_ts_path_and_filename, d_tstimes becomes d_ts_times, etc.
- Change a few more functions to parse xml file data on the fly (instead of parsing DOMs), eg: queryEndiannessAndBits()
- Use NULL instead of 0.
- Pass strings as 'const &' instead of copying them every time.
- Cosmetic: Alphabetize/sort #includes, line up variables, untabbify, white space, {} around one line ifs, English.
M Core/ProblemSpec/ProblemSpec.h
M Core/ProblemSpec/ProblemSpec.cc
- Added constructor to take in a string buffer to create a new DOM from. (Use: stream parse an xml file and put the
needed pieces into a string, then turn only that information into a DOM.)
- Added routines to help in the stream parsing of an XML file: findBlock()
- checkForInputError() replaced.
- parseIntVector replaced with fromString() in respective files.
M Core/Util/DynamicLoader.cc
- Alphabetize #includes.
M Core/Util/sub.mk
- Alphabetize
A Core/Util/XMLUtils.h
A Core/Util/XMLUtils.cc
- Added new helper functions for parsing xml files on the fly.
M StandAlone/compare_uda.cc
- Cosmetic: Alphabetize/sort #includes, line up variables, untabbify, white space, {} around one line ifs, English.
- numLevels() is unsigned now.
- Bulletproofing - verify input args are actually directories.
M StandAlone/restart_merger.cc
- Minor cosmetic fixes.
M StandAlone/sus.cc
- Bulletproofing - catch case when user accidentally runs sus with a UDA and not a UPS.
128 lines of code changed in 3 files:
Remove REDSTORM artifacts.
0 lines of code changed in 4 files:
A number of clean ups for Vulcan (LLNL - BGQ machine). Most
importantly, fixed memory usage reporting (see ProcessInfo.cc).
M CCA/Components/SimulationController/AMRSimulationController.h
M CCA/Components/SimulationController/AMRSimulationController.cc
- Added white space, removed tabs, line up params, etc.
M CCA/Components/SimulationController/SimulationController.cc
- Moved toHumanUnits() to ProcessInfo so it can be used by others.
- Line up variables, add white space.
- Consistent (correct) capitalization/punctuation of some comments.
M CCA/Components/LoadBalancers/SingleProcessorLoadBalancer.h
M CCA/Components/LoadBalancers/SingleProcessorLoadBalancer.cc
- inNeighborhood() did not have the same declaration as its parent
(superclass)... removed "const MaterialSubset*" to fix this... not
sure how this worked before.
- White space.
M CCA/Components/LoadBalancers/LoadBalancerCommon.h
- Add "d_" to several class variables to be consistent and conform to coding standard.
- Removed extra (unnecessary) level of indentation.
- White space, line up variables.
M CCA/Components/LoadBalancers/LoadBalancerCommon.cc
- Add "d_" to several class variables to be consistent and conform to coding standard.
- White space.
M CCA/Components/Schedulers/OnDemandDataWarehouse.cc
- Alphabetize #includes to more easily determine what is included.
- White space, line up variables.
M CCA/Components/Schedulers/UnifiedScheduler.cc
- Minor white space, coding standard update.
M CCA/Components/Schedulers/MPIScheduler.cc
- Minor white space fix.
M CCA/Components/Schedulers/SchedulerCommon.cc
- ProcessInfo GetMemoryResident() renamed with lowercase 'g' to match coding standard.
- Added processor rank to commented out memory usage print statement.
M CCA/Components/Arches/Radiation/RadPropertyCalculator.h
- Alphabetize #includes, remove superfluous ;s, remove tabs, line up functions/variables.
M Core/OS/ProcessInfo.h
M Core/OS/ProcessInfo.cc
- Fixed (added) BGQ memory usage querying code. Previously, it was
calling the generic linux version, and trying to open
/proc/<pid>/status to get memory info. On BGQ machines, this file
does not exist (except on the I/O node - where, while it exists,
it does not have valid information in it). In the generic linux
version, if the file fails to open, then an error is NOT thrown,
and 0 is returned... most likely we should fix this so that it
throws an error...
Note, the code for using the "mallinfo" (for BGQ) version of memory
checking was also added to this file, but is commented out and not
used because it uses an integer as the return variable and thus
does not correctly handle systems with more than 2 GB of memory.
- Coding standard updates.
- Moved toHumanUnits() here.
- Use lowercase letter for first letter in function names to be consistent and follow standard.
M Core/Parallel/Parallel.cc
M Core/Thread/Thread_pthreads.cc
- Minor white space, capitalization/punctuation.
M StandAlone/sub.mk
- For static builds, need Wasatch 3P libs to link.
M StandAlone/sus.cc
- Alphabetize #includes.
140 lines of code changed in 3 files:
Finish stripping out Windows support.
Also update UPS_Spec from previous commit.
4 lines of code changed in 3 files:
Fix for inconsistencies with the namespace boost uses for its
filesystem utilities. I had originally seen this on Vulcan, but have
duplicated it on Todd's Debian system Fin. On Fin there are two
versions of boost installed. While the both claim to be using the
boost filesystem version 3, only one of them actually uses the
namespace boost::filesystem3... The other, and the default on many
machines, is boost::filesystem.
I have added a test in configure to determine which namespace to use
and then updated the code to make the correct choice. I have tested
this on Fin, and will verify that it works on other systems after
this commit (in addition to letting the RT take a swing at it too).
- Dav
M src/configure.ac
M src/configure
- Remove the include dir with_boost/include/boost as the '/boost'
comes from the #include line.
- Added check for namespace boost::filesystem3.
M src/build_scripts/build_wasatch_3p.sh
- If boost in in the default /usr location, then don't use CMAKE boost
flags.
M src/include/sci_defs/boost_testdefs.h.in
- Add "#define UINTAH_BOOST_FILESYSTEM_NAMESPACE_V3 1" as necessary to this file.
M src/Core/OS/Dir_boost.cc
- #include sci_defs/boost_defs.h to determine which boost filesystem namespace to use.
5 lines of code changed in 1 file:
Excise Windows support in Core/OS
44 lines of code changed in 7 files:
Fixes for the previous major commit (for Vulcan/BGQ) that broke other OS builds...
M Math/sci_lapack.cc
- Need to #include uintah_defs to get the FORTRAN_UNDERSCORE policy
in order to properly append the _ to fortran function names.
M OS/Dir_boost.cc
- On Vulcan, the boost library uses the namespace boost::filesystem3. However,
it appears that no one else does this. This is a temporary hack until I
determine a better/cleaner way to know which namespace to use.
1 lines of code changed in 1 file:
Updates to allow for a (almost) clean configure/build on LLNL's BGQ Vulcan machine.
M configure.ac
M configure
- Various cleanups.
- Added --with-essl so you have to explicitly ask for it (previously configure
tried to find it for you on certain machines).
- Removed stray 'echo'.
- Added test to determine if we are running on a BGQ machine.
- Don't use 'type' to find an executable, use 'which' instead.
'type' doesn't always behave the way one would expect.
- Allow for using the mpigcc compiler directly.
- Fix the specification of FORTRAN_UNDERSCORE for xlf.
- Use a .d for depend files on BGQ machines.
- If compiling on a BGQ, then use -lmass instead of -lm for the math library.
- Removed test for PNG (we don't need/use/support it anymore).
PNG was need for NRRD support at one point.
- Removed test for MPEG (Fairly sure this was only needed for SCIRun).
- Allow for --with-mpi=built-in (Ie, using mpigcc compilers)
We originally didn't like using mpiCC compilers as they never quite
worked with configure. However, it appears that a lot of work as been
done over the years making them work better. Also, at LLNL, there are
so many paths that are needed (and separate libraries) for MPI support
that it is just easier to allow the mpigcc compiler to provide them.
- Update the --with-mpi check to loop over possible libraries instead of
having to hard code every possible test in a separate SCI_TRY_LINK.
- Fix --with-petsc so that if you give it an invalid dir, it let's you know.
- Add magic PETSc compiler flags for BGQ as we are using the gcc compiler
but PETSc was compiled with xlc...
M configVars.mk.in
- Add IS_BGQ flag.
- Remove PNG and MPEG references.
- Hack: Add in PETSc flags for xlc petsc use with gcc.
- Make a few HAVE_lib/lib_INCLUDE/lib_LIBRARY sections be laid out consistently.
- Don't put @flag@ in the INCLUDES variable... make a stand-alone var for them
and use them the way all the other vars are used.
- Move the specification of the CORE_STATIC_LIBS to after the specification of
the F_LIBRARY (fortran library) as they need to reference it.
M aclocal.m4
- Renamed 'debugging' flag to 'verbose'.
- Remove RPATH from the link line when linking statically.
M Makefile.in
- Make 'cleanreally' remove all libs (lib/*.so lib/*.a lib/*.dylib) in
addition to all .o files.
M build_scripts/so_epilogue.mk
- No real changes, just indentation so it is readable...
M StandAlone/sub.mk
- Remove unwanted tabs, add whitespace to make more readable.
M StandAlone/tools/puda/sub.mk
- Add Core/Datatypes to the list of required static libs for linking.
M include/sci_defs/error_testdefs.h.in
- Added in a #if check that allows an individual file to override
the SCI_ASSERTION_LEVEL. This is required (at least for now)
because on Vulcan (BGQ), if assertions are off, then the Datatypes.cc
file causes a run time segfault (in both debug and optimized
builds). However, if assertions are turned on for this one file,
then it runs correctly. Still need to dig into what is going on...
M CCA/Components/Models/Radiation/sub.mk
- Needed INCLUDE flags should be specified for each library.
M CCA/Components/MPM/ConstitutiveModel/fortran/sub.mk
- Some of the fortran flags that are hard coded for this library
are not valid on the BGQ machine.
M CCA/Components/MPMArches/fortran/param4.h
M CCA/Components/Models/Radiation/fortran/m_param4.h
M CCA/Components/Arches/fortran/param4.h
- Remove C style comments as they don't compile using xlf on Vulcan.
M CCA/Components/MPM/ConstitutiveModel/fortran/MIGUtils.cc
- Add some magic macro hackiness to fix symbol names (dealing with
fortran underscores) instead of hard coding them.
M CCA/Components/DataArchiver/DataArchiver.cc
- Fix spacing/whitespace issues.
- Check for BGQ and if so, don't use fdatasync() as it is not implemented.
M Core/OS/Dir_boost.cc
- Newest ? boost uses boost:filesystem3, so we need to check and use the
correct namespace.
M Core/Math/sci_lapack.cc
- Add some magic macro hackiness to fix symbol names (dealing with
fortran underscores) instead of hard coding them.
- Add warning messages if lapacksvd is called but the code path is
not implemented...
M Core/Datatypes/Matrix.h
M Core/Datatypes/SearchGrid.h
M Core/Datatypes/Datatype.h
M Core/Datatypes/Color.h
M Core/Datatypes/TypeName.h
M Core/Datatypes/NrrdData.h
M Core/Datatypes/String.h
M Core/Datatypes/MatrixOperations.h
M Core/Datatypes/DenseColMajMatrix.h
M Core/Datatypes/SparseRowMatrix.h
M Core/Datatypes/PropertyManager.h
M Core/Datatypes/DenseMatrix.cc
M Core/Datatypes/Image.h
M Core/Datatypes/ColumnMatrix.h
M Core/Datatypes/DenseMatrix.h
- share.h has been removed (Windows dll thing)... Therefore remove
(now) meaningly SCISHARE.
M Core/Malloc/malloc.cc
- If DISABLE_SCI_MALLOC is not defined, then we should not be prototyping
malloc(), free(), etc.
5 lines of code changed in 1 file:
getFilenamesBySuffix()
fixed one minor memory leak
5 lines of code changed in 2 files:
Dir::copy(DIR)
- ignore copying files that already exist in the new directory.
- removed stray cout.
21 lines of code changed in 1 file:
DataArchiver.cc
- reduceUdaSetup()
Just use a plain copy for coping the checkpoints.
Now copying the original ups and input.xml.org files
-beginOutputTimestep()
fixed compiler warning
- copy_outputProblemSpec()
cleaned up output statement
AMRSimulationController.cc
- run() inside reduceUda conditional block
setting the intital physical time from reduceUda
Dir.cc
- getFilenamesBySuffix()
comment out cout statement
UdaReducer.cc
- polished, added comments, cleaned up user output
- added user message
- initialize()
set delt = 0. You must have this if outputInitTimestep was used.
- computeLabels()
removed code that will never be used.
- deleted finalizeTimestep related code. It's not needed
- needRecompile()
recompile the taskgraph every timestep. The saved variable list
and the number of matls could change from timestep to timestep.
reduce_uda is in final testing phase. All initial tests show that it's working.
1 lines of code changed in 1 file:
Updated copyright information.
9 lines of code changed in 9 files:
Squashed commit of the following:
commit 140a531e374ae16718f57324fdcc8c1c1c070de1
Author: John Schmidt <john.schmidt@utah.edu>
Date: Thu Jan 2 15:32:48 2014 -0700
Implement recursive copying of a directory.
commit e1093af35e37c88e63869ed1683a45be407f3f27
Author: John Schmidt <john.schmidt@utah.edu>
Date: Thu Dec 12 18:43:16 2013 -0700
Remove preprocessor directors for filesystem version 2 & 3 that were
found around the filesystem::copy().
commit 56f2c3f9944fdc9fe60bdbad336bd79c96207673
Author: John Schmidt <john.schmidt@utah.edu>
Date: Sun Nov 24 19:02:14 2013 -0700
Add boost filesystem to the configure check and the Dir_boost.cc to the sub.mk.
commit 0a5a9119fc3501ea8ce56112812296bf57fbe0b4
Author: John Schmidt <john.schmidt@utah.edu>
Date: Sun Nov 24 19:01:17 2013 -0700
Add the boost filesystem calls for doing directory manipulations.
335 lines of code changed in 2 files: