1.7. Troubleshooting

• 1.7.1. Introduction
• 1.7.2. General Advice on Debugging Run time Problems
  • 1.7.2.1. List of common problems to watch out for when developing / reviewing code
    • 1.7.2.1.1. Possible sources of model crashes or non-physical results
      • 1.7.2.1.1.1. Potential for divide-by-zero
      • 1.7.2.1.1.2. Potential for other floating point exceptions (e.g,. raising 0 to a negative power, etc.)
      • 1.7.2.1.1.3. Potential for a quantity that should be non-negative to go negative, due to rounding errors
    • 1.7.2.1.2. Possible sources of science bugs and maintainability problems
      • 1.7.2.1.2.1. Block of code is copy & pasted, or effectively duplicated
      • 1.7.2.1.2.2. Variable is missing from the restart file, or is on the restart file when it doesn't need to be
      • 1.7.2.1.2.3. Variable is used on right-hand side of an equation before it has its "final" value
    • 1.7.2.1.3. Possible sources of answer changes with changing processor count (PEM test failures, also seen in ERP tests)
      • 1.7.2.1.3.1. Problems specific to parallelization
        • 1.7.2.1.3.1.1. Missing or incorrect broadcast for a namelist variable
        • 1.7.2.1.3.1.2. Processor count dependence of a parallel algorithm
      • 1.7.2.1.3.2. Incorrect indexing of a subgrid variable
      • 1.7.2.1.3.3. Scalar variable used before it is set in a given loop iteration
        • 1.7.2.1.3.3.1. Missing setting of c, l or g in a loop
        • 1.7.2.1.3.3.2. Scalar variable set in a conditional but accessed outside that conditional
    • 1.7.2.1.4. Possible sources of threading bugs
      • 1.7.2.1.4.1. Whole-array assignment
      • 1.7.2.1.4.2. Forgetting to index into a variable for assignment
      • 1.7.2.1.4.3. Improper argument passing
      • 1.7.2.1.4.4. Identify specific loops with issues by turning off threading for loops
      • 1.7.2.1.4.5. Incorrect list of private variables in threaded loops
    • 1.7.2.1.5. Possible sources of performance problems
      • 1.7.2.1.5.1. Doing an operation on all array elements rather than just active points
      • 1.7.2.1.5.2. Nested loops in the wrong order for cache-friendliness and/or vectorizability
    • 1.7.2.1.6. Other things to look for in a PR review
      • 1.7.2.1.6.1. Binary files committed without the use of git lfs
  • 1.7.2.2. A specific troubleshooting example
  • 1.7.2.3. Use the point_of_interest module
  • 1.7.2.4. Run with a smaller set of processors
  • 1.7.2.5. Run in serial mode with a single processor
  • 1.7.2.6. Run at a lower resolution
  • 1.7.2.7. Run a simpler case
  • 1.7.2.8. Run with a debugger