MLMG: Iteration interrupted with resid/bnorm = 0? #3834

solarmaterial · 2024-03-21T16:07:31Z

solarmaterial
Mar 21, 2024

Hey AMReX team,

I used HeatEquation_EX3_C as a stencil to solve non-linear heat equation (e.g. $(A\alpha - B\nabla \cdot \beta (\phi^{n+1} ) \nabla)\phi^{n+1} = \phi^n$.

I add a new iteration in advance.cpp to make sure that the residual between the two $\phi^{n+1}$ in the left hand side of the equation meets my requirements of accuracy(maxnorm_phi_res = $\left|{\phi_{s+1}^{n+1}- \phi_{s}^{n+1}}\right|<10^{-6}$).
However, something went wrong when the MLMG calculates the $\phi^{n+1}$ after a few time steps.
At the begining, the output of the MLMG is below, and seemed to be correct.

......
Advanced step 3
MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 8
MLMG: Initial rhs               = 4.593323727
MLMG: Initial residual (resid0) = 4.593323727
MLMG: Iteration   1 Fine resid/bnorm = 0.09932378164
MLMG: Iteration   2 Fine resid/bnorm = 0.005557900331
......
MLMG: Iteration   8 Fine resid/bnorm = 4.711403199e-10
MLMG: Iteration   9 Fine resid/bnorm = 5.775111876e-11
MLMG: Final Iter. 9 resid, resid/bnorm = 2.652695841e-10, 5.775111876e-11
MLMG: Timers: Solve = 0.000923256 Iter = 0.000763125 Bottom = 7.861e-05
maxnorm_phi_res = 4.593323727
MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 8
MLMG: Initial rhs               = 4.593323727
MLMG: Initial residual (resid0) = 0.02081921324
MLMG: Iteration   1 Fine resid/bnorm = 4.910990327e-05
MLMG: Iteration   2 Fine resid/bnorm = 1.711277226e-06
......
MLMG: Iteration   6 Fine resid/bnorm = 1.667936817e-10
MLMG: Iteration   7 Fine resid/bnorm = 1.726730742e-11
MLMG: Final Iter. 7 resid, resid/bnorm = 7.931433288e-11, 1.726730742e-11
MLMG: Timers: Solve = 0.000533002 Iter = 0.000454154 Bottom = 3.9444e-05
maxnorm_phi_res = 0.01586238751

......

MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 8
MLMG: Initial rhs               = 4.593323727
MLMG: Initial residual (resid0) = 2.775631347e-07
MLMG: Iteration   1 Fine resid/bnorm = 8.342856204e-10
MLMG: Iteration   2 Fine resid/bnorm = 3.546062979e-11
MLMG: Final Iter. 2 resid, resid/bnorm = 1.628821522e-10, 3.546062979e-11
MLMG: Timers: Solve = 0.000341422 Iter = 0.000221046 Bottom = 2.1691e-05
maxnorm_phi_res = 1.589532199e-07
fine_phi_res = 1.589532199e-07
Advanced step 4

But in the next time step, the output became weird,

Advanced step 4
MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 8
MLMG: Initial rhs               = 4.573800836
MLMG: Initial residual (resid0) = 4.573800836
MLMG: Iteration   1 Fine resid/bnorm = 0
MLMG: Final Iter. 1 resid, resid/bnorm = 0, 0
MLMG: Timers: Solve = 0.000356591 Iter = 0.000156444 Bottom = 1.3495e-05
maxnorm_phi_res = 4.573800836
MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 8
MLMG: Initial rhs               = 4.573800836
MLMG: Initial residual (resid0) = 0
MLMG: No iterations needed
MLMG: Timers: Solve = 0.000137328 Iter = 0 Bottom = 0
maxnorm_phi_res = 0
fine_phi_res = 0
Advanced step 5

In Iteration 1 , the resid/bnorm becomes 0 and thus no further iterations are executed. The calculations of the following time steps thus all went wrong. I'm wondering how this happens, while steps 1 to 3 are executed normally. I have tried to change maxnorm_phi_res to $10^{-10}$ or something else, and Real tol_rel to 1.e-14; or something else, the calculation may go on a few more time steps but finally will come to the same situation as in Advanced step 4. This issue bothers me a lot, and I'm really looking forward to your reply! It means a lot to me!

Yours sincerely,
solar

Answered by WeiqunZhang

Mar 25, 2024

You need to give the solver an initial guess.

diff --git a/HeatEquation_EX3_C/Source/advance.cpp b/HeatEquation_EX3_C/Source/advance.cpp
index 19c4d81..23ae9bc 100644
--- a/HeatEquation_EX3_C/Source/advance.cpp
+++ b/HeatEquation_EX3_C/Source/advance.cpp
@@ -213,6 +213,8 @@ void advance (MultiFab& phi_old,
     const Real tol_rel = 1.e-10;
     const Real tol_abs = 0.0;
 
+    phi_iter_solve.setVal(0.0);
+
     // Solve linear system
     mlmg.solve({&phi_iter_solve}, {&phi_old}, tol_rel, tol_abs);//phi_old remain unchanged during the non-linear iteration

https://amrex-codes.github.io/amrex/docs_html/LinearSolvers.html?highlight=guess

to solve the problem given an initial guess and a …

View full answer

WeiqunZhang · 2024-03-21T16:48:16Z

WeiqunZhang
Mar 21, 2024
Maintainer

Could you turn on the bottom solver verbosity by calling mlmg.setBottomVerbose(2)? Could you also build with DEBUG=TURE (assuming gnu make is used) and run with amrex.fpe_trap_invalid=1 amrex.fpe_trap_zero=1 amrex.fpe_trap_overflow=1?

7 replies

WeiqunZhang Mar 21, 2024
Maintainer

It might be the rhs passed to mlmg has NaNs.

solarmaterial Mar 21, 2024
Author

Thank you so much again! I'll check the multifab of the rhs.

solarmaterial Mar 23, 2024
Author

Hey AMReX team, I‘ve checked the multifab of the rhs before the broken calculation . The rhs passed to MLMG is the multifab that came from the previous of the time step(phi_old), and it was well calculated(because the previous time step seemed to have been correctly iterated by MLMG), so I'm quite sure that the data has no NaNs (but there are some numbers quite close to -inf, like 3.1213314078052337e-290, is that matter?). I've also checked the $\beta(\phi^{n+1})$ of the left hand side of the equation, and they are still numbers before the next broken advanced time step, sadly...So I come here and ask for your advice again...Is there anything else I can do to address this issue? I’m really looking forward to your reply!!

WeiqunZhang Mar 23, 2024
Maintainer

-inf would certainly be a problem. But the example you gave is close to 0, not -inf.

solarmaterial Mar 24, 2024
Author

ops, my oversight. yeah that is not -inf but 0. Thank you for your reply! I'm still checking the multifab to find the NaNs now, hoping I can catch them...

zingale · 2024-03-23T17:45:10Z

zingale
Mar 23, 2024
Collaborator

can you try running with amrex.fpe_trap_invalid=1 -- this will abort it NaNs are generated and give you a backtrace telling you where. It might also help to compile with DEBUG=TRUE or TEST=TRUE.

1 reply

solarmaterial Mar 24, 2024
Author

Thank you for your reply, I really appreciate it! Now I'm trying to figure out where the NaNs hide through this way. Many thanks!

solarmaterial · 2024-03-24T11:40:44Z

solarmaterial
Mar 24, 2024
Author

Many thanks for your reply!! Here I set int bottom_verbose = 2; mlmg.setBottomVerbose(bottom_verbose); in advance.cpp, and DEBUG = TURE in GNUmakefile, also run with ./main1d.gnu.ex inputs_1d amrex.fpe_trap_invalid=1 amrex.fpe_trap_zero=1 amrex.fpe_trap_overflow=1 and some of the output are below:

......
Advanced step 7
MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 8
MLMG: Initial rhs               = 4.517595429
MLMG: Initial residual (resid0) = 4.517595429
MLCGSolver_BiCGStab: Initial error (error0) =        6.490679779e-23
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 3.138539586e-07
MLMG: Iteration   1 Fine resid/resid0 = 0.09735831857
MLCGSolver_BiCGStab: Initial error (error0) =        1.992170189e-24
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 3.138539586e-07
MLMG: Iteration   2 Fine resid/resid0 = 0.005288856293
MLCGSolver_BiCGStab: Initial error (error0) =        1.557418222e-26
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 3.138539586e-07
MLMG: Iteration   3 Fine resid/resid0 = 0.0003019520394
MLCGSolver_BiCGStab: Initial error (error0) =        2.312622901e-27
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 3.138539586e-07
MLMG: Iteration   4 Fine resid/resid0 = 1.459452108e-05
MLCGSolver_BiCGStab: Initial error (error0) =        9.692635265e-29
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 3.138539586e-07
MLMG: Iteration   5 Fine resid/resid0 = 7.110863756e-07
MLCGSolver_BiCGStab: Initial error (error0) =        4.118718602e-30
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 3.138539586e-07
MLMG: Iteration   6 Fine resid/resid0 = 6.323693757e-08
MLCGSolver_BiCGStab: Initial error (error0) =        4.782343735e-31
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 3.138539586e-07
MLMG: Iteration   7 Fine resid/resid0 = 3.986332055e-09
MLCGSolver_BiCGStab: Initial error (error0) =        4.774063705e-33
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 3.138539586e-07
MLMG: Iteration   8 Fine resid/resid0 = 3.601848673e-10
MLCGSolver_BiCGStab: Initial error (error0) =        2.724992463e-33
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 3.138539586e-07
MLMG: Iteration   9 Fine resid/resid0 = 4.087795375e-11
MLCGSolver_BiCGStab: Initial error (error0) =        5.60364527e-35
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 3.138539586e-07
MLMG: Iteration  10 Fine resid/resid0 = 5.506688181e-12
MLCGSolver_BiCGStab: Initial error (error0) =        1.813076116e-35
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 3.138539586e-07
MLMG: Iteration  11 Fine resid/resid0 = 6.49777016e-13
MLMG: Final Iter. 11 resid, resid/resid0 = 2.935429677e-12, 6.49777016e-13
MLMG: Timers: Solve = 0.000980062 Iter = 0.000887718 Bottom = 0.000239599
maxnorm_phi_res = 4.517595429
MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 8
MLMG: Initial rhs               = 4.517595429
MLMG: Initial residual (resid0) = 0.0216081078
MLCGSolver_BiCGStab: Initial error (error0) =        1.786240951e-29
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.725493976e-08
MLMG: Iteration   1 Fine resid/bnorm = 4.292920308e-05
MLCGSolver_BiCGStab: Initial error (error0) =        9.443526696e-31
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.725493976e-08
MLMG: Iteration   2 Fine resid/bnorm = 1.445079045e-06
MLCGSolver_BiCGStab: Initial error (error0) =        3.119677068e-31
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.725493976e-08
MLMG: Iteration   3 Fine resid/bnorm = 9.742235379e-08
MLCGSolver_BiCGStab: Initial error (error0) =        2.430863024e-32
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.725493976e-08
MLMG: Iteration   4 Fine resid/bnorm = 1.14031828e-08
MLCGSolver_BiCGStab: Initial error (error0) =        8.731260419e-34
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.725493976e-08
MLMG: Iteration   5 Fine resid/bnorm = 7.642826681e-10
MLCGSolver_BiCGStab: Initial error (error0) =        1.657638489e-35
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.725493976e-08
MLMG: Iteration   6 Fine resid/bnorm = 1.24931575e-10
MLCGSolver_BiCGStab: Initial error (error0) =        2.717552897e-36
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.725493977e-08
MLMG: Iteration   7 Fine resid/bnorm = 1.330460219e-11
MLCGSolver_BiCGStab: Initial error (error0) =        9.750431825e-38
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.725493976e-08
MLMG: Iteration   8 Fine resid/bnorm = 1.954442758e-12
MLCGSolver_BiCGStab: Initial error (error0) =        3.110140087e-38
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.725493976e-08
MLMG: Iteration   9 Fine resid/bnorm = 2.610904318e-13
MLMG: Final Iter. 9 resid, resid/bnorm = 1.179500941e-12, 2.610904318e-13
MLMG: Timers: Solve = 0.000771911 Iter = 0.000706318 Bottom = 0.000184086
maxnorm_phi_res = 0.01636835945
......
MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 8
MLMG: Initial rhs               = 4.517595429
MLMG: Initial residual (resid0) = 8.755140871e-06
MLCGSolver_BiCGStab: Initial error (error0) =        3.898018091e-34
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.726447364e-08
MLMG: Iteration   1 Fine resid/bnorm = 2.213865013e-08
MLCGSolver_BiCGStab: Initial error (error0) =        5.326619312e-35
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.726447364e-08
MLMG: Iteration   2 Fine resid/bnorm = 9.234946501e-10
MLCGSolver_BiCGStab: Initial error (error0) =        9.908983978e-36
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.726447364e-08
MLMG: Iteration   3 Fine resid/bnorm = 6.145622474e-11
MLCGSolver_BiCGStab: Initial error (error0) =        6.868223097e-37
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.726447364e-08
MLMG: Iteration   4 Fine resid/bnorm = 4.936044372e-12
MLCGSolver_BiCGStab: Initial error (error0) =        6.377954612e-39
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.726447364e-08
MLMG: Iteration   5 Fine resid/bnorm = 4.215194923e-13
MLMG: Final Iter. 5 resid, resid/bnorm = 1.904254532e-12, 4.215194923e-13
MLMG: Timers: Solve = 0.000453702 Iter = 0.000390654 Bottom = 0.000104827
maxnorm_phi_res = 6.356800182e-06
MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 8
MLMG: Initial rhs               = 4.517595429
MLMG: Initial residual (resid0) = 1.423944203e-06
MLCGSolver_BiCGStab: Initial error (error0) =        6.313982452e-36
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.726447364e-08
MLMG: Iteration   1 Fine resid/bnorm = 3.437582979e-09
MLCGSolver_BiCGStab: Initial error (error0) =        4.924656618e-36
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.726447364e-08
MLMG: Iteration   2 Fine resid/bnorm = 1.541962146e-10
MLCGSolver_BiCGStab: Initial error (error0) =        1.591616907e-36
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.726447364e-08
MLMG: Iteration   3 Fine resid/bnorm = 9.380774747e-12
MLCGSolver_BiCGStab: Initial error (error0) =        2.816041278e-37
MLCGSolver_BiCGStab: Final: Iteration    1 rel. err. 7.726447364e-08
MLMG: Iteration   4 Fine resid/bnorm = 7.706886241e-13
MLMG: Final Iter. 4 resid, resid/bnorm = 3.481659405e-12, 7.706886241e-13
MLMG: Timers: Solve = 0.0003779 Iter = 0.000317316 Bottom = 8.2224e-05
maxnorm_phi_res = 8.352754076e-07
fine_phi_res = 8.352754076e-07
Advanced step 8
MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 8
Erroneous arithmetic operation
See Backtrace.0 file for details

I'm very upset because I still don't quite understand where the problem lies. Still looking forward to your advise!

I've made a big mistake...in the previous reply, when I turn DEBUG=TRUE, I typed 'TRUE' as 'TURE', and then I ran with main1d.gnu.ex not main1d.gnu.DEBUG.ex . After I successfully compile with DEBUG=TRUE, the calculation broken at the beginning. The output above was not printed by the main1d.gnu.DEBUG.ex but main1d.gnu.ex. So the problem may not be the NaNs? Since the initial residual is given 0 with DEBUG=TRUE. The output printed by main1d.gnu.DEBUG.ex is as follows

./main1d.gnu.DEBUG.ex inputs_1d
AMReX (23.05) initialized
MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 8
MLMG: Initial rhs               = 4.654473497
MLMG: Initial residual (resid0) = 0
MLMG: No iterations needed
MLMG: Timers: Solve = 0.000941939 Iter = 0 Bottom = 0
maxnorm_phi_res = 0
fine_phi_res = 0
Advanced step 1

Still looking forward to your advice! I'm really, really appreciate it!!

4 replies

WeiqunZhang Mar 24, 2024
Maintainer

(0) Run make clean
(1) Compile with DEBUG=TRUE
(2) Run with ./main1d.gnu.DEBUG.ex inputs_1d amrex.fpe_trap_invalid=1 amrex.fpe_trap_zero=1 amrex.fpe_trap_overflow=1
(3) Examine Backtrace.0

solarmaterial Mar 25, 2024
Author

Yes, that's what I did. I've shown the Backtrace.0 in the #3843 (comment) , and I 've also uploaded the code to my github repository amrex. Thank you for your help!!

WeiqunZhang Mar 25, 2024
Maintainer

You need to give the solver an initial guess.

diff --git a/HeatEquation_EX3_C/Source/advance.cpp b/HeatEquation_EX3_C/Source/advance.cpp
index 19c4d81..23ae9bc 100644
--- a/HeatEquation_EX3_C/Source/advance.cpp
+++ b/HeatEquation_EX3_C/Source/advance.cpp
@@ -213,6 +213,8 @@ void advance (MultiFab& phi_old,
     const Real tol_rel = 1.e-10;
     const Real tol_abs = 0.0;
 
+    phi_iter_solve.setVal(0.0);
+
     // Solve linear system
     mlmg.solve({&phi_iter_solve}, {&phi_old}, tol_rel, tol_abs);//phi_old remain unchanged during the non-linear iteration

https://amrex-codes.github.io/amrex/docs_html/LinearSolvers.html?highlight=guess

to solve the problem given an initial guess and a right-hand side. Zero is a perfectly fine initial guess.

Answer selected by solarmaterial

solarmaterial Mar 25, 2024
Author

It works!!!!!!!! Thank you so much for your help for these days!!! Really!! THANKS!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MLMG: Iteration interrupted with resid/bnorm = 0? #3834

{{title}}

Replies: 3 comments 12 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

MLMG: Iteration interrupted with resid/bnorm = 0? #3834

solarmaterial Mar 21, 2024

Replies: 3 comments · 12 replies

WeiqunZhang Mar 21, 2024 Maintainer

WeiqunZhang Mar 21, 2024 Maintainer

solarmaterial Mar 21, 2024 Author

solarmaterial Mar 23, 2024 Author

WeiqunZhang Mar 23, 2024 Maintainer

solarmaterial Mar 24, 2024 Author

zingale Mar 23, 2024 Collaborator

solarmaterial Mar 24, 2024 Author

solarmaterial Mar 24, 2024 Author

WeiqunZhang Mar 24, 2024 Maintainer

solarmaterial Mar 25, 2024 Author

WeiqunZhang Mar 25, 2024 Maintainer

solarmaterial Mar 25, 2024 Author

solarmaterial
Mar 21, 2024

Replies: 3 comments 12 replies

WeiqunZhang
Mar 21, 2024
Maintainer

WeiqunZhang Mar 21, 2024
Maintainer

solarmaterial Mar 21, 2024
Author

solarmaterial Mar 23, 2024
Author

WeiqunZhang Mar 23, 2024
Maintainer

solarmaterial Mar 24, 2024
Author

zingale
Mar 23, 2024
Collaborator

solarmaterial Mar 24, 2024
Author

solarmaterial
Mar 24, 2024
Author

WeiqunZhang Mar 24, 2024
Maintainer

solarmaterial Mar 25, 2024
Author

WeiqunZhang Mar 25, 2024
Maintainer

solarmaterial Mar 25, 2024
Author