bug with statement functions

[ecossevin]

What happened?

I have a bug in routines with statement functions : FOLH in this case.

The code has a strange behavior, in my test case there are two fortran routines, one with comment and one without. The bug is occuring in one case and not in another.
The bug seems to be due to FOLH type that is lost at some point...

I mentioned already this problem in the loki chat a few weeks ago.

I'm using the lastest version of loki with Python 3.10.12.

bug_statement.tar.gz

I have other test cases, that are bigger. And i still have strange behavior : bug when i remove fortran comments or not, bug if i do a print inside python loki or not.

What are the steps to reproduce the bug?

untar the attached file
load loki venv
run python3 test.py :: change the file_name variable in test.py.

Version

latest, from today

Platform (OS and architecture)

Linux pxdesrtls020 6.8.0-47-generic #47~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Oct 2 16:16:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Relevant log output

dbg1 FOLH
Traceback (most recent call last):
  File "/home/cossevine/bug_loki/substitute/test.py", line 19, in <module>
    routine_t1 = routine.clone()
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/subroutine.py", line 293, in clone
    return super().clone(**kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/program_unit.py", line 450, in clone
    obj = super().clone(**kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/scope.py", line 343, in clone
    return type(self)(**kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/subroutine.py", line 89, in __init__
    self.__initialize__(
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/subroutine.py", line 117, in __initialize__
    super().__initialize__(
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/program_unit.py", line 96, in __initialize__
    self.rescope_symbols()
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/scope.py", line 300, in rescope_symbols
    AttachScopes().visit(self, scope=self)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 338, in visit
    return super().visit(o, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 425, in visit_Subroutine
    o.body = self.visit(o.body, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 338, in visit
    return super().visit(o, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 362, in visit_Node
    children = tuple(self.visit(i, **kwargs) for i in o.children)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 362, in <genexpr>
    children = tuple(self.visit(i, **kwargs) for i in o.children)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 338, in visit
    return super().visit(o, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 352, in visit_list
    return tuple(self.visit(c, **kwargs) for c in o)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 352, in <genexpr>
    return tuple(self.visit(c, **kwargs) for c in o)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 338, in visit
    return super().visit(o, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 362, in visit_Node
    children = tuple(self.visit(i, **kwargs) for i in o.children)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 362, in <genexpr>
    children = tuple(self.visit(i, **kwargs) for i in o.children)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 338, in visit
    return super().visit(o, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 352, in visit_list
    return tuple(self.visit(c, **kwargs) for c in o)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 352, in <genexpr>
    return tuple(self.visit(c, **kwargs) for c in o)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 338, in visit
    return super().visit(o, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 362, in visit_Node
    children = tuple(self.visit(i, **kwargs) for i in o.children)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 362, in <genexpr>
    children = tuple(self.visit(i, **kwargs) for i in o.children)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 338, in visit
    return super().visit(o, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/ir/expr_visitors.py", line 346, in visit_Expression
    return self.expr_mapper(o, scope=kwargs['scope'])
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 534, in __call__
    return super().__call__(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 141, in __call__
    result = method(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 672, in map_sum
    children = self.rec(expr.children, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 534, in __call__
    return super().__call__(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 154, in __call__
    return self.map_foreign(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 220, in map_foreign
    return self.map_tuple(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 602, in map_tuple
    children = [self.rec(child, *args, **kwargs) for child in expr]
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 602, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr]
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 534, in __call__
    return super().__call__(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 141, in __call__
    result = method(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 672, in map_sum
    children = self.rec(expr.children, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 534, in __call__
    return super().__call__(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 154, in __call__
    return self.map_foreign(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 220, in map_foreign
    return self.map_tuple(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 602, in map_tuple
    children = [self.rec(child, *args, **kwargs) for child in expr]
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 602, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr]
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 534, in __call__
    return super().__call__(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 141, in __call__
    result = method(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 678, in map_quotient
    return expr.__class__(self.rec(expr.numerator, *args, **kwargs),
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 534, in __call__
    return super().__call__(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 141, in __call__
    result = method(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 672, in map_sum
    children = self.rec(expr.children, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 534, in __call__
    return super().__call__(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 154, in __call__
    return self.map_foreign(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 220, in map_foreign
    return self.map_tuple(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 602, in map_tuple
    children = [self.rec(child, *args, **kwargs) for child in expr]
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 602, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr]
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 534, in __call__
    return super().__call__(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 141, in __call__
    result = method(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 672, in map_sum
    children = self.rec(expr.children, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 534, in __call__
    return super().__call__(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 154, in __call__
    return self.map_foreign(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 220, in map_foreign
    return self.map_tuple(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 602, in map_tuple
    children = [self.rec(child, *args, **kwargs) for child in expr]
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 602, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr]
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 534, in __call__
    return super().__call__(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 141, in __call__
    result = method(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 495, in map_call_with_kwargs
    function = self.rec(expr.function, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 534, in __call__
    return super().__call__(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/pymbolic/mapper/__init__.py", line 141, in __call__
    result = method(expr, *args, **kwargs)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/mappers.py", line 813, in map_variable_symbol
    new_expr.type = SymbolAttributes(dtype=BasicType.DEFERRED)
  File "/home/cossevine/tmp/loki_env_test/lib/python3.10/site-packages/loki/expression/symbols.py", line 243, in type
    elif _type is not self.scope.symbol_attrs.lookup(self.name):
AttributeError: 'NoneType' object has no attribute 'symbol_attrs'

Accompanying data

No response

Organisation

météo france

[mlange05]

Hi @ecossevin , thanks for reporting this. I've had a bit of trouble deciphering the actual problem form the bug description in this issue. However, I think I've managed to distill the actual bug and fix it. Please see PR #439 .

For the reproducer itself, I had the following issues (all tested on current main):

• The fcttrm.func.h file was missing
• Both, the "bug" and non-bug version of the test would fail for me on the original setup
• Once I had fixed the external header import via Sourcefile.from_file(file_name, preprocess=True, includes=['./']), both cases would succeed for me. This is due to the fact that the actual bug, would only trigger on DeferredTypeSymbol objects from the unidentified stmt funcs (FHOLM, etc.), which is the actual bug fixed in PR Sanitise: Update scope on unchanged expressions in remove_associates #439 - or so I think.

So, this is really a good example of why we recommend keeping bug reproducers as minimal as possible. Instead of using full-scale model code, distilling this into a minimal failing example (see the associated test in the PR), already highlights the problem clearly and avoids strange behaviour.

But anyway, thanks again for reporting this and please let me know if this fixes your problem. 😄

[ecossevin]

Hi @mlange05,

Thanks for your fast answer.

I would like not to have to read the fcttrm.func.h file; when I don't read this file, the bug is still there. Can you confirm that without reading (preprocess=True, includes=['./'])) the header file with the function definition the bug can't be avoid ?

As for the reproducer as small as possible, it's complicated in this case, because the error is occuring very deep in pymbolic, and there are some unpredictable behaviors (adding a print in the code can remove the bug).

Regards,
Erwan

[mlange05]

Hi, yes, with the fix in PR #439 the test runs for me with and without preprocessing the fcttrm.func.h. Have you tried the branch from the PR?

As for the unpredictable behaviour, I did not manage to reproduce this. Both files ("bug"/"no bug") behave exactly the same for me.

Regards,
M

[ecossevin]

I used the correct branch indeed. The bug has gone. When I said "the bug is still there", I was in fact mentioning an other bug. However, the bugs are very similar (same unpredictable weird behavior) as they both concern statement functions. This is why I thought it was the same bug.

Concerning that other bug, I don't have a simple reproducer. I'm going to try to have the test case smaller and open a new issue.