Undesirable behavior in the execution of the Show built-in

[dianacgr]

I have digged a bit into the behavior reported in (#100 ). I managed to get deeper and confine the problem. Right now my hypothesis is that the bug is not related to the detection of the space stability but to the execution of the Show built-in. Consider the following piece of code that should show failed but prints succeeded instead.

proc{Foo R}
   {Show true}
   fail
end
S = {Space.new Foo}
{Show {Space.ask S}}

I added the following print statements inside the VM implementation:

• When the space is created: https://github.com/dianacgr/mozart2/blob/practica/vm/vm/main/modules/modspace.hh#L65
• When the execution of the thread gets into MOZART_TRY: https://github.com/dianacgr/mozart2/blob/practica/vm/vm/main/emulate.cc#L359
• When the thread execution falls into MOZART_CATCH: https://github.com/dianacgr/mozart2/blob/practica/vm/vm/main/emulate.cc#L1235
• On the execution of the sendMsgG instruction: https://github.com/dianacgr/mozart2/blob/practica/vm/vm/main/emulate.cc#L649
• When Space.ask is executed: https://github.com/dianacgr/mozart2/blob/practica/vm/vm/main/reifiedspace.hh#L129

The execution of the program above leads to the following output:

Output explanation:

• Text between dotted lines represents an execution of the thread creating the space (i.e. executing Foo).
• 0x7fe88ae95188 is the thread executing Foo. This thread starts runnable (second line after the first dotted line) as expected. However after the first execution it is suspended (last line before the second dotted line).
• sendMsg is the instruction generated by the compiler to implement the Show statement. GPC(1) should be a node with Show the built-in procedure, however in the first execution it is a transient (sixth line of the first execution block). This is the reason for the execution to get into MOZART_CATCH and when things start to get really weird to us.
• The second execution block reflects the execution of the “main” thread at the toplevel space. As you can see this thread finds statusvar to not be a transient (11th line in the second execution block). A consequence of the suspended thread.
• In the third execution block GPC(1) is no longer a transient. Moreover, this is what we expect the execution to be in the first place. However, space stability was already (wrongly) detected. Notice that at this time GPC(1) is a procedure <P/1 Show> (lines 6 and 7 in the third block).

Conclusions/ Guesses:

Our guesses: at some point Show is not “ready” and hence the VM suspends the thread. The problem now is to know why it was not ready. To continue the hunt we would like to know if any of you share the same guess or have another clue about what can be producing this behavior.

[sjrd]

It is normal for Show not to be "ready", i.e., be transient, until it is first requested. This is how lazy values in Oz work, and functions imported from other modules are lazy. The Show function is actually not directly a builtin. It is defined here:
https://github.com/mozart/mozart2/blob/master/lib/main/sys/System.oz#L57-L59

You can force it to be ready before you start creating any space with {Wait Show}.

[ggutierrez]

On Wed, Dec 3, 2014 at 3:28 PM, Sébastien Doeraene <[email protected]

wrote:

It is normal for Show not to be "ready", i.e., be transient, until it is
first requested. This is how lazy values in Oz work, and functions imported
from other modules are lazy.

Is this design decision present in 1.4.* implementation or was it
introduced in 2.0?.

The Show function is actually not directly a builtin. It is defined here:

https://github.com/mozart/mozart2/blob/master/lib/main/sys/System.oz#L57-L59

You can force it to be ready before you start creating any space with {Wait
Show}.

I agree that the Wait statement will force it to be ready. However, I am
not that sure tu understand your explanation. Let me present my point with
the following program

declare
X = 1
thread
{Show X}
end

Now let us suppose that Show is lazy as you said and that is only "ready"
in the actual execution of the line {Show X} or the corresponding VM
instruction. If Show is not "ready" then the VM will suspend the execution
of the thread until it becomes bound and then will re-schedule it?.

[sjrd]

Is this design decision present in 1.4.* implementation or was it introduced in 2.0?.

It was like that in 1.4.x as well.

If Show is not "ready" then the VM will suspend the execution of the thread until it becomes bound and then will re-schedule it?.

That's it. It is just like any {ByNeedFuture} variable.

[ggutierrez]

On Dec 3, 2014, at 6:13 PM, Sébastien Doeraene [email protected] wrote:

Is this design decision present in 1.4.* implementation or was it introduced in 2.0?.

It was like that in 1.4.x as well.

We have tried the following program and it does behaves as expected:

declare
{Wait Show}
proc{Foo R}
{Show true}
fail
end
S = {Space.new Foo}
{Show {Space.ask S}}

The only modification is in the second line: {Wait Show}. The fact that it behaves as expected (i.e. shows failed) is fine. However we still have some questions:

1. Why the Wait statement is not required in the previous version?
2. Do we have to add a Wait statement for every built-in inside Foo just to get the right execution?

If Show is not "ready" then the VM will suspend the execution of the thread until it becomes bound and then will re-schedule it?.

That's it. It is just like any {ByNeedFuture} variable.

We think that loading the modules when they are required is a wise decision. However, doing it using ByNeedFuture does not sound right to us. Take into account for instance the System module. Every built-in in that module is “defined” and hence “ready”. In consequence, when System.show is used it has to be executed immediately by the VM. Or are we missing something?=

[sjrd]

1. Why the Wait statement is not required in the previous version?

Probably because System was a built-in module in the previous version, which was exposed directly at the user level. This was basically an implementation artifact surfacing at the user level, which was not a good idea in theory. But never mind the theory: the practice in Mozart 2 was to implement as much as possible in Oz (as long as it did not cause gigantic performance problems, see Pickle ^^), which means that some functions in System are written in Oz, on top of other built-in functions. The real builtin BootSystem is hidden.

A builtin module (boot module) can be constant-folded by the compiler, causing a direct reference to builtin functions to be stored in the code, which makes Show immediately available, because it was basically a literal constant.

User modules need to be lazily loaded.

2. Do we have to add a Wait statement for every built-in inside Foo just to get the right execution?

I don't know. But it seems plausible (except for true builtins - Show is not a builtin, . is).

However, doing it using ByNeedFuture does not sound right to us.

It is the only reasonable thing to do. How would you do it otherwise?

Every built-in in that module is “defined” and hence “ready”.

Again, System is not a builtin module. It is written in Oz. So that statement does not hold.

[ggutierrez]

On Dec 12, 2014, at 6:48 PM, Sébastien Doeraene <[email protected] mailto:[email protected]> wrote:

1. Why the Wait statement is not required in the previous version?

Probably because System was a built-in module in the previous version, which was exposed directly at the user level. This was basically an implementation artifact surfacing at the user level, which was not a good idea in theory. But never mind the theory: the practice in Mozart 2 was to implement as much as possible in Oz (as long as it did not cause gigantic performance problems, see Pickle ^^), which means that some functions in System are written in Oz, on top of other built-in functions. The real builtin BootSystem is hidden.

I see. Probably at the time that constraint programming was implemented in the original mozart was safe to rely on such artifacts. The problem that we face now is that aspects like stability detection relies on those “artefacts” and is easy to get an undesirable behavior just by running old code on the new system. At least from the constraint programming standpoint. I would like to know your opinion on how to tackle this problem. In my opinion the first thing to do is to warn the user when situations like this occurs. For instance, I would rise an exception with an appropriate message. I think an exception is far better than the current behavior. Is it possible to detect this situation in a consistent way?. I have thought about:

1. Detect if the running thread is in a space different from the toplevel space.
2. Detect if there is any other thread waiting on the status variable of the thread’s space.
3. Throw an exception if the execution of the thread suspends because it has to wait for a procedure to be available.

A builtin module (boot module) can be constant-folded by the compiler, causing a direct reference to builtin functions to be stored in the code, which makes Show immediately available, because it was basically a literal constant.

Now it makes sense.
User modules need to be lazily loaded.

2. Do we have to add a Wait statement for every built-in inside Foo just to get the right execution?

I don't know. But it seems plausible (except for true builtins - Show is not a builtin, . is).

This will be difficult and will lead to ugly code, in my opinion. Moreover, the code would not be a direct consequence of the problem constraint being solved but just to work around an old design decision.

I think the first step is to detect the problem and hence my idea of throwing an exception. This will ensure program termination in a consistent way. This is not only Show dependent but the idea is to catch the problem for any other user module (or better: Oz defined procedure that is lazily loaded).

Another way is to change the way in which space stability is detected. However this will require more changes at both implementation and design levels. This is why I hesitate to go in this way but still a possibility.

[sjrd]

Before going to all these things, could you check what happens in Mozart 1.x if you use a simulated lazy Show instead of the builtin. You can do so by declaring

LazyShow = {ByNeedFuture fun {$} Show end}

and use LazyShow instead of Show in the space.

How does Mozart 1 behaves in that case? Does it detect stability as well?

[ggutierrez]

On Dec 15, 2014, at 10:26 AM, Sébastien Doeraene [email protected] wrote:

Before going to all these things, could you check what happens in Mozart 1.x if you use a simulated lazy Show instead of the builtin. You can do so by declaring

LazyShow = {ByNeedFuture fun {$} Show end}
and use LazyShow instead of Show in the space.

How does Mozart 1 behaves in that case? Does it detect stability as well?

It behaves correctly. It does detect stability and shows failed as expected.

Do you have any clue on where to start looking for the problem?

[sjrd]

So I suppose Mozart 1 did something funny to not consider a space stable if it waits for something like a ReadOnly (aka Future). Which is completely anti-semantic (since ReadOnlys are supposed to be only an optimization over normal unbound variables). That's where I would look for.

[ggutierrez]

On Dec 15, 2014, at 4:48 PM, Sébastien Doeraene [email protected] wrote:

So I suppose Mozart 1 did something funny to not consider a space stable if it waits for something like a ReadOnly (aka Future). Which is completely anti-semantic (since ReadOnlys are supposed to be only an optimization over normal unbound variables). That's where I would look for.

Not sure to understand, but in the current 2.0 implementation the space IS considered stable. That is actually the mistake that leads to the false-positive in stability deduction. Actually this is why the space is considered succeeded earlier than it should be.=