Resolve conflicts with openEO (e.g. status)

[m-mohr]

It looks like in #292 that the next version will be a breaking release, i.e. 2.0.

This might be a chance to resolve various hard conflicts between OGC API - Processes and openEO that we face in the GDC API work.

With hard conflicts I mean things that can't be resolved easily. For example, if a link relation type is A in openEO and B in OGC API - Processes that's not a hard conflict because you could just provide both link relation types in separate links. If there's a field X in a response which is an enum of 1 and 2 as values but in openEO it is an enum of A and B, it is a hard conflict.

The crosswalk at Open-EO/openeo-api#494 gives some hints at conflicts.

The only obvious conflict is that in GET /jobs and GET /jobs/{jobID} the status field conflicts:

OAP 1.0	openEO
accepted	created
n/a	queued
running	running
successful	finished
failed	error
dismissed	canceled

Changing the allowed enum values to created, running, finished, error and cancelled would resolve the conflict.

I'm not sure whether this is something the SWG wants to cater for, but I thought I'd bring it up as it seems the SWG is going for a breaking release anyway. It would help align both standards though. For alignment, changes on both sides would be required and I hope there's some willingness to do so from both sides. openEO is also looking into further alignment, especially around relation types and additional fields that openEO doesn't have yet such as jobControlOptions.

[fmigneault]

I think we must also consider that accepted, running, succeeded (not successful), failed, dismissed were used in previous WPS standard. It makes it easier for users to migrate from WPS to OGC API. Even if OAP "could" be breaking by going from 1.0 to 2.0, I think we want to keep things as closely as possible to previous iterations, given that many industries have not even yet completely adopted OAP 1.0.

Regardless of the choice, that would mean that most implementations would have to handle all variations simultaneously (including openEO's ones if supported by the service), since many implementers want to preserve backward compatibility.

[m-mohr]

It seems like a rather weak argument given that succeeded/successful is not backward compatible... 🤔

Maybe the more general question to ask is whether we (OGC API - Processes / openEO) aim at getting compatible at some point.

If yes, we need to start somewhere. This would be a good start.
If such a change like this seeams unfeasible though, it seems that the answer should be no as then we'll always hit the backward-compatibility reason in both specifications and we'll likely never align.
I no, we should clearly state that somewhere and focus on other things (e.g. how to execute each other's workflows, guide implementors towards the right specification depending on the use case, etc.).

[jerstlouis]

I second @fmigneault 's comment on keeping any future OGC API - Processes iteration to break as little as possible. From my understanding, so far with the current draft 1.1/2.0, most 1.0 clients would still be able to execute processes from 2.0 servers without modifications in many (most?) cases.

focus on other things (e.g. how to execute each other's workflows

As indicated at opengeospatial/ogcapi-geodatacubes#8 (comment) ,
I believe we should focus on defining the ability to submit a workflow at a new end-point (regardless of whether it is defined as an openEO process graph or an extended Processes execution request) for ad-hoc execution, with at least a "Collection Output" execution mode, but perhaps also a sync/async execution mode working very similarly to OGC API - Processes.

This should enable integrating openEO and OGC API - Processes workflows into the other, if each implementation also supports submitting a workflow intended for the other which results in a virtual GDC from which it can retrieve data as input on demand.

With Processes - Part 2, an Application Package could also be used to deploy an openEO workflow as an OGC API - Processes at a /processes/{processId}.

[m-mohr]

Okay. It's probably made obsolete by the OGC EPC anyway?!

[gfenoy]

I agree with @fmigneault about the backward compatibility with WPS.

But would the list below be acceptable?

1. created (POST on /jobs the status would reflect the fact that the job entity was created)
2. accepted (POST on /jobs/{jobId}, in case the server accepts to handle the job execution)
3. queued (GET on /jobs/{jobId}, in case the server has put the job in a queue for later processing)
4. running (GET on /jobs/{jobId}, when the job is running)
5. succeeded (GET on /jobs/{jobId}, the job has completed its execution)
6. failed (GET on /jobs/{jobId}, the job has failed)
7. dismissed (DELETE on /jobs/{jobId}, the job has successfully been removed)

We add two status values (1 and 3) to the one currently available in OGC API - Processes - Part 1: Core Standard (not defined in the WPS Standards) and rename successful to succeeded (as defined in WPS Standards). I am unsure if we should have both created and accepted or keep accepted only, but I am open to adding new values.

This way, on the 6 status codes defined in openeo, we would have 7 in OGC API - Processes - Part 1: Core with three similar to openeo (rather than only one at the beginning). Having openeo changing three values (5, 6 and 7) would solve the issue. The same works for changing these three values in OGC API - Processes - Part 1: Core, but it leads to losing the initial values defined in the WPS Standard.

[pvretano]

@gfenoy this would not be acceptable. Well, at least any part that does agree with what is currently documented in the OAPIP Part1 standard.

Part 1 and Part 2 were almost ready for submission to the OAB for review and then RFC. These changes would completely derail this schedule and before we do that we need to discuss in the SWG and perhaps do a code sprint or two to validate/verify the approach.

I think that a lot of these requirements are arising from TB20 ... yes? If this is the case then, at the very least, you should game out the approach in TB20, write an engineering report about it and then feed that back to the SWG for consideration.

[fmigneault]

My take on the acceptable points:

1. Yes
2. • Yes, provided that "something" was explicitly indicated in (1) to wait for execution (status=pending)
   • Otherwise (default/nothing provided), forbidden (or other HTTP code) to indicate the Job is already "submitted"
3. No (as in not "queued" necessarily). Just the usual detail of the job, regardless of its execution status. I don't think there is a need for a distinction of the status when querying that endpoint.
4. See (3)
5. See (3)
6. See (3)
7. Not "removed", but dismissed. Currently, this is left up to the implementation to decide if dismissing includes removal of the artifacts and the job itself as well, or only "stop" execution (as applicable) and leave the job definition available (eg: it could be auto-cleaned up later by the server).

rename successful to succeeded (as defined in WPS Standards)

Both are handled on our end, so that does not matter for us.
However, I would consider this a breaking change for any other existing implementations.

have both created and accepted or keep accepted only

IMO, created is not needed. When the execution is accepted in async, the actual "Job" object could exist (or not for that matter) in the backend. Whether it exists or not is determined by GET /jobs/{jobId}. If 404, it is not created yet, 200 it was created. Therefore, created is redundant. However, accepted makes sense since, although the "Job" could be created (200 returned), it could still be waiting for resources to start, hence the accepted.

I think that, if a pending status is introduced for POST /jobs with a flag for "wait execution", the HTTP status could be 201 Created to distinguish from the 202 Accepted. However, the status should probably still indicate pending even if 201 was returned, since created could cause confusion (is the job executing if it is in created state? Normally no since that is represented by running).

Part 1 and Part 2 were almost ready for submission to the OAB for review and then RFC. These changes would completely derail this schedule and before we do that we need to discuss in the SWG and perhaps do a code sprint or two to validate/verify the approach.

Indeed. This should be stabilized before reshuffling definitions. There should be an intermediate milestone/version release beforehand, so that users that want only what is in the current main branch can use it without opt-in proposed POST /jobs (or whatever alternative).

arising from TB20

Yes, but not limited to GDC requirements IMO.
It could provide strong arguments about recommendations for future updates of the standard.

[m-mohr]

Honestly, I've given up on aligning openEO and OAP. With what I hear there's no way we'll ever be compatible. So if someone wants to implement both specs they are required to keep the endpoint trees separate. We can probably execute each others workflows but, I guess, that's it.

I think that a lot of these requirements are arising from TB20 ... yes? If this is the case then, at the very least, you should game out the approach in TB20, write an engineering report about it and then feed that back to the SWG for consideration.

No, it's originating from TB19 and the ER is available online. This issue is an outcome of that and here it is for SWG consideration.

[fmigneault]

Following #437 (comment)

The following is a more appropriate mapping for OAP Part 1: Core:

OAP 1.0	openEO
n/a	created
accepted	queued

With OAP Part 4: Job Management, created would be used as well.

[m-mohr]

See also #437 (comment)

I thought we agreed on a common set of status codes, not a mapping.