Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal OGC API - Processs - Part4: Job Management #437

Merged
merged 31 commits into from
Oct 18, 2024
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
717321a
Boostrap part 4 (with wrong document number)
gfenoy Aug 29, 2024
ca03fbd
Fix typos
gfenoy Aug 29, 2024
6b7acb3
Fix typos
gfenoy Aug 29, 2024
4aaf777
Fix typos
gfenoy Aug 29, 2024
8a6d469
Fix typo
gfenoy Aug 29, 2024
3281c3d
Fix files names
gfenoy Aug 29, 2024
7b7ed5d
Delete extensions/job_management/standard/20-044.xml.abort
gfenoy Aug 29, 2024
160869a
Remove the quotation and replace Recommendation by Permission
gfenoy Aug 29, 2024
c127cb7
Update statusCode.yaml to use created as mentioned in #419
gfenoy Aug 29, 2024
c40f9f3
Use in place of in statusInfo
gfenoy Sep 2, 2024
de992c9
Typo in security section
gfenoy Sep 2, 2024
18bad72
Fix Abstract test conformance class
gfenoy Sep 5, 2024
5469b35
Use PATCH method rather PUT to update the JOB
gfenoy Sep 5, 2024
dfe6472
Fix typo in front_material nad add reference to the exception when st…
gfenoy Sep 5, 2024
a3ed0c0
Fix typo in sequence diagram
gfenoy Sep 5, 2024
a36f218
reference to the update exception job locked
gfenoy Sep 5, 2024
16c0fda
Update after hearing from the GDC SWG meeting
gfenoy Sep 10, 2024
33677d8
Typo in processes-core/statusInfo.yaml in required fields
gfenoy Sep 12, 2024
ad78f24
Merge branch 'opengeospatial:master' into proposal-part4-initial
gfenoy Sep 12, 2024
df465c5
Take into acount the remarks made by @fmignault
gfenoy Sep 24, 2024
dacc370
Add @fmignault as author of the draft
gfenoy Sep 24, 2024
60c04ab
Fix typo and add paths to the clause_6
gfenoy Sep 24, 2024
166d805
Fix typos
gfenoy Sep 24, 2024
d12e0a6
Update extensions/job_management/standard/sections/annex_bibliography…
gfenoy Sep 24, 2024
3737503
Update part 1 schema to remove the limitation for the type of job
gfenoy Sep 24, 2024
80cbc5d
Add unsupported-schema requirement for job-management/create
gfenoy Sep 24, 2024
40e94ab
Update following @fmignault comments https://github.com/opengeospatia…
gfenoy Sep 26, 2024
06656a8
Update the requirements / permissions / recommendations conform to OG…
gfenoy Sep 30, 2024
05209d9
Merge branch 'opengeospatial:master' into proposal-part4-initial
gfenoy Sep 30, 2024
7b79c50
Revert to processID in the statusInfo.yaml schema
gfenoy Sep 30, 2024
efb4e8e
Update part 4 with required missing files identified by @fmigneault
gfenoy Oct 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
.DS_Store
.idea
.vscode
*.pdf
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,5 +20,5 @@ All questions regarding this submission should be directed to the editors or the
| Panagiotis (Peter) A. Vretanos _(editor)_ | CubeWerx Inc.
| Gérald Fenoy _(editor)_ | GeoLabs
| Pedro Gonçalves | Terradue Srl.
| Francis Charette Migneault | Centre de Recherche en Informatique de Montréal (CRIM)
|===

14 changes: 14 additions & 0 deletions extensions/job_management/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Standard template

This folder contains the content for the OGC API - Processes - Part 4: Job Management (JM).

This extension provides the ability to create new jobs without starting its execution. Meaning that you can prepare a job and execute it later. This extension also provides ways to track history associated with the execution, starting with the job definition.

The repo is organized as follows:

* standard - the main standard document content
- organized in multiple sections and directories (recommendations, requirements, etc.)
* xml - normative XML/XSD components specified by the standard
* examples - JSON and XML examples

The schemas associated with this extension are stored from the root directory in `openapi/*{api,parameters,path,responses,schemas}*/processes-job-management`.
1 change: 1 addition & 0 deletions extensions/job_management/examples/json/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add JSON examples to this directory. If not used, delete the directory.
1 change: 1 addition & 0 deletions extensions/job_management/examples/xml/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add XML examples to this directory. If not used, delete the directory.
56 changes: 56 additions & 0 deletions extensions/job_management/standard/24-051.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
= OGC API - Processes - Part 4: Job Management
:doctype: standard
:docsubtype: implementation
:docnumber: 24-051
:status: draft
:copyright-year: 2019
:edition: 1.0
:language: en
:published-date: yyyy-mm-dd
:received-date: yyyy-mm-dd
:issued-date: yyyy-mm-dd
:external-id: http://www.opengis.net/doc/IS/ogcapi-processes-4/1.0
:keywords: process, instance, spatial, data, openapi, job, create, update, delete, add, remove, REST, PATCH, POST, DELETE
:submitting-organizations: Geolabs; Terradue Srl.; Computer Research Institute of Montréal (CRIM).
:editor: Gérald Fenoy
:mn-document-class: ogc
:mn-output-extensions: xml,html,doc,pdf

////
Make sure to complete each included document
////
include::sections/clause_0_front_material.adoc[]

include::sections/clause_1_scope.adoc[]

include::sections/clause_2_conformance.adoc[]

include::sections/clause_3_references.adoc[]

include::sections/clause_4_terms_and_definitions.adoc[]

include::sections/clause_5_conventions.adoc[]

include::sections/clause_6_job_management.adoc[]

include::sections/clause_7_ogcapi-processes.adoc[]

include::sections/clause_8_openeo.adoc[]

include::sections/clause_9_provenance.adoc[]

include::sections/clause_10_oas.adoc[]

include::sections/clause_11_media_types.adoc[]

include::sections/clause_12_security_considerations.adoc[]

include::sections/annex_ats.adoc[]

////
Revision History should be the last annex before the Bibliography
Bibliography should be the last annex
////
include::sections/annex_history.adoc[]

include::sections/annex_bibliography.adoc[]
1 change: 1 addition & 0 deletions extensions/job_management/standard/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This folder contains the text for part 4 of the OGC API Processes standard.
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[[ats_jm]]
[conformance_class]
.Job Management
====
[%metadata]
identifier:: http://www.opengis.net/spec/ogcapi-processes-4/1.0/conf/job-management
subject:: <<rc_job-management,http://www.opengis.net/spec/ogcapi-processes-4/1.0/conf/job-management>>
classification:: Target Type:Web API
conformance-test:: /conf/dru/deploy/post-op
====

==== Create operation

include::jm/create/ATS_post-op.adoc[]
27 changes: 27 additions & 0 deletions extensions/job_management/standard/abstract_tests/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
This folder contains the Abstract Test Suite.

Each file describes a single test. The naming convention for these files is:

"cc/TESTn.adoc" where "cc" corresponds to the identifier for the requirements class and "n" corresponds to the test number. Numbers should have preceeding zeros appropriate for the total number of tests in the conformance class (e.g., the first test could be TEST001 if less than 1000 tests are anticipated).

The test is expressed according to this pattern:

````
===== Test case title

(( additional discussion, if needed ))

====== a) Test Purpose:
(( description ))

====== b) Pre-conditions:
* (( list all preconditions ))

====== c) Test Method:
* (( steps to execute and assertions to test ))

====== d) References:
* <<req_cc_req,Requirement /req/cc/req>>
````

NOTE: for each test, there must be one or more requirements in the "requirements" folder.
15 changes: 15 additions & 0 deletions extensions/job_management/standard/abstract_tests/cc/TEST001.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
===== Test case title

(( additional discussion, if needed ))

====== a) Test Purpose:
(( description ))

====== b) Pre-conditions:
* (( list all preconditions ))

====== c) Test Method:
* (( steps to execute and assertions to test ))

====== d) References:
* <<req_cc_req,Requirement /req/cc/req>>
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
[[ats_jm_deploy_post-op]]

[abstract_test]
====
[%metadata]
identifier:: /conf/jm/create/post-op
target:: <<req_job-management_create_post-op,/req/job-management/create/post-op>>
test-purpose:: Validate that the server support HTTP POST operation at the path /jobs/
test-method::
+
--
1. Construct a path for the {root}/jobs path.

2. Issue an HTTP POST request for each path.

3. Validate that the response header does not contain `405 Method not allowed`
--
====

5 changes: 5 additions & 0 deletions extensions/job_management/standard/figures/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Figures go here.

Each figure is a separate file with the naming convention:

"PTm_FIGn.xxx" where "m" is a number with leading zeroes appropriate for the total number of parts, "n" is a number with leading zeroes appropriate for the total number of figures and "xxx" is the appropriate extension for the file type.
9 changes: 9 additions & 0 deletions extensions/job_management/standard/images/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
Image files for graphics go here.

Each graphic is a separate file with the naming convention:

"FIGn.xxx" where "n" is a number with leading zeroes appropriate for the total number of figures and "xxx" is the appropriate extension for the file type.

or, for other graphics not associated with figures:

"GRPn.xxx" where "n" is a sequential number with leading zeroes appropriate for the total number of graphics and "xxx" is the appropriate extension for the file type.
2 changes: 2 additions & 0 deletions extensions/job_management/standard/recommendations/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
OGC API - Processes - Part 4: Job Management recommendations.

Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_job-management_additional-status-codes]]
[permission]
====
[%metadata]
label:: /per/core/additional-status-codes
part:: Servers MAY support other HTTP protocol capabilities. Therefore, the server may return other status codes than those listed in <<status_codes>>.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_job-management_create_body]]
[permission]
====
[%metadata]
label:: /per/job-management/create/body
part:: A server MAY support any encoding in the body of a HTTP POST operation.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_job-management_create_content-schema]]
[permission]
====
[%metadata]
label:: /per/job-management/create/content-schema
part:: The `Content-Schema` header MAY be an URI to a JSON or OpenAPI 3.0 Schema document that describes the structure of the request body.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[[rec_job-management_create_body-ogcapi-processes]]
[recommendation]
====
[%metadata]
label:: /rec/job-management/create/body-ogcapi-processes

part:: If the job can be encoded as an <<rc_ogcapi-processes,OGC API - Processes - Workflow Execute Request>>, implementation SHOULD consider supporting the <<rc_ogcapi-processes,OGC API - Processes - Workflow Execute Request>> encoding.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Content-Schema: https://schemas.opengis.net/ogcapi/processes/part3/1.0/openapi/schemas/execute-workflows.yaml reference for this encoding (i.e.: https://github.com/opengeospatial/ogcapi-processes/blob/72c8a8e90de37d8e7f1ac246f23797246c872ce5/openapi/schemas/processes-workflows/execute-workflows.yaml) should be recommended to align with per_job-management_create_content-schema (see other comment about duplicate mentions).

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[[rec_job-management_create_body-openeo]]
[recommendation]
====
[%metadata]
label:: /rec/job-management/create/body-openeo

part:: If the job can be encoded as an <<rc_openeo,OpenEO graph>>, implementation SHOULD consider supporting the <<rc_openeo,OpenEO graph>> encoding.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As for the OAP Workflow Execute Request case, this should probably recommend an established Content-Schema for openEO (see other comment about duplicate mentions).

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The question for openEO is, what you actually submit here? A job or a process? I guess it's a job description - and then I'm wondering whether there's a way to align that a bit more across encodings/specs...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For submission, it would be the contents of https://api.openeo.org/#tag/Batch-Jobs/operation/create-job
For the response, the https://api.openeo.org/#tag/Batch-Jobs/operation/describe-job

I'd expect Content-Schema: https://openeo.org/documentation/1.0/developers/api/openapi.yaml#/components/schemas/batch_job, but let us know if there is a more appropriate reference.

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_job-management_update_body]]
[permission]
====
[%metadata]
label:: /per/job-management/update/body
part:: A server MAY support any encoding in the body of a HTTP PATCH operation.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_job-management_update_content-schema]]
[permission]
====
[%metadata]
label:: /per/job-management/update/content-schema
part:: The `Content-Schema` header MAY be used to indicate the schema of a request body for updating a job.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[[rec_job-management_update-ogcapi-processes]]
[recommendation]
====
[%metadata]
label:: /rec/job-management/update/body-ogcapi-processes

part:: If a job can be created from an <<rc_ogcapi-processes,OGC API - Processes - Workflow Execute Request>>, implementations SHOULD consider supporting the <<rc_ogcapi-processes,OGC API - Processes - Workflow Execute Request>> encoding.

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[[rec_job-management_update-openeo]]
[recommendation]
====
[%metadata]
label:: /rec/job-management/update/body-openeo

part:: If a job can be created from an <<rc_openeo,OpenEO Process Graph>>, implementations SHOULD consider supporting the <<rc_openeo,OpenEO Process Graph>> encoding.

====
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_ogcapi-processes_create_content-schema]]
[permission]
====
[%metadata]
label:: /per/ogcapi-processes/create/content-schema
part:: The `Content-Schema` header MAY be pointing to the OpenAPI 3.0 schema https://github.com/opengeospatial/ogcapi-processes/blob/master/openapi/schemas/processes-workflows/execute-workflows.yaml[execute-workflows.yaml].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally, all endpoints should use the URI to the expected location when published
(i.e: https://schemas.opengis.net/ogcapi/processes/part3/1.0/openapi/schemas/execute-workflows.yaml), so that they can refer to an eventual static reference.

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_ogcapi-processes_update_content-schema]]
[permission]
====
[%metadata]
label:: /per/ogcapi-processes/update/content-schema
part:: The `Content-Schema` header MAY be pointing to the OpenAPI 3.0 schema https://github.com/opengeospatial/ogcapi-processes/blob/master/openapi/schemas/processes-workflows/execute-workflows.yaml[execute-workflows.yaml].
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_openeo_create_content-schema]]
[permission]
====
[%metadata]
label:: /per/openeo/create/content-schema
part:: The `Content-Schema` header MAY be pointing to https://raw.githubusercontent.com/Open-EO/openeo-processes/master/meta/subtype-schemas.json#/definitions/process-graph[OpenEO Process Graph schema].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly, we should look for an "official" endpoint where a static/versioned document can be found.

@m-mohr does one exist ?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_openeo_update_content-schema]]
[permission]
====
[%metadata]
label:: /per/openeo/update/content-schema
part:: The `Content-Schema` header MAY be pointing to https://raw.githubusercontent.com/Open-EO/openeo-processes/master/meta/subtype-schemas.json#/definitions/process-graph[OpenEO Process Graph schema].
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[req_deploy-replace-undeploy_deploy_body]]
[requirement]
====
[%metadata]
label:: /req/job-management/create/body
part:: The body of the POST request SHALL be in JSON format.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On one hand, it says in a permission that any encoding is allowed, but here it's required to be JSON?!

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[req_job-management_create_content-type]]
[requirement]
====
[%metadata]
label:: /req/job-management/create/content-type
part:: The `Content-Type` https://tools.ietf.org/html/rfc2616#section-14.17[header] SHALL be used to declare the media type of the request.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[req_job-management_create_post-op]]
[requirement]
====
[%metadata]
label:: /req/job-management/create/post-op
part:: The server SHALL support the HTTP POST operation at the path `/jobs`.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[[req_job-management_create_response-body]]
[requirement]
====
[%metadata]
label:: /req/job-management/create/response-body
part:: The response SHALL include a body that contains a status information of the created job that conforms to the https://schemas.opengis.net/ogcapi/processes/part1/1.0/openapi/schemas/statusInfo.yaml[statusInfo.yaml] schema.
part:: The response SHALL contain a `created` status code and the `jobId` property that contains the job identifier.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is currently not quite aligned with openEO. There we just return a 201 with a Location header that points to GET /jobs/:id, which then includes created, id (not jobId, I thought we settled on id?), etc. Is that allowed, too?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an alignment with OGC API /processes/{processId}/execution that returns minimally the status and job ID to make the response consistent with either endpoint.

The Location should be mentioned. It is also used by OAP: https://docs.ogc.org/is/18-062r2/18-062r2.html#req_core_process-execute-success-async

And yes, regarding the jobId/id naming.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discard my comment about Location, it is already in the other requirement.

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[req_job-management_create_response-jobid]]
[requirement]
====
[%metadata]
label:: /req/job-management/create/response-jobid
part:: If the operation completes, the server SHALL generate a job identifier (i.e. `{jobId}`) for the created job.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
part:: If the operation completes, the server SHALL generate a job identifier (i.e. `{jobId}`) for the created job.
part:: If the operation completes, the server SHALL generate a job identifier for the created job.

I this variable is confusing w/o context.

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[[req_job-management_create_response_success]]
[requirement]
====
[%metadata]
label:: /req/job-management/create/response-success
part:: A successful execution of the operation SHALL be reported as a response with a HTTP status code `201`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that some servers could want to handle the case evoked in the meeting, (ie: job "created" and deleted right away to do a "sync" execution), a 200 OK or 202 Accepted, might be considered. This could also be used by async execution, to indicate that the job was accepted, but not yet "running".

Copy link

@m-mohr m-mohr Oct 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this requirement relate to https://github.com/opengeospatial/ogcapi-processes/pull/437/files#r1789867436 ? (Comment is unrelated to Francis' comment)

part:: A response with HTTP status code `201` SHALL include a `Location` header with the URI of the deployed processes (path: `/jobs/{jobId}`).
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[[req_job-management_create_unsupported-media-type]]
[requirement]
====
[%metadata]
label:: /req/job-management/create/unsupported-media-type

part:: If the server does not support the Content-Type header associated with the request body, the code of the response SHALL be `415 Unsupported Media Type`.
gfenoy marked this conversation as resolved.
Show resolved Hide resolved
part:: The content of that response SHALL be based upon the OpenAPI
3.0 schema https://raw.githubusercontent.com/opengeospatial/ogcapi-processes/master/core/openapi/schemas/exception.yaml[exception.yaml].
part:: The `type` of the exception SHALL be “http://www.opengis.net/def/exceptions/ogcapi-processes-4/1.0/unsupported-media-type”.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[[req_job-management_definition_get-op]]
[requirement]
====
[%metadata]
label:: /req/job-management/definition/get-op
part:: For every jobs (using method: POST on path: /jobs/{jobId}), the server SHALL support the HTTP GET operation at the path `/jobs/{jobId}/definition`.
Copy link

@m-mohr m-mohr Oct 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the definition meant to return? The process? (e.g. CWL or openEO UDP?)

We include that directly in GET /jobs/:id - I guess that's fine and we can add an additional endpoint, but maybe this is more an optional endpoint for cases where a definition can't be embedded? And it can be explored via a link in GET /jobs/:id?

What is POST /jobs/:id?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CWL or openEO UDP -> Yes

POST /jobs/:id -> typo

I'm fine with making this endpoint optional since servers can indeed return the definition directly and avoid the extra request. However, if that is the case, I suggest that a requirement says as such explicitly, or that providing a link reference is required (either pointing to the job itself or /jobs/{jobId}/definition). The important aspect is that the job definition can be found "somewhere".

part:: The parameter `jobId` is each `id` property in the job-list response (JSONPath: `$.jobs[*].id`).

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[req_job-management_definition_response-body]]
[requirement]
====
[%metadata]
label:: /req/job-management/definition/response-body
part:: A response with HTTP status code `200` SHALL include a body that contains the request body used to create or update the job.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe needs refining of the wording. It is not really clear what is the intent here?

Is it that, given a POST /jobs {something}, the {something} should be included in the GET /jobs/{jobID} response?

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[req_job-management_definition_response-success]]
[requirement]
====
[%metadata]
label:: /req/job-management/definition/response-success
part:: A successful access to the resource SHALL be reported as a response with a HTTP status code `200`.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[[req_job-management_start_response]]
[requirement]
====
[%metadata]
label:: /req/job-management/start/response
part:: A successful execution of the operation SHALL be reported as a response with a HTTP status code '200'.
Copy link
Contributor

@fmigneault fmigneault Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using POST /jobs/{jobId}/results, the server should probably still have the option to negotiate the execution mode. In other words, sync and 200 would be used by default, but a Prefer: respond-async would allow the server to trigger this job, although it would not respond with results right away. If async is selected this way, the response would instead be the usual Job Status with monitoring. Also, a 202 would have to be used, since no job is "created" in that case. Once completed, the results can be retrieved from GET /jobs/{jobId}/results.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you not expect the sync result already from POST /jobs?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could do either of the following:

  1. POST /jobs + Prefer: wait=X to respond results inline (no need for other request)

  2. POST /jobs + Prefer: respond-async responds with Job Status

    1. starts the job whenever resources are ready
    2. GET /jobs/{jobId}/results to retrieve results once status: succeeded
  3. POST /jobs + Prefer: wait=X + status: create

    1. places the job in created state, until triggered later on
    2. POST /jobs/{jobId}/results to trigger the job and return the results inline
  4. POST /jobs + Prefer: respond-async + status: create

    1. places the job in created state, until triggered later on
    2. POST /jobs/{jobId}/results to trigger the job and return with Job Status
    3. GET /jobs/{jobId}/results to retrieve results once status: succeeded

Here, the 202 would make sense for 4.ii, whereas 200 for 3.ii

Copy link

@m-mohr m-mohr Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like you have too many options in OAP. Honestly, this feels overengineered. You are implementing things server side that usually a client would handle. Then the server (and client) implementations are much simpler. openEO API is much simpler, but can still do everything you do here it's their clients easily.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, we've had users expecting all these combinations so far, so we must support them to accommodate everyone. If Part 4 only allows working with openEO style jobs, it's not much of an extension for OAP.

Copy link

@m-mohr m-mohr Oct 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Users often don't really know what they need and that there are alternatives... If I would add averything to openEO that users just ask for, it would be a mess. Often it is possible to show them alternatives that work equally good.

Anyway, having multiple possible ways of doing the same thing should IMHO be avoided and I still think a client can simplify API and server development.

The way I understood our discussions initially the OAP version of the openEO jobs is slightly different/extended, but this is a completely different spec with a small subset being openEO. If that's what you are looking for I give up. I don't think the way it is is any useful. Then better don't align at all, which makes it less confusing for users because things are clearly separated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first 2 are a direct mapping of what POST /processes/{processId}/execution defines. Those have to be mapped as is on POST /jobs to allow a server/client to work only with the /jobs endpoint and integrating workflows.

The other 2 are for openEO alignment, allowing the creation of a job first, and then triggering its execution with a second request. At the moment in OAP, there is no such concept of job creation not queued right away, so the first 2 points needs to be available for servers that do not intend to create jobs this way, but still want to define workflows.

What you're asking for is to consider only openEO's perspective, which of course is easier, because you're thinking only about your use cases.

And to clarify, the "users" here that I was referring to are the participants of TB20-GDC. We have had discussions trying to show others (including you) alternatives, but nobody wants to concede anything. So, we are in this situation where all 3 modes of sync, async, and create/trigger execution are supported by POST /jobs as alternatives of running processes by different users.

The best simplification that can be done is by merging (3) and (4), basically not allowing switching Prefer when doing POST /jobs/{jobId}/results. It would have to be configured from the get go when doing POST /jobs or by follow-up PATCH /jobs/{jobId} to modify it.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first 2 are a direct mapping of what POST /processes/{processId}/execution defines. Those have to be mapped as is on POST /jobs to allow a server/client to work only with the /jobs endpoint and integrating workflows.

Why if there's already an alternative?

The other 2 are for openEO alignment, allowing the creation of a job first, and then triggering its execution with a second request. At the moment in OAP, there is no such concept of job creation not queued right away, so the first 2 points needs to be available for servers that do not intend to create jobs this way, but still want to define workflows.

3 and 4 are not available in openEO right now?! 4 would be if you remove the "and return with Job Status".
I still think not all of this needs to be available in favor of clients that can make this happen in one call if you want.
But I understand that historically OGC was about servers, not clients, which I think is a massiv mistake.

What you're asking for is to consider only openEO's perspective, which of course is easier, because you're thinking only about your use cases.

No.

And to clarify, the "users" here that I was referring to are the participants of TB20-GDC. We have had discussions trying to show others (including you) alternatives, but nobody wants to concede anything. So, we are in this situation where all 3 modes of sync, async, and create/trigger execution are supported by POST /jobs as alternatives of running processes by different users.

No. We had discussions where each side was making compromises and we were on a good way, but it seems we are not following the first meeting where I was still available. I guess we should listen to the first meeting recording again. The thing is, if no one make compromises we should NOT align, because then we end up with an overengineered complex API. Then it's easier to have two separate ones. I was under the impression that we were on a good way, but this PR is not reflecting this as far as I can tell...

The best simplification that can be done is by merging (3) and (4), basically not allowing switching Prefer when doing POST /jobs/{jobId}/results. It would have to be configured from the get go when doing POST /jobs or by follow-up PATCH /jobs/{jobId} to modify it.

I think (2) and (3) can be easily be removed. You can simply solve 2 with two subsequent requests and why should there be two ways to process synchronously? We need to simplify not allow everything to be solved in 100 ways.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why if there's already an alternative?

It is not a direct alternative, because /processes/{processId}/execution does not allow all capabilities that /jobs offers. Notably, defining workflow/graphs that do not have only one top-level process, and providing additional job endpoints for metadata.

3 and 4 are not available in openEO right now?!

You misunderstood. It's the other way around. It is not available in OAP!
Only openEO defines such capability to create a job without putting it in queue. In OAP, when you POST the job, it is either executed in sync or async, but it is done right away. OAP does not allow creating a job, modify it, and submit it in queue after. This is added by Part 4.

think (2) and (3) can be easily be removed. You can simply solve 2 with two subsequent requests and why should there be two ways to process synchronously?

I agree about removing (3), and keeping only (4) for the openEO-style execution.

However, (2) should remain. It is completely different from (3)/(4) use case, because it locks the job right away and places it in queue (just like if it was executed in sync right away). Why force the client to do a subsequent request to trigger the job, when this hint can be supplied immediately via the Prefer header? In most situations, clients that submit a job intend for it to run right away, because their goal is to obtain the result from it. Avoiding an extra request for every single time a job is submitted effectively cuts the network traffic by 2. If the client intends to do more with the job (modify it, review it, whatever) before executing it, it makes more sense IMO that this detail should be specified explicitly (ie: status: create). Then, the client/server agree explicitly that a subsequent request is needed for the actual execution trigger.

part:: A response SHALL be a document that conforms to statusInfo.yaml.
====
Loading
Loading