What Priority means in Jakarta?

[Emily-Jiang]

Further to last week's discussion, I spent a bit more time looking at the usage in CDI and try to recall what we discussed in the past regarding priority understanding. For priority used for overwrite reasons, such as converter and configsources.

We mainly discussed the invocation order for interceptors etc. No example was found on overwrite purpose as we used in Config.

I did further digging and found in CDI spec, the following explains clearly that highest priority wins.

https://jakarta.ee/specifications/cdi/3.0/jakarta-cdi-spec-3.0.html#unsatisfied_and_ambig_dependencies

All the beans left are alternatives with a priority, or producer methods or fields of beans that are alternatives with a priority, then the container will determine the highest priority value, and eliminate all beans, except for alternatives with the highest priority and producer methods and fields of alternatives with the highest priority value. If there is exactly one bean remaining, the container will select this bean, and the ambiguous dependency is called resolvable.

The above sentence clearly says the highest priority wins. MP Config priority is consistent with this behaviour.

[joakime]

The @Priority feature of CDI is great and all.
However, Jakarta Config, does not need to do anything to allow CDI to use it.
The priority concepts and layers we have discussed (in the past) in Jakarta Config are just not necessary to exist in Jakarta Config.
CDI + Priority will still work.

[Emily-Jiang]

The priority concepts and layers we have discussed (in the past) in Jakarta Config are just not necessary to exist in Jakarta Config.

The priority concept needs to exist in Jakarta Config to specify the order of config source and converters.

[joakime]

I disagree, this priority is just a crutch your specific implementation of Config has implemented.

Others (like Eclipse Jetty, Google, etc) will base the order on other things, such as a topological sort from enabled feature sets.

It should not be present in Config itself.

[jansupol]

In Jakarta REST, the Spec says:

Execution chains for extension points ContainerRequest, PreMatchContainerRequest, ClientRequest, ReadFrom and WriteTo
are sorted in ascending order; the lower the number the higher the priority. Execution chains for extension
points ContainerResponse and ClientResponse are sorted in descending order; the higher the number the
higher the priority. These rules ensure that response filters are executed in reversed order of request filters.

For the request the lower the number the higher the priority. It has the following consequences:

1. If only one is to be used, the one with the lowest number wins
2. If more are used, the one with the highest number is used the first, the one with the lowest number is used the last so that it can override all previous and win.

[joakime]

Per the restful-ws TCK the @jakarta.annotation.Priority annotation is used to layer extensions points between the REST Client and the REST Endpoint.

REST Client -> [extensions] -> REST endpoint

The difference you see in what they consider important is due to how the request vs the response flows through the list of extensions

Highest value is closest to REST endpoint side, the most important.

  ExtFirst registered at priority 10
  ExtSecond registered at priority 20
  ExtThird registered at priority 30

  Request  from REST   Client -> ExtFirst -> ExtSecond -> ExtThird -> REST Endpoint
  Response from REST Endpoint -> ExtThird -> ExtSecond -> ExtFirst -> REST Client

restful-ws recommends that you use the predefined values from jakarta.ws.rs.Priorities as the value for @jakarta.annotation.Priority(int).

If you look at the predefined priority list, you'll understand this order better. (do authn/authz first, then headers, then decoders/encoders, then user).

Looking back the history of restful-ws, you'll see that this entire layer exists because they were unhappy with the lack of control they have with the servlet spec.
So they implemented a decent chunk of the servlet spec behaviors themselves just to exert this control they desired. (servlet auth layer + servlet filter order + late config/binding)
Once Jakarta Core HTTP exists, this entire layer in restful-ws has no reason exist anymore.

[jansupol]

I stand corrected, thank you @joakime. My memory betrayed me. To sum the behaviour (sorry for confusion):

1. If only one is used, the one with the lowest number wins
2. If more are used, the one with the lowest number is used the first, the one with the highest number is used the last.

[Emily-Jiang]

It still suggests the higher priority the most important. Whatever called last will have the final say. CDI is very clear that highest priority wins for alternative beans.

[Emily-Jiang]

I discussed the priority usage in CDI weekly call this week.
For CDI:

1. If only one is used, the one with the highest number wins (e.g. alternatives)
2. If more are used, the one with the lowest number is used the first, the one with the highest number is used the last (e.g. CDI interceptors)

[struberg]

As written on the list: JAX-RS sadly uses both directions. There are cases where the highest value in @Priority wins and cases where the lowest value wins. Don't ask me why, but that's the way it is. Also the JAX-RS spec did use Priority a long time after it's introduction. So probably we should not take care too much about JAX-RS at all?

[joakime]

Here's a breakdown of a specific setup with servlet, but without Application created ConfigSource instances (I can flesh this out for that as well)

JVM
  Distribution
    libs/
      lib-foo.jar!/META-INF/config.properties (lib specific, has dependencies)
      lib-bar.jar!/META-INF/config.properties (lib specific, has dependencies)
    modules/
      technology-foo/
        config.properties (technology specific, has dependencies)
      technology-bar/
        config.properties (technology specific, has dependencies)
    default/
      config-a.properties (dist specific, has dependencies)
      config-b.properties (dist specific, has dependencies)
  Instance
    libs/
      lib-zed.jar!/META-INF/config.properties (lib specific, has dependencies)
    config.d/
      technology.properties (technology dependencies)
      tech-a.properties (instance specific, has dependencies)
      tech-b.properties (instance specific, has dependencies)
    webapps/
      bar-b.war  
        WEB-INF/
          config.properties (webapp specific properties)
          lib/
            flax.jar!/META-INF/config.properties (lib specific)
      bar-b.xml
        <config> section

[ljnelson]

The root question behind all this priority and ordinal talk is: will Jakarta Config require a total (not partial) order of configuration value sources within an application? It seems like that's what getOrdinal() on MicroProfile Config's ConfigSource is trying to help with, but of course it cannot by itself ever guarantee a total order (total ordering of any application components within an application is definitionally an application-level concern, not a component-level concern; your ConfigSource and my ConfigSource can happily report the same ordinal of, say, 37 and are now considered indistinguishable; this is a partial order). Maybe Jakarta Config only wants a partial order, in which case: is there really a point to it, given that in such a case any configuration value source can be indistinguishable from any other?

[ljnelson]

Perhaps we are in agreement. I am using "total order" in the set-theory sense of the term: every element is distinguishable from every other element.

You've presented several scenarios that are governed by total ordering of configuration data sources. That is: for any given instance of an application starting up, in your examples there is an order that can be computed for that application's configuration data sources (I'm using abstract terms on purpose) that ensures that no randomness is present, i.e. for that application instance you can consult a rule book somewhere and determine which configuration data sources will be stacked in which order. You've said as much above.

More importantly, if you start that very same application instance 100 times, there is apparently no chance that it will start with configuration data sources chosen at random. Therefore, there is a total ordering of configuration data sources within that application instance.

Now, if you change the application's configuration and redeploy the application, you're running what amounts to a "new" application instance. Likewise, for this application instance, while the actual configuration data sources used may be different, there will still be a total ordering of them (every source will be distinguishable from every other source and there is a rule book that tells you definitively which ones will be used). If you start it 100 times, there is, likewise, no chance that it will start with configuration data sources organized in a different way each time. This too is a total order of configuration data sources within this application instance, just a different one.

By contrast, with MIcroProfile Config, and now with Jakarta Config, if your library ships with a ConfigSource that returns 37 as a hard-coded value from its getOrdinal() method and "config" as a hard-coded value from its getName() method, and my library ships with one that also returns 37 as a hard-coded value from its getOrdinal() method and "config" as a hard-coded value from its getName() method, but our respective ConfigSources do wildly different things and return wildly different values, and our libraries containing them are combined into an application, the MicroProfile Config specification indicates our ConfigSources must be sorted first by ordinal, then by name. That's fine, but as you can see in this example even after this sorting, our ConfigSources remain indistinguishable (according to the specification). Hence MicroProfile Config requires only a partial order of configuration data sources within an application, and if this hypothetical application containing these two sources is started 100 times, it is legal for the MicroProfile Config implementation being used to just…pick one at random each time. We start the application two times, and the first time we get your ConfigSource trumping mine, and the second time mine trumps yours.

This seems bad to me and worth changing in the Jakarta Config specification. What is "bad" here is the partial ordering within any given application instance; it seems to me that Jakarta Config should have verbiage in it somewhere saying something like if two ConfigSources within an application instance can't be distinguished for any reason during startup then an error should be raised. Then at least the person putting the application together could arrange for a different ConfigSource with a higher ordinal to be used and to do something else…unless our sources' ordinals were already Integer.MAX_VALUE :-). (With the current specification, there is no guaranteed way to be able to correct the problem.)

All of this is because the larger problem is that we're talking about ordering of configuration data sources (I'm using abstract terms on purpose) within an application, but the application's configuration ordering semantics are currently dictated by the data sources. That leads inexorably to conflicts unless you can also specify a means of resolving those conflicts when putting your application together. Ideally you'd also be able to detect this state of affairs before trying things out.

[joakime]

More importantly, if you start that very same application instance 100 times, there is apparently no chance that it will start with configuration data sources chosen at random. Therefore, there is a total ordering of configuration data sources within that application instance.

There is no guarantee of order for a ClassLoader.
Requesting content from the ClassLoader can result in different order with each execution.
This even impacts ServiceLoader.

This is also not a guarantee in the Servlet spec (when Servlet 3.0 introduced Servlet annotations and Web Fragments as concepts).
If you are a lazy developer and include WEB-INF/lib/*.jar content that has overlapping classes you are in an unfavorable place with repeatable/stable execution runs.
(This is one of the reasons so many "duplicate class check" plugins exist in maven and gradle.)
This also causes web-fragments to order differently during initialization as a result of the non-order of a ClassLoader.
There's some provisions in the web descriptor to order specific things before others, but it's highly limited and cannot compensate for many things.

Keep in mind that audience for Jakarta Config is the entire Jakarta EE ecosystem, not just MicroProfile.
The MicroProfile use cases are useful.
But those use cases don't have a special place or trump other use cases.

What is "bad" here is the partial ordering within any given application instance; it seems to me that Jakarta Config should have verbiage in it somewhere saying something like if two ConfigSources within an application instance can't be distinguished for any reason during startup then an error should be raised. Then at least the person putting the application together could arrange for a different ConfigSource with a higher ordinal to be used and to do something else…unless our sources' ordinals were already Integer.MAX_VALUE :-). (With the current specification, there is no guaranteed way to be able to correct the problem.)

If Jakarta Config adopts ...

• ConfigSources a new class and is the responsibility of the Jakarta Config Implementation to create.
• ConfigSource without an ordinal.

Then we have a Jakarta Config that is adaptable for the MicroProfile case and also the other cases that can't / won't use ordinal.

[ljnelson]

OK; I'm now confused so I'll stop here. I think? maybe? I agree with you? that ideally within the Jakarta EE world there should always be a way to configure any application deterministically. I hope I'm not saying anything other than that. MicroProfile Config, and as of this writing Jakarta Config in its current state do not allow for this. I am not a fan at all of either effort so far. I'll leave it at that.

[struberg]

There is no guarantee of order for a ClassLoader. Requesting content from the ClassLoader can result in different order with each execution. This even impacts ServiceLoader.

Yes, which is the reason we try to not rely on it by picking up ALL ConfigSources first, and then sort them via ordinal and names. If there are 2 ConfigSources with exactly the same name and ordinal, well then it will start to behave randomly.
https://github.com/eclipse/ConfigJSR/blob/f15f4ee1ef9e68ce14af9b714899a94663afede3/api/src/main/java/javax/config/spi/ConfigSource.java#L112

If two ConfigSources have the same ordinal and name, AND contain the same config keys with different values, then non-portable behaviour results. We might also issue a warning or error in this case.

Also note that in practice this does not happen as frequent as one might think as implementations usually use the full classpath location (resource path or file path). We probably should explicitly mandate this for the default provided property file ConfigSources though!

[struberg]

If Jakarta Config adopts ...

* `ConfigSources` a new class and is the responsibility of the Jakarta Config Implementation to create.

* `ConfigSource` without an ordinal.

Then we have a Jakarta Config that is adaptable for the MicroProfile case and also the other cases that can't / won't use ordinal.

Could you probably elaborate on this idea a bit more in detail? I don't really get what the 2 should do and where they should differ. txs!

[dmlloyd]

We could probably unpin this now?

[ljnelson]

Unpinning.