Defining Optionality

[dmlloyd]

In order to satisfactorily define optionality, we must revisit some foundational requirements (some of which we have informally discussed previously). In the following list, I am assuming that the term "configuration source" refers to the means by which a user can supply configuration values to the configuration system.

Requirements

Some of the following requirements establish background for optionality, and others are the logical consequence of those foundational requirements.

Empty values

Here are the requirements we must establish or reassert:

• The user should be able to specify a non-empty simple value in a configuration source, such as a text string or a number (obviously)

This is the foundational concept of configuration.

• The user should be able to provide a non-missing but empty simple value, such as an empty string

This capability is not consistently present in MP Config and is not possible in the SmallRye Config implementation, which was the topic of some controversy; we have the ability to rectify this architecturally with Jakarta Config and the consensus has been consistent in that we wish to do so.

Missing values

• A value is implicitly missing when it is not present in a configuration source
• The user should be able to override any value from a given configuration source with a new value from a higher priority source

Therefore, a configuration source should be consulted for a value only if the value is implicitly missing from all higher-priority configuration sources.

In this way, the definition of the term "empty value" and the term "missing value" are made distinct for the purposes of this specification as desired.

• The user should be able to override any value from a given configuration source with an explicitly missing value

Note well: This is distinct from the ability to restore a value to its default or from the ability to provide an empty value such as an empty list or string). Furthermore, an explicitly missing value is not implicitly missing (i.e. it overrides the value of any lower-priority configuration source which may exist).

• A default value of a property is the value returned for a that property when the property is implicitly missing from all configuration sources

This establishes the exact interaction between empty, missing, and default values.

Lists

While there is still much to sort out about lists of simple values, with this model I think it is safe to establish that any list (or set or sequence or array etc.) of simple values cannot contain any missing values. If a member of such a list would correspond to a missing value, that value shall be skipped and not count towards the content of the list in any way.

Implications

Config sources

In order to meet these requirements, configuration sources must be able to react to a query in one of the following three ways:

• Respond with a value when that configuration source has a value for the key
• Forward the query to a lower-priority data source when the configuration source does not have an entry for the key
• Respond with "missing" when that configuration source has an entry for the key whose value is explicitly missing

This should be possible through conscientious API design; I would add the following constraint as well:

• It should be impossible for a configuration source implementation to return a value that does not correspond to one of the above three categories.

Note that practically speaking, this means that while the primary specification APIs (as well as any specified implementations of interface methods) should never return null to help avoid user error, the configuration source (which will likely be an extensible API in the final specification) should embrace null as a representation for a missing value for that very same reason: otherwise, we will have to specify the behavior of the system when null is returned from a user configuration source implementation, which is not ideal.

Syntax

In order to support the requirement to distinguish empty values from missing values, the syntax for an explicitly missing value must be distinct from the syntax for an explicitly empty value of any given type (e.g. string, array, list) for all configuration sources. While rich formats like YAML might have idiomatic ways to do this (for example, using the special YAML-specific null/~ value to indicate explicitly missing), file formats such as the properties file format or sources such as system properties or environment properties might have a harder time expressing this in a useful way without the use of some kind of special value syntax.

We have (at least) the following options to deal with this:

• Design a syntax specific to these types
• Leverage the property expression syntax in some way to be able to produce a "missing" value
• Restrict these sources from producing either empty values or explicitly missing values

Alternatives

Instead of supporting the idea of differentiated explicitly missing, implicitly missing, and "empty" value behaviors, we could try another approach.

Use the SmallRye Config method

One alternative could be to drop the difference between explicitly missing and "empty" values, using an empty string as a canonical (unconverted) representation for explicitly missing. This has all of the previously discussed downsides, but on the other hand, the model is internally consistent.

Eliminate the concept of explicitly missing values

We could simply decide to make it impossible for the user to override a value with an explicitly empty value. This approach has a few downsides:

• A property with a default value could never be empty, thus optional properties with default values don't really make sense
• Missingness carries semantic meaning; preventing the user from being able to express missingness in some circumstances limits the user's ability to express the full semantic range of behaviors for a given configuration

[ljnelson]

To make a few assumptions above explicit:

1. This assumes we're going to implement the thing that provides the "configuration object" described in Should Jakarta Config Specify A Programmatic API? #110 by having "configuration sources" supply its parts
2. Corollary: this assumes that the "parts" of a "configuration object" are acquired or made available in a way different from how the "configuration object" is loaded by the component developer.
3. This assumes that if there is such a thing as a "configuration source" there are actually going to be potentially many of them, notionally stacked in priority order, with the capability to shadow one another

I think these are big assumptions worthy of further discussion elsewhere and I won't comment on them here.

I agree that absence, presence and determinism ("optionality") need to be defined in the specification and made explicit, regardless of how they are implemented:

• An absent value is one that, for a given run of a program, does not exist, regardless of what it might otherwise be
  • An absent value may be absent because it is simply not designated or mentioned in any way, or it may be absent because a configuration author has explicitly noted that it must be so
• A present value is one that, for a given run of a program, exists
• A deterministic value is an absent or a present value that, for a given run of a program, does not subsequently change
  • Since we apparently ruled out "mutability" in Mutability discussion #109 it follows that in our world once a value has been determined to be absent or present it will always be so; all values, in other words, are deterministic in our world; see also Should loading a configuration POJO be specified to have singleton semantics? #119

I therefore also agree that authors must be able to specify a value's absence.

[dmlloyd]

I view absence and presence as properties of a value, while thinking of determinism as a property of the overall system. I think optionality (as well as requiredness) is a property of the definition of a value. When an optional value is realized, it is either present or absent as you say, giving us determinism. Thus I do not view optionality and determinism as equivalent concepts either broadly speaking or specifically in the context of this specification.

[ljnelson]

Then I'm searching for the name of the property of an absent or present value that describes its lifespan, as in the following: I have called some method and it has given me some value (or a value's absence). If I call the method again, am I guaranteed to receive the very value I have in my hand? Are you saying there is no such lifespan-oriented property of the value?

Transience/permanence? Perdurance (ugh)?

[dmlloyd]

If we describe any individual configuration as immutable (which I think we really have to do), then it follows that each property within the configuration is consistently either present or absent once the configuration is resolved, and if present, the value is also immutable. An instance method on a configuration mapping interface would thus always return the same object given the same instance of that interface. So maybe "consistency" is the word?

[ljnelson]

As we used to say many decades ago, let me check the "literature" on this. "Consistency" almost feels right but then its opposite is…an inconsistent value? That smells weird. There must be a de facto standard term that I'm either simply forgetting or never learned for this. (Far from being academic, I think getting this term right is important both for (a) describing what we're doing and (b) describing what we're not doing.)

[ljnelson]

Ah! Literally just this second remembered (brains are cool; mine was obviously working on this while I was sleeping).

The term I was forgetting is determinate, and sure enough if you search the literature for "determinate value" you get a lot of hits describing (academically) exactly what we're talking about. A deterministic function produces a determinate value/outcome; a non-deterministic function can produce, perhaps surprisingly, a determinate or indeterminate value/outcome. The property that the value/outcome has is called determinacy.

Using these terms you are saying that all methods on a configuration object, for example, return determinate values (currently, in our world, in conjunction with #119), and the determinate values so returned are either (therefore, in our world, forever, during a given application run) absent or present.