Governance usability: Incremental updates of large proposals

[eddyashton]

To update the constitution or JS application code, we currently require the entire new contents to be submitted in a single proposal. This is to reduce the risk of invalid/corrupt code being submitted (relying on missing or incorrect dependencies). We also think this makes audit simpler - the current value of these items is found from a single write transaction, rather than a projection across multiple write transactions.

However, this brings a few barriers to usability:

1. Incremental updates require global knowledge. There are reasonable governance actions which are incremental, such as "add support for this action", or "add this JS endpoint". We support the equivalent for many actions (ie - "add this user", rather than "make the list of all users this"), but not here. To implement such an incremental action, the proposer must have access to the existing value, so they can modify it and upload the entire modified value. We could support modification rather than replacement proposals for consistency with other actions.
2. Absorbing upstream changes. Perhaps a subcase of the point above, but deserves extra attention. The common pattern we expect is for CCF services to start from our sample constitutions, and make modifications to those (their own resolve logic, adding their own actions). When we make new releases which those services want to upgrade to, they need to reconcile any changes in the original sample constitution with their custom constitution. We think these changes are rare and well-documented (eg - adding cert validity actions), but they may still need to be consumed by all apps. We should make it easy to integrate changes from the sample constitutions.
3. Viewing diffs of these changes. When members want to make a semantically incremental change ("add this action"), but do so by replacing the entire constitution, it makes proposal validation more difficult. As a supporting member viewing this proposal, it is non-obvious that the change was incremental, unless I manually compare the proposed value with the existing value through an external diff tool.

So the points to discuss:

• Can we separate our sample constitutions into more obviously recombinable parts?
  • Yes, we can split actions.js into multiple files
• Can we add proposals that make incremental changes rather than wholesale replacements?
  • One approach would be to split the current monolithic values into multiple key/value pairs. That's a significant rewrite.
  • We could also add actions which modify an existing value, rather than blindly putting. As a first pass, these could append. Maybe the ideal form is a git-style diff?
• Can we display proposals in a more friendly form?
  • Should we add more builtin endpoints to access existing monolithic values?
  • Can we create diff views from the current proposals?

[achamayou]

Also see #2466 and #1585 about the performance aspect, and #3537 with regards to action versioning.

In general, I think we should try to leverage existing mechanisms and tooling for diffing and modularity, rather than try to roll our own, as much as possible. If that means new endpoints to expose the right input for those tools, that seems fine. Allowing an easy app or constitution diff is certainly an obvious improvement.

I am a little sceptical of the incremental approach, because typical quality gates (type checking and testing) do not work that way, and require the whole of the code to validate. Trading off safety for convenience seems nowhere less appealing than in the constitution.
We discussed adding support for JS modules to the constitution around 1.0 time, and ultimately decided against it, but it may be time to revisit this decision?