Adding parameters dynamically

[dpavlic]

This is based on #912. I've had a few different ideas on how to support a raw stream case and the simplest solution is to simply pass the required information to inputs. However, I'd prefer to add the information to the catalog and then just deliver the raw catalog object (i.e., don't have kedro try to save / load for me) to my node.

Since we already have a "params:" directive I thought I could jimmy a new directive "raw_catalog:" or something like that, where the the matching catalog dict is passed directly to the function without kedro trying to load it. It seemed to me that the somewhat hackish, but also 'good enough' way would be to do this in before_pipeline_run() hook. To keep things simple, at this point I'd just add dynamic parameters to the node where "raw_catalog" directive is transformed to a "params:" parameter with the appropriate catalog dict. The parameter would then be dynamically created at this point.

But... I have no idea how to actually pass 'dynamic' parameters to the pipeline in the hook. Is there a way to add not-previously-defined parameters within the before_pipeline_run() hook and pass that to the pipeline?

[datajoely]

So we have a large running issue #817 that covers our research into people's existing painpoints with respect to configuration.

Would you look at proposal 4 at the bottom and comment what you think?