Use shared ticket format to share an export ticket between clients

[devinrsmith]

Currently, this integration relies on setting a random variable name in the global scope so that it can be passed to the JS client:

• https://github.com/deephaven/deephaven-ipywidgets/blob/v0.6.0/deephaven_ipywidgets/deephaven.py#L146
• https://github.com/deephaven/deephaven-ipywidgets/blob/v0.6.0/deephaven_ipywidgets/deephaven.py#L152

With the introduction of deephaven/deephaven-core#5340, there should be a much cleaner mechanism to share exports between clients.

[devinrsmith]

This may depend also on a python wrapper on the server side and pydeephaven side.

[devinrsmith]

We may need to consider also if there is a way to get a callback into the python side; a way for the JS client to let the python side that it has successfully exported the shared token (in the python client can then release its own export if desired).

[devinrsmith]

Here is the pydeephaven issue, deephaven/deephaven-core#5372

[mofojed]

Should be able to use this with plots and widgets as well. May need deephaven/deephaven-core#5666

[mofojed]

@niloc132 how do we use this ticket from the JS API side?

[mofojed]

Needs deephaven/deephaven-core#5666 to be completed so that we can use this from the JS API.

[niloc132]

deephaven/deephaven-core#5854 is ready for review from the JS API side.

[jnumainville]

Even with the JS API, there are still some open problems @mofojed and I am unsure how to proceed

1. The SharedTicket, as far as I can tell, only supports tables: publish_tables. This means that the chunk at 152 could not be removed because we would still need it for non-tables.
2. Additionally, it seems like the way we'll have to bind objects to tickets is through pydeephaven. This is not ideal in the embedded server case because we'll have to either create a session or force one to be passed.
3. I guess I could try to proceed with the enterprise fix at 142 but I'm not sure if that's worth starting considering only tables can be bound anyways... which we already have.

[devinrsmith]

1. This is something @jmao-denver may need to add support for; the underlying API supports other types as well. (java client api has io.deephaven.client.impl.Session#publish(HasTicketId, HasTicketId))

2. I think the concept of sharing must be tied to a session. It's possible we could create a "server local" SessionState, but we don't have anything like that atm. (It would also be possible to use the java session api as opposed to pydeephaven. I believe the requirements for interop w/ pydeephaven was a specific ask from DHE, but doesn't need to be limited to that here IMO). We may want to have the widget own a "jupyter local session" that could be managed by the widget code behind the scenes. But that said, we still probably want a server-local way to transfer an already created server-side object into a session's share.

To recap what we want (I think), in the case where we are running script session server w/ Jupyter:

# some server side table we create into a global scope
my_table = ...

# some server side table that is only reachable through ref to DeephavenWidget
DeephavenWidget(my_table.view(["SomeNewColumn=..."]))

In this case, we want some way to ensure that the reference to the derivative table is kept alive (I'm assuming we hand off that responsibility to jupyter JS side, although I don't know the specific python mechanism that will keep it alive; maybe DeephavenWidget python object is kept alive via some jupyter scoping?) And then, we want to hand off a proper shared reference.

cc @niloc132, @rcaudy, @nbauernfeind, we may need some concept of creating a shared ticket from a server-local object, or some other means of accomplishing this? (It's interesting to think about SessionState as a means of accomplishing server-side scoping in a more general fashion.)

[niloc132]

@devinrsmith after a quick call with joe, I think this is much easier - if you're running server side code, you're doing so via the grpc ConsoleService/ExecuteCommand call, and so there is an exec context you can grab that knows who the caller was, and can create server-side exports for you - no sharing required at all. The client then owns the export and can request it, release it, etc? This is basically how bidi plugins works (and how flightsql is expected to work), with the one extra bit that we need to communicate clearly to the client "okay, here's the ticket for the command you just ran".

That said, this doesn't belong in this ticket at all - we already have a functional (if entirely unsatisfying) way of getting tables and objects from a server-side jupyter session to a client, and the ticket is aimed at getting objects from a client-side jupyter session to another client.

[jnumainville]

I'm having a tough time wrapping my head around how I can use deephaven/deephaven-core#5854, I think because I'm not terribly experienced with the JS API. It seems like we need the actual logic to use that PR to pull within embed-widget in web-client-ui. Our ipywidget would pass the shared ticket uuid as a param to the iframe, where it's processed in embed-widget. Is there something I'm missing?

[devinrsmith]

if you're running server side code, you're doing so via the grpc ConsoleService/ExecuteCommand call,

In the case of executing a jupyter notebook, it's more like you are running in an interactive session with the jupyter kernel, and not through ExecuteCommand. Similarly, you could imaging getting rid of the jupyter abstraction and running via directly via python:

from deephaven_server import Server
server = Server()
server.start()

def my_func():
  my_table = ...
  my_shareable_ticket = some_func(my_table)
  print(my_shareable_ticket)
  block()
    
my_func() 

Of course, when you have something pre-defined like this, maybe application mode (and application scoped tickets) is really what we want; but I don't know if something like application mode makes sense for interactive jupyter notebook cells.

[mofojed]

From @niloc132 above:

If you're running server side code, you're doing so via the grpc ConsoleService/ExecuteCommand call, and so there is an exec context you can grab that knows who the caller was, and can create server-side exports for you - no sharing required at all

@niloc132, do you have any details on how to create an export ticket from the server side code? We'd want that ticket to be alive for the life of the DeephavenWidget object on the server, and then the client can fetch it.
Aiming to replace our "hack" where we assign a random object ID to the globals (polluting globals space):

deephaven-ipywidgets/deephaven_ipywidgets/deephaven.py

Line 151 in 270e598

# Add the table to the main modules globals list so it can be retrieved by the iframe

We should also be cleaning up that object after the DeephavenWidget is deleted (Jupyter keeps it around while it is being displayed in an output cell).

[devinrsmith]

We aren't going to be able to solve for that case right now afaik, see some of my earlier comments. I think we'll only be able to cleanly solve the pydeephaven.table.Table case right now (and the pydeephaven.<X> cases more generally once other exports can be shared).

deephaven-ipywidgets/deephaven_ipywidgets/deephaven.py

Lines 140 to 145 in 270e598

	elif _str_object_type(deephaven_object) == "pydeephaven.table.Table":
	session = deephaven_object.session
	server_url, token = _check_session(session, params)
	session.bind_table(object_id, deephaven_object)