Where is package loading disabled in the codebase?

[rikhuijzer]

A question for the Pluto developers. I'm running the following:

using Pluto: Cell, Pluto, Notebook, ServerSession, update_save_run!

function run_notebook!(nb::Notebook, session)
    cells = [nb.cells_dict[cell_uuid] for cell_uuid in nb.cell_order]
    for cell in cells
        update_save_run!(session, nb, cell; run_async=false, save=false)
        if cell.errored
            body = cell.output.body
            msg = body[:msg]
            stacktrace = body[:stacktrace]
            msg = """
                $msg

                Details:
                $stacktrace
                """
            error(msg)
        end
    end
    return nothing
end

session = ServerSession()
session.options.evaluation.workspace_use_distributed = false

nb = Notebook([
    Cell("""(using Pkg; Pkg.add("TOML");)"""),
    Cell("x = 1")
])
run_notebook!(nb, session)

Normally, run_notebook! works fine, but as soon as anything with packages is added, it crashes with weird errors (see also rikhuijzer/PlutoStaticHTML.jl#42). The code above crashes with:

LoadError: ArgumentError: Package Pkg not found in current path:
  - Run `import Pkg; Pkg.add("Pkg")` to install the Pkg package.

So, I think this is done by PlutoRunner, but I'm unsure where exactly. Which code of Pluto handles the disabling of Pkg?

[fonsp]

Do you have a stack trace? Did you set any configurations to different values (like workspace_use_distributed=false)?

[rikhuijzer]

I'm so stupid. The main idea of this question was "where is package loading disabled in the codebase when workspace_use_distributed=false" but I didn't type that...

I've updated the code above, but cannot reproduce the issue. Let's consider this solved for now.

[fonsp]

workspace_use_distributed=false is not really meant to be used for regular use, I don't think plutopkg is enabled there, it would be hard to implement and test.

You can open an issue if it's important?

[rikhuijzer]

For anyone running into this discussion, I've had a talk with Fons and he helped me out. The problem of run_notebook! above is that it runs the cells in the order that they are displayed in the notebook and not in the topological order, so that is asking for trouble basically because Pluto does things like involved macro evaluation and that kind of stuff. Instead, use PlutoRunner.open. For the immediate error handling that run_notebook! does, an alternative would be to run Pluto async, look for cells going into error status and shutdown the notebook as soon as that happens.

For the use_distributed=false part. It may is useful for PlutoStaticHTML.jl especially in the context of Documenter.jl builds inside CI jobs because CI jobs have only 2 threads available and don't want to compile Julia from scratch for each notebook. In that case, it is required to make sure that package loading is done in the main module before starting to evaluate the notebook. Or at least do not mutate global things like packages and imports in one notebook and expect it to not affect the others. If things run in the same process, it will affect each other. So, this is basically the same restrictions as normally with using Documenter eval.