Filesystem based on SQLite

[hkrutzer]

Maybe this is a silly idea, but I was looking at deploying Livebook on Azure, which has a product called Azure Storage which includes blob storage. Unfortunately it is not S3 compatible. Now I could try to create a filesystem module that uses Azure Storage. Then I thought of Litestream which replicates SQLite databases to blob storages. Having a filesystem module that uses sqlite would allow it to be used with Litestream and replicate to Azure, Backblaze B2, and several others. Also, Fly.io can backup these databases with LiteFS, and hosted SQLite like turso.tech could be used as well, and there are probably more options. So perhaps this is some merit to this idea.

[josevalim]

This is an interesting question! I need to think more about it in general but one immediate downside I can think of is that we would need to add native dependencies to sqlite, which can potentially complicate how Livebook is deployed/distributed/installed.

If you are looking for using Azure, you can also add this: https://github.com/gaul/s3proxy - and we can always include direct support for Azure too.

[hkrutzer]

Direct support for Azure would also be great. I am aware of s3proxy but it is a Java application so I'm worried about high memory usage even though it won't serve many requests.

[josevalim]

We would definitely consider a pull request that adds Blob Storage support as a file system. :) Implementing the filesystem/storage behaviour in Livebook would be a great beginning. The rest of the changes are mostly UI (and I would not change it now because we are revamping the UI bits of that system).

[hkrutzer]

I saw S3 uses :httpc for HTTP calls, while Req is also a dependency. Is that intentional?

[josevalim]

A couple things:

1. We added Req later on :)

2. For now, Req is only used in the ZTA code, which we will move to an external dependency in the future

3. Other libraries, such as Explorer, may want to download from file storage, and those use httpc to not add additional dependencies, therefore writing the code with httpc will allow us to reuse it elsewhere

TL;DR: let's use httpc for this particular use case exclusively.

[hkrutzer]

Alright, cool. I understand the concerns wrt the sqlite dependency. If you decide it is acceptable or can be worked with, I would be interested in contributing there. I won't be creating a driver for Azure Blob Storage anytime soon so if anyone is reading this who wants to (perhaps unlikely because of the title of this discussion), feel free :)

[filmor]

@josevalim I was just looking into this (Azure Blob storage support). What's the current stance on additional dependencies? Would you still prefer an implementation that only uses the standard library?

[josevalim]

We are glad to support Azure Blob Storage and yes, the preference is still on minimal dependencies, because we pretty much use the same code in Explorer as well. In fact, you may want to get started with this in Explorer, depending on what you want to build.

• https://github.com/elixir-explorer/fss/
• https://github.com/elixir-explorer/explorer/blob/main/lib/explorer/fss/

Pretty much as a copy and paste of S3. This would allow you to read and write Explorer files directly to Azure. And then finally it can be a filesystem in Livebook too.

[filmor]

Just so that I understand: Is the plan to eventually use fss in livebook? I didn't see any directory listing support in fss.

[josevalim]

Not at the moment, because FSS is basically an exchange format between projects (such as Kino and Explorer) but the formats we use inside Livebook itself is not necessarily the same the user wants (e.g. Livebook can be compiled with v0.1.1 but the runtime uses v0.2.0).

This may change in the future if we want to provide a FSS implementation that performs operations such as directory listing, writing, etc. This implementation could be shared by Explorer and Livebook, but we are not there yet (at the moment we only share download, which is quite minimal).