Is it possible to serialize a 100GB data (vector of tables) into a single FlatBuffer?

[lee-junjie]

I appreciate the excellent work you’ve done on FlatBuffers.

I have a dataset that I’d like to serialize. It’s a vector of tables, and the total size is over 100GB (approximately 1000 items, each around 0.1GB).

I have a few questions that I hope you could help me with:

• Is it possible to serialize the entire dataset into a single FlatBuffer?
• Alternatively, do I need to divide the dataset into 1000 segments and serialize/deserialize each one individually?
• Do you have any other recommendations for achieving the fastest serialization/deserialization speed for my dataset?

Note: currently, offset64 doesn’t support vectors of tables. When I attempt to set offset64, I receive the following message: “only vectors of scalars are allowed to be 64-bit.”

Can you provide any guidance on this? Thanks in advance!