Cannot access blob data inserted with datajoint-python>=0.12 using mym 2.8.2

[kabilar]

Feature Request

Description

Cannot access blob data that are inserted with datajoint-python version >=0.12 using mym version 2.8.2.

Reproducibility

Access with mym version 2.8.2 does not work.

• OS MAC
• MATLAB 2020a
• MariaDB 10.2.33 (This is a copy of the tables used on the original database.)
• mYm 2.8.2
• DataJoint MATLAB 3.4.3

rec_schema = dj.Schema(dj.conn, 'recording', ['u19_recording'])
fetch(rec_schema.v.RecordingModality, '*')

Error using mym
Blob data ingested utilizing DataJoint-Python version >=0.12 not yet supported.

Thanks!

[kabilar]

Hi @Alvalunasan, one workaround is to insert NumPy data types, as Python native data types (e.g. lists) are not currently supported.

[stephenholtz]

Hi @kabilar, would you mind elaborating on how this allows you to avoid blob insertion in Python? Wouldn't you still need to put the NumPy data into a blob?

Any other suggestion would be greatly appreciated -- this seems to be workflow breaking for all sizeable matlab -> python data steps as far as I can tell.

[dimitri-yatsenko]

In datajoint-python 0.12, we added support for all python data types. The old serialization protocol was mYm and the new is called dj0. dj0 is a superset of mYm. So as long as you are only using the data types that were supported before 0.12, it stays in mYm and is matlab compatible. This includes all numpy numerical arrays. But if you start adding python-specific data types such as dict and str, it goes into dj0 and matlab will not be able to read them.

[dimitri-yatsenko]

You can prohibit python from using dj0 by setting dj.config['enable_python_native_blobs'] = False in python.

[stephenholtz]

Thank you @dimitri-yatsenko! So it is not all blobs inserted with new versions of datajoint-python >= 0.12, just those with some serialized python data types?

As long as I have that correct, I think it would be worth updating the error message in the next release to reflect that it is blob content and not the database datatype that is incorrect.

[dimitri-yatsenko]

That's correct. What's the current error message?

[stephenholtz]

I believe it is what is in the issue above, or something very similar.

Blob data ingested utilizing DataJoint-Python version >=0.12 not yet supported.

[dimitri-yatsenko]

Yes. A better error message would be something like, "Blobs with Python-specific data structures are not yet supported in DataJoint-Matlab."