Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add memoized_result + update calcium imaging utility functions #121

Merged
merged 113 commits into from
Aug 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
113 commits
Select commit Hold shift + click to select a range
da03b82
support loading multiple caiman results for multi-plane
Aug 16, 2023
9d365bf
Merge branch 'main' into multi-plane-caiman
Aug 18, 2023
9bf69ff
new caiman loader - handles multi-plane results
Aug 20, 2023
05d7c35
improve non-rigid motion correction loading
Aug 20, 2023
b050265
added routine to combine single-page tiffs into one bigtiff
Aug 22, 2023
512e5af
run_caimain - more robust
Aug 22, 2023
7d6ea0e
add `caiman_compatible` mode
Aug 22, 2023
55e5a5c
minor cleanup, version bump
Aug 24, 2023
cb08034
Update caiman_loader.py
Aug 25, 2023
85078fd
Fix regex error
kushalbakshi Aug 28, 2023
7b815ff
bugfix
Aug 28, 2023
0bf78ed
Resolve TypeError: Add `.stem()`
kushalbakshi Aug 28, 2023
706df04
Debug `FileNotFoundError` at output_dir
kushalbakshi Aug 28, 2023
5a99468
Merge pull request #2 from kushalbakshi/multi-plane-caiman
Aug 29, 2023
df0d9b8
Raise exception to debug summary image dimensions
kushalbakshi Aug 29, 2023
13e6e94
Revert previous commit
kushalbakshi Aug 29, 2023
b76429d
Merge branch 'main' into multi-plane-caiman
Sep 5, 2023
a2cf2b2
Merge pull request #88 from ttngu207/multi-plane-caiman
kushalbakshi Sep 5, 2023
20188ad
Update CHANGELOG for #88
kushalbakshi Sep 5, 2023
697e804
add `validation` argument to dandi upload
Sep 6, 2023
3b4b61c
Merge branch 'multi-plane-caiman' of https://github.com/kushalbakshi/…
kushalbakshi Sep 13, 2023
7e5be44
Check for z_fields in 2nd cycle of acqusition
kushalbakshi Sep 13, 2023
20d5418
Merge pull request #89 from kushalbakshi/multi-plane-caiman
Sep 13, 2023
b374a41
passing as argument to `fit_file`
ttngu207 Oct 12, 2023
7e26b82
Merge pull request #90 from ttngu207/multi-plane-caiman
kushalbakshi Oct 12, 2023
a558909
Fix typo indicies -> indices
kushalbakshi Oct 16, 2023
f14366d
Merge pull request #91 from kushalbakshi/multi-plane-caiman
sidhulyalkar Oct 16, 2023
23e14b4
Update params -> parameters
kushalbakshi Oct 16, 2023
a562033
Merge pull request #92 from kushalbakshi/multi-plane-caiman
sidhulyalkar Oct 16, 2023
fff8aba
Set `indices=None` in `fit_file()` + comments for clarity
kushalbakshi Oct 23, 2023
72c4feb
Merge branch 'multi-plane-caiman' of https://github.com/datajoint/ele…
kushalbakshi Oct 23, 2023
1efb4d2
Fix typos
kushalbakshi Oct 23, 2023
32d667f
Merge pull request #94 from kushalbakshi/multi-plane-caiman
Oct 23, 2023
fb95db7
Merge pull request #96 from datajoint/multi-plane-caiman
Nov 28, 2023
da0ef6b
fix(prairie_view_loader): fix file search for single-plane dataset
ttngu207 Jan 30, 2024
7de446f
Merge pull request #98 from ttngu207/multi-plane-caiman
kushalbakshi Jan 30, 2024
d217fe8
Merge remote-tracking branch 'upstream/staging' into multi-plane-caiman
Feb 1, 2024
eb4ac75
fix(prairie_view_loader): fix handling of `num_frames` and plane sear…
Feb 1, 2024
daa7df6
Merge pull request #100 from ttngu207/multi-plane-caiman
kushalbakshi Feb 1, 2024
b2368a1
Merge pull request #99 from datajoint/multi-plane-caiman
Feb 1, 2024
b3f6829
Merge branch 'main' into dev_memoized_results
Feb 2, 2024
66fa788
Bugfix - remove hardcoded `n_fields` from `prairie_view_loader.py`
kushalbakshi Feb 16, 2024
3ce2989
Merge pull request #101 from kushalbakshi/staging
Feb 16, 2024
a2b8f9c
Merge pull request #102 from ttngu207/dev_memoized_results
JaerongA Feb 16, 2024
46d8df4
Add meaningful exception error in `write_single_bigtiff()`
kushalbakshi Feb 19, 2024
4226910
Update element_interface/prairie_view_loader.py
kushalbakshi Feb 20, 2024
8579eba
Merge pull request #103 from kushalbakshi/staging
Feb 20, 2024
84b4b1e
Catch expection for caiman compatible files in prairie_view_loader
kushalbakshi Feb 20, 2024
be1cc87
Merge pull request #104 from kushalbakshi/staging
Feb 20, 2024
6747d8a
Add `Exception` to `raise` statements
kushalbakshi Feb 21, 2024
1c9b392
Merge pull request #105 from kushalbakshi/staging
kushalbakshi Feb 21, 2024
012e838
Merge branch 'main' into dev_memoized_results
ttngu207 Mar 20, 2024
11a1a0f
Revert "Merge branch 'main' into dev_memoized_results"
ttngu207 Mar 20, 2024
f4d8478
format: minor reformatting of the docstring
ttngu207 Mar 20, 2024
cd9f2b7
Merge branch 'staging' of https://github.com/datajoint/element-interf…
ttngu207 Apr 17, 2024
deff0fa
fix{prairie_view_loader): bugfix framerate
ttngu207 Apr 18, 2024
d7b98be
Merge branch 'staging' into dev_prairieview_bigtif
ttngu207 Apr 18, 2024
3ad3ed4
Check for bruker-generated multi_tiff
kushalbakshi Apr 23, 2024
b5803e7
feat: handle PrairieView new multi-page tif format
ttngu207 May 20, 2024
83649ba
Merge branch 'main' into dev_prairieview_bigtif
ttngu207 May 20, 2024
ff80a7c
Merge pull request #108 from ttngu207/dev_prairieview_multipagetif
kushalbakshi May 22, 2024
a7663bc
Merge branch 'main' into dev_prairieview_multipagetif
ttngu207 May 23, 2024
8ac8c00
Merge branch 'main' into dev_memoized_results
ttngu207 May 28, 2024
f96d8be
update: minor var name change
ttngu207 May 28, 2024
8bd3dcb
Merge pull request #111 from ttngu207/dev_memoized_results
ttngu207 May 28, 2024
5f036ff
Merge branch 'staging' of https://github.com/datajoint/element-interf…
kushalbakshi Jun 4, 2024
1f3ab21
Update(run_caiman.py) use single thread to avoid multiprocessing error
kushalbakshi Jun 4, 2024
ed2228c
Merge pull request #107 from kushalbakshi/staging
ttngu207 Jun 4, 2024
3fc7c17
Fix(run_caiman.py): backend for cluster
kushalbakshi Jun 4, 2024
1e06ac8
Revert backend to "multiprocessing"
kushalbakshi Jun 4, 2024
53502c0
Fix(run_caiman.py) Remove "single_thread" argument from setup_cluster
kushalbakshi Jun 4, 2024
e71eb37
Merge pull request #112 from kushalbakshi/staging
ttngu207 Jun 4, 2024
647142f
Update(run_caiman.py) logic to handle caiman output files
kushalbakshi Jun 5, 2024
12a79de
Run shutil with full path
kushalbakshi Jun 5, 2024
f78acfc
Copy files instead of move out of temp_dir
kushalbakshi Jun 5, 2024
843ac6d
Fix(caiman_loader.py): pass generator obj as list to np.dstack
kushalbakshi Jun 5, 2024
ddf8a83
Merge pull request #113 from kushalbakshi/staging
ttngu207 Jun 5, 2024
50e8977
Use memmap arrays to generate bigtiffs
kushalbakshi Jun 20, 2024
24d8afa
Fix tiff file generation for caiman compatability
kushalbakshi Jun 20, 2024
4b3fc89
Remove `ome` kwarg
kushalbakshi Jun 20, 2024
d39ca6c
Add(prairie_view_loader.py) pyvips to process ome.tif files
kushalbakshi Jun 21, 2024
2a69303
Fix typo
kushalbakshi Jun 21, 2024
7045aa0
Fix(prairie_view_loader.py) set `bigtiff=True` in pyvips
kushalbakshi Jun 21, 2024
3d4d4ca
Fix(prairie_view_loader.py) - remove pyvips, add tiffwriter code from…
kushalbakshi Jun 24, 2024
188954a
Fix(prairie_view_loader.py) append to existing tiff if one already ex…
kushalbakshi Jun 24, 2024
990f6b1
fix(run_caiman): update `cnmf_output_file`
ttngu207 Jun 26, 2024
c189c48
Merge branch 'staging' of https://github.com/datajoint/element-interf…
ttngu207 Jun 26, 2024
bc548de
Merge branch 'staging' into dev_prairieview_multipagetif
ttngu207 Jun 26, 2024
dbec434
update: deprecate "caiman_compatible" argument
ttngu207 Jun 26, 2024
eff0d7b
Merge pull request #114 from ttngu207/dev_prairieview_multipagetif
kushalbakshi Jun 26, 2024
315c19a
fix: additional safeguard to close the file and delete the `tffl` object
ttngu207 Jun 28, 2024
02962d3
Merge branch 'staging' of https://github.com/datajoint/element-interf…
ttngu207 Jun 28, 2024
e025b29
Merge pull request #115 from ttngu207/dev_prairieview_multipagetif
kushalbakshi Jun 28, 2024
f3aea53
feat: add `use_cuda` for caiman
ttngu207 Jul 6, 2024
288ae92
Merge branch 'staging' of https://github.com/datajoint/element-interf…
kushalbakshi Jul 10, 2024
86f1f85
Merge pull request #116 from ttngu207/dev_use_cuda_flag
kushalbakshi Jul 11, 2024
e9e17f3
Merge branch 'dev_memoized_results' into staging
ttngu207 Jul 19, 2024
436412d
Merge remote-tracking branch 'upstream/dev_memoized_results' into sta…
ttngu207 Jul 19, 2024
4df8a08
Merge branch 'staging' of https://github.com/datajoint/element-interf…
ttngu207 Jul 19, 2024
e2cfc6a
Merge remote-tracking branch 'upstream/main' into staging
ttngu207 Jul 19, 2024
b525c8c
feat: set `gb_per_file` and write multiple bigtiff
ttngu207 Jul 23, 2024
dc98ebc
chore: add warning with `gb_per_file` for `is_multipage` tiff
ttngu207 Jul 23, 2024
ebb6adc
Merge pull request #117 from ttngu207/dev_write_multiple_bigtiff
ttngu207 Jul 23, 2024
e38352e
fix: remove old tif files if `overwrite=True`
ttngu207 Jul 24, 2024
0b3b306
fix: incorrect dimension ordering for single-plane caimain loading
ttngu207 Jul 24, 2024
57a8158
Merge pull request #118 from ttngu207/dev_write_multiple_bigtiff
kushalbakshi Jul 24, 2024
eb5254d
feat(run_caiman): use `output_dir` for `CAIMAN_TEMP`
ttngu207 Aug 6, 2024
19a0d27
Merge pull request #119 from ttngu207/dev_write_multiple_bigtiff
kushalbakshi Aug 6, 2024
f633cd8
Merge branch 'staging' of https://github.com/datajoint/element-interf…
kushalbakshi Aug 9, 2024
686bed6
Update docs, GHA, version, CHANGELOG
kushalbakshi Aug 9, 2024
bcd6386
Apply black formatting
kushalbakshi Aug 9, 2024
76c27b0
Remove notebooks from GHA checks
kushalbakshi Aug 9, 2024
d7b6211
Fix version + CHANGELOG
kushalbakshi Aug 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/mkdocs-release-caller.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
name: mkdocs-release
on:
workflow_dispatch:

jobs:
mkdocs_release:
uses: datajoint/.github/.github/workflows/mkdocs_release.yaml@main
permissions:
contents: write
10 changes: 10 additions & 0 deletions .github/workflows/release.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
name: Release
on:
workflow_dispatch:
jobs:
make_github_release:
uses: datajoint/.github/.github/workflows/make_github_release.yaml@main
mkdocs_release:
uses: datajoint/.github/.github/workflows/mkdocs_release.yaml@main
permissions:
contents: write
7 changes: 7 additions & 0 deletions .github/workflows/semantic-release-caller.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
name: semantic-release
on:
workflow_dispatch:

jobs:
call_semantic_release:
uses: datajoint/.github/.github/workflows/semantic-release.yaml@main
35 changes: 35 additions & 0 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Test
on:
push:
pull_request:
workflow_dispatch:
schedule:
- cron: "0 8 * * 1"
jobs:
devcontainer-build:
uses: datajoint/.github/.github/workflows/devcontainer-build.yaml@main
tests:
runs-on: ubuntu-latest
strategy:
matrix:
py_ver: ["3.9", "3.10"]
mysql_ver: ["8.0", "5.7"]
include:
- py_ver: "3.8"
mysql_ver: "5.7"
- py_ver: "3.7"
mysql_ver: "5.7"
steps:
- uses: actions/checkout@v3
- name: Set up Python ${{matrix.py_ver}}
uses: actions/setup-python@v4
with:
python-version: ${{matrix.py_ver}}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flake8 "black[jupyter]"
- name: Run style tests
run: |
python_version=${{matrix.py_ver}}
black element_interface --check --verbose --target-version py${python_version//.}
17 changes: 0 additions & 17 deletions .github/workflows/u24_element_before_release.yaml

This file was deleted.

28 changes: 0 additions & 28 deletions .github/workflows/u24_element_release_call.yaml

This file was deleted.

14 changes: 0 additions & 14 deletions .github/workflows/u24_element_tag_to_release.yaml

This file was deleted.

10 changes: 9 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,18 @@
Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
[Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.

## [0.7.0] - 2024-08-09

+ Add - `memoized_result` decorator to cache function results
+ Update - `prairie_view_loader.py` to create big tiff files from `.ome.tif` files
+ Update - `run_caiman.py` to run latest version of CaImAn
+ Update - `caiman_loader.py` to process output of latest version of CaImAn
+ Fix - general fixes and improvements

## [0.6.1] - 2023-08-02

+ Update DANDI upload funtionality to improve useability


## [0.6.0] - 2023-07-26

+ Update - `prairieviewreader.py` -> `prairie_view_loader.py`
Expand Down Expand Up @@ -83,6 +89,8 @@ Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and

+ Add - Readers for: `ScanImage`, `Suite2p`, `CaImAn`.


[0.7.0]: https://github.com/datajoint/element-interface/releases/tag/0.7.0
[0.6.0]: https://github.com/datajoint/element-interface/releases/tag/0.6.0
[0.5.4]: https://github.com/datajoint/element-interface/releases/tag/0.5.4
[0.5.3]: https://github.com/datajoint/element-interface/releases/tag/0.5.3
Expand Down
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
MIT License

Copyright (c) 2021 DataJoint NEURO
Copyright (c) 2024 DataJoint

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
22 changes: 15 additions & 7 deletions docs/src/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,12 @@ how to use various Elements.
`utils.str_to_bool` converts a set of strings to boolean True or False. This is implemented
as the equivalent item in Python's `distutils` which will be removed in future versions.

`utils.memoized_result` is a decorator that caches the result of a function call based
on input parameters and the state of the output. If the function is called with the same
parameters and the output files in the directory remain unchanged, it returns the
cached results; otherwise, it executes the function and caches the new results along
with metadata.

### Suite2p

This Element provides functions to independently run Suite2p's motion correction,
Expand All @@ -46,13 +52,15 @@ Requirements:

### PrairieView Reader

This Element provides a function to read the PrairieView Scanner's metadata file. The
PrairieView software generates one `.ome.tif` imaging file per frame acquired. The
metadata for all frames is contained in one `.xml` file. This function locates the
`.xml` file and generates a dictionary necessary to populate the DataJoint ScanInfo and
Field tables. PrairieView works with resonance scanners with a single field, does not
support bidirectional x and y scanning, and the `.xml` file does not contain ROI
information.
This Element provides a `PrairieViewMeta` class to handle different types of output from
the PrairieView Scanner. The PrairieView software either generates one `.ome.tif`
imaging file per frame acquired or multi-page `.ome.tif` files. The metadata for all
frames is contained in one `.xml` file. This class contains methods that locate the
`.xml` file and generate a dictionary necessary to populate the DataJoint ScanInfo and
Field tables. The class also contains methods to create a big tiff file from the
individual `.ome.tif` files. PrairieView works with resonance scanners with a single
field, does not support bidirectional x and y scanning, and the `.xml` file does not
contain ROI information.

## Element Architecture

Expand Down
2 changes: 2 additions & 0 deletions docs/src/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,6 @@ a number of other Elements.

- Data ingestion, see [`ingest_csv_to_table` API](./api/element_interface/utils/#element_interface.utils.ingest_csv_to_table)

- Code execution, see [`memoized_result` API](./api/element_interface/utils/#element_interface.utils.memoized_result)

Visit the [Concepts page](./concepts.md) for more information on these tools.
Loading
Loading