Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major refactor! Remove additional variation of the "ephys" modules! Support using SpikeInterface for spike sorting #201

Draft
wants to merge 198 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 185 commits
Commits
Show all changes
198 commits
Select commit Hold shift + click to select a range
ec9fcf3
first prototype in separating `Clustering` into multiple steps
Nov 30, 2022
c0d139f
Merge branch 'main' into modular_clustering
Nov 30, 2022
c88f0ef
Merge branch 'main' into modular_clustering
Dec 6, 2022
59c36f3
Merge branch 'main' into modular_clustering
Jan 5, 2023
f572438
Update ecephys_spike_sorting.py
Jan 5, 2023
edf1578
Update ecephys_spike_sorting.py
Jan 5, 2023
7e267c5
fix typo
Jan 5, 2023
6e7ddf1
Update ecephys_spike_sorting.py
Jan 6, 2023
7fd9bb4
improve log messages
Jan 6, 2023
26a56e7
fix key_source
Jan 6, 2023
654bc52
Update ecephys_spike_sorting.py
Jan 18, 2023
f75e14f
bugfix
Jan 19, 2023
a32d1d2
bugfix
Jan 19, 2023
4723bba
Merge branch 'main' into modular_clustering
Jan 20, 2023
ef0ad94
Merge branch 'main' into modular_clustering
Jan 20, 2023
3bea775
Update kilosort_triggering.py
Jan 20, 2023
4f955b3
fix docstring
Jan 23, 2023
53854d0
added description
Jan 23, 2023
3e9fc4b
Merge branch 'main' into modular_clustering
Jan 24, 2023
fe9955c
refactor `_supported_kilosort_versions`
Jan 25, 2023
aea325d
remove unused imports
Jan 25, 2023
4f648cc
add new file for spike interface modularized clustering approach
sidhulyalkar Feb 2, 2023
60091ac
add spike interface clustering and post processing modules
sidhulyalkar Feb 3, 2023
bca5fa9
edit typos
sidhulyalkar Feb 3, 2023
1c4b0b5
removed module_status from table keys
sidhulyalkar Feb 3, 2023
56c9941
remove _ from SI table names
sidhulyalkar Feb 3, 2023
ce14098
bugfix
sidhulyalkar Feb 3, 2023
7c836f1
change si related table names
sidhulyalkar Feb 3, 2023
dd63664
bugfix
sidhulyalkar Feb 3, 2023
54888ed
update initial comment
sidhulyalkar Feb 4, 2023
50a4f6c
Merge branch 'datajoint:main' into modular_spikeinterface
sidhulyalkar Feb 4, 2023
f804233
fix preprocessing file loading issues
sidhulyalkar Feb 9, 2023
8e1b73d
set file saving and file loading to pickle format
sidhulyalkar Feb 9, 2023
7520735
Merge branch 'modular_spikeinterface' of https://github.com/sidhulyal…
sidhulyalkar Feb 9, 2023
f0b7497
sglx preprocessing modifications
sidhulyalkar Feb 11, 2023
13fe31c
sglx testing progress
sidhulyalkar Feb 13, 2023
d41c7f3
wip parametrize preprocessing
sidhulyalkar Feb 15, 2023
109a71a
post processing waveform extractor extensions
sidhulyalkar Feb 18, 2023
1febd7e
post processing waveform extractor extensions
sidhulyalkar Feb 18, 2023
a478e06
Fix data loading bug related to cluster_groups and KSLabel df key
sidhulyalkar Feb 20, 2023
634761d
waveform extraction wip
sidhulyalkar Feb 22, 2023
ff0dfee
modification to handle spike interface waveforms
sidhulyalkar Feb 23, 2023
cb31229
adjust post processing
sidhulyalkar Feb 23, 2023
6098421
bugfix in postprocessing definition
sidhulyalkar Mar 6, 2023
6006464
add SI ibl destriping and catGT implementations
sidhulyalkar Mar 8, 2023
c050875
remove preprocess params list
sidhulyalkar Mar 8, 2023
4ea56c0
preprocessing changes
sidhulyalkar Mar 8, 2023
1e74457
Merge branch 'datajoint:main' into modular_spikeinterface
sidhulyalkar May 15, 2023
0a87579
Update requirements.txt
sidhulyalkar May 15, 2023
fb71cea
Merge branch 'datajoint:main' into modular_spikeinterface
sidhulyalkar Jun 13, 2023
22f1f65
fix spikeglx stream loading
sidhulyalkar Jun 14, 2023
5a0f624
Merge branch 'modular_spikeinterface' of https://github.com/sidhulyal…
JaerongA Aug 3, 2023
1a46369
Merge branch 'datajoint:main' into si
JaerongA Oct 10, 2023
b62f162
build: :pushpin: update requirements.txt & add env,.yml
JaerongA Oct 11, 2023
849b576
refactor: :art: clean up spikeinterface import & remove unused import
JaerongA Oct 11, 2023
7836a8b
modify key_source in PreProcessing
JaerongA Oct 13, 2023
6bee166
feat: :sparkles: improve to_probeinterface
JaerongA Dec 14, 2023
9ae6b44
create preprocessing.py
JaerongA Dec 15, 2023
9d5eee6
add SI_SORTERS , SI_READERS
JaerongA Dec 15, 2023
1fceb90
feat: :art: si_clustering.PreProcessing
JaerongA Dec 15, 2023
df8ed74
feat: :art: si_clustering.SIClustering
JaerongA Dec 20, 2023
2ed337b
feat: :sparkles: add PostProcessing table & clean up
JaerongA Dec 20, 2023
f6e3e46
fix: :bug: fix input/output data directory
JaerongA Jan 3, 2024
e1c0d68
check for presence of recording file
JaerongA Jan 3, 2024
653e7e8
fix: :bug: fix path & typo
JaerongA Jan 3, 2024
8c25bd2
code review
ttngu207 Jan 5, 2024
1084aae
Merge pull request #4 from ttngu207/spike-interface-patch-1
JaerongA Jan 5, 2024
fd5e9fa
Minor fixes
kushalbakshi Jan 10, 2024
90d1ba4
feat: :sparkles: modify QualityMetrics make function
JaerongA Feb 1, 2024
cacefac
update si_spike_sorting.PreProcessing make function
JaerongA Feb 6, 2024
f98e1ed
update SIClustering make function
JaerongA Feb 7, 2024
7a060ef
update PostProcessing make function
JaerongA Feb 7, 2024
6daf0c5
feat: :sparkles: add n.a. to ClusterQualityLabel
JaerongA Feb 7, 2024
e41ff1d
extract all waveforms
JaerongA Feb 7, 2024
e8d9854
feat: :sparkles: modify CuratedClustering make function for spike int…
JaerongA Feb 8, 2024
00b82f8
refactor: :recycle: import si module & re-organize imports
JaerongA Feb 8, 2024
b01c36c
update WaveformSet ingestion
JaerongA Feb 12, 2024
853b66f
Update element_array_ephys/ephys_no_curation.py
JaerongA Feb 13, 2024
67ebf4e
Update element_array_ephys/ephys_no_curation.py
JaerongA Feb 13, 2024
5fe6043
Update element_array_ephys/ephys_no_curation.py
JaerongA Feb 13, 2024
ac08163
ci: run test only on the main branch
JaerongA Feb 13, 2024
545aa92
Merge remote-tracking branch 'upstream/datajoint-spikeinterface' into…
JaerongA Feb 13, 2024
e95331c
build: :heavy_plus_sign: add spikingcircus dependencies
JaerongA Feb 13, 2024
be5135e
refactor: fix typo & black formatting
JaerongA Feb 13, 2024
4b6fc0e
feat: :sparkles: add EphysRecording.Channel part table
JaerongA Feb 15, 2024
4802511
fix: :bug: fix get_logger missing error
JaerongA Feb 16, 2024
bca67b3
fix typo & remove sorter_name
JaerongA Feb 16, 2024
1faa8f2
feat: :sparkles: add memoized_result implementation in SIClustering
JaerongA Feb 16, 2024
58f3a44
create a folder for storing recording pickle object
JaerongA Feb 16, 2024
4fcea51
install element_interface from datajoint upstream
JaerongA Feb 16, 2024
4f0e020
add required_key for parameters
JaerongA Feb 16, 2024
83e7a16
set recording channel info
JaerongA Feb 16, 2024
1d18a39
fix loading preprocessor
JaerongA Feb 16, 2024
b0a863f
make all output dir non-sharable
JaerongA Feb 16, 2024
7d28351
refactor & accept changes from code review
JaerongA Feb 19, 2024
95f5286
remove memoized_result for testing
JaerongA Feb 19, 2024
a7ebb9a
build: :heavy_plus_sign: add element-interface to required packages
JaerongA Feb 20, 2024
134ff54
update pre-commit with the latest hooks
JaerongA Feb 21, 2024
7972426
build: :heavy_plus_sign: Add numba as required package
JaerongA Feb 27, 2024
ed11526
adjust extract_waveforms parameters
JaerongA Feb 27, 2024
bab86b7
refactor: :recycle: update the output dir for CuratedClustering
JaerongA Feb 29, 2024
0898ce5
feat: :sparkles: add quality label mapping
JaerongA Feb 29, 2024
727af24
feat: :sparkles: Ingest EphysRecording.Channel
JaerongA Mar 1, 2024
1df41ea
get channel to electrode mapping in CuratedClustering
JaerongA Mar 1, 2024
417219f
refactor: :recycle: Fix metrics directory in QualityMetrics
JaerongA Mar 1, 2024
f70ae4e
feat: :sparkles: replace get_neuropixels_channel2electrode_map with c…
JaerongA Mar 1, 2024
0ccbec9
fix CuratedClustering make function
JaerongA Mar 1, 2024
86bb8d7
improve try except logic
JaerongA Mar 1, 2024
673faed
docs: :memo: update comments in ephys_no_curation
JaerongA Mar 1, 2024
8ab4c58
refactor: :recycle: improve if else block in EphysRecording
JaerongA Mar 1, 2024
226142b
feat: :sparkles: Update WaveformSet make function
JaerongA Mar 1, 2024
ea39839
refactor: :fire: remove & get_neuropixels_channel2electrode_map and g…
JaerongA Mar 5, 2024
5bfe201
Update element_array_ephys/ephys_no_curation.py
JaerongA Mar 5, 2024
1d80584
add generate_electrode_config_name
JaerongA Mar 5, 2024
88ce139
refactor: :recycle: change sorter_name
JaerongA Mar 7, 2024
7eaefa4
address review comments for generate_electrode_config_entry
JaerongA Mar 8, 2024
90b81e3
Merge pull request #179 from JaerongA/datajoint-spikeinterface
Mar 8, 2024
d47be56
refactor: :art: refactor PostProcessing
JaerongA Mar 11, 2024
8dfc858
chore: :art: run docker if the package is not built into spikeinterface
JaerongA Mar 11, 2024
6e20a11
refactor: :recycle: clean up import & docstring
JaerongA Mar 11, 2024
8d04e10
revert: :art: replace SI_READERS with si_extractor
JaerongA Mar 11, 2024
37a35d3
Merge branch 'datajoint:datajoint-spikeinterface' into datajoint-spik…
JaerongA Mar 12, 2024
bb39194
fix acq_software name
JaerongA Mar 12, 2024
528fcf9
Merge branch 'datajoint-spikeinterface' of https://github.com/Jaerong…
JaerongA Mar 12, 2024
01ff816
feat: :ambulance: make all secondary attributes nullable in QualityMe…
JaerongA Mar 12, 2024
cd65539
Merge pull request #184 from JaerongA/datajoint-spikeinterface
Mar 12, 2024
aff15bf
Merge remote-tracking branch 'upstream/datajoint-spikeinterface' into…
ttngu207 Apr 4, 2024
67a1ffc
feat: save spike interface results with relative path
ttngu207 Apr 17, 2024
41e4d6d
Merge branch 'datajoint:main' into main
kushalbakshi Apr 18, 2024
d44dbaa
fix(spikeglx): bugfix loading spikeglx data
ttngu207 Apr 26, 2024
1819cbc
Merge remote-tracking branch 'upstream/datajoint-spikeinterface' into…
ttngu207 Apr 26, 2024
d86928b
fix: bugfix inserting `ElectrodeConfig`
ttngu207 Apr 26, 2024
6a6e7b7
Merge pull request #186 from ttngu207/datajoint-spikeinterface
MilagrosMarin Apr 26, 2024
f8ffd77
feat(spikesorting): save to phy and generate report
ttngu207 Apr 3, 2024
7309082
chore: cleanup init
ttngu207 Apr 26, 2024
d778b1e
fix: update channel-electrode mapping
ttngu207 Apr 26, 2024
015341c
feat: test spikeinterface for spikeglx data
ttngu207 Apr 26, 2024
05ccfdb
fix: update ingestion from spikeinterface results
ttngu207 Apr 29, 2024
c819d19
Merge pull request #187 from ttngu207/datajoint-spikeinterface
MilagrosMarin Apr 29, 2024
93895a9
Refactor Quality Metrics Logic + blackformatting
MilagrosMarin May 7, 2024
bd3bb8e
Update si_spike_sorting.py
ttngu207 May 8, 2024
c8882c6
Merge branch 'datajoint-spikeinterface' of https://github.com/datajoi…
ttngu207 May 8, 2024
403d1df
update `postprocessing` logic
MilagrosMarin May 9, 2024
bdf2b9f
Merge remote-tracking branch 'upstream/datajoint-spikeinterface' into…
ttngu207 May 14, 2024
c934e67
feat: prototyping with the new `sorting_analyzer`
ttngu207 May 24, 2024
3666cda
feat: update ingestion to be compatible with spikeinterface 0.101+
ttngu207 May 24, 2024
06a3f70
format: black formatting
ttngu207 May 24, 2024
07a09f6
chore: code clean up
ttngu207 May 24, 2024
3fcf542
update: update requirements to install `SpikeInterface` from github (…
ttngu207 May 24, 2024
76dfc94
fix: minor bug in spikes ingestion
ttngu207 May 28, 2024
9094754
update: bump version
ttngu207 May 28, 2024
51e2ced
feat: add `memoized_result` on spike sorting
ttngu207 May 28, 2024
2a4b166
Merge pull request #190 from ttngu207/dev_spikeinterface_v101
MilagrosMarin May 28, 2024
0afb452
chore: minor code cleanup
ttngu207 May 28, 2024
e8f445c
fix: merge fix & formatting
ttngu207 May 28, 2024
6155f13
fix: calling `_run_sorter()`
ttngu207 May 28, 2024
f6a52d9
chore: more robust channel mapping
ttngu207 May 29, 2024
92547d7
Merge pull request #191 from ttngu207/dev_spikeinterface_v101
ttngu207 May 29, 2024
1ff92dd
fix: use relative path for phy output
ttngu207 May 29, 2024
b459709
feat: in data ingestion, set peak_sign="both"
ttngu207 May 29, 2024
e497a15
Merge pull request #192 from ttngu207/dev_spikeinterface_v101
MilagrosMarin May 29, 2024
1a1b18f
feat: replace `output_folder` with `folder` when calling `run_sorter`…
ttngu207 Jun 3, 2024
212a54f
Merge pull request #193 from ttngu207/dev_spikeinterface_v101
kushalbakshi Jun 4, 2024
4e645eb
fix: remove `job_kwargs` for sparsity calculation - memory error in l…
ttngu207 Jun 5, 2024
82d73c9
Merge pull request #194 from ttngu207/dev_spikeinterface_v101
ttngu207 Jun 5, 2024
38fdfb2
feat: separate `export` (phy and report) into a separate table
ttngu207 Jul 5, 2024
a4a8380
fix: export default to `False`
ttngu207 Jul 5, 2024
5a31632
Merge pull request #195 from ttngu207/dev_spikeinterface_v101
ttngu207 Jul 5, 2024
1f05998
fix: `spikes` object no longer available from `ComputeSpikeLocations`…
ttngu207 Jul 5, 2024
a22561a
Merge pull request #196 from ttngu207/datajoint-spikeinterface
ttngu207 Jul 5, 2024
7cd8ac8
chore: code cleanup
ttngu207 Jul 6, 2024
c87e493
fix: recording_extractor_full_dict is deprecated (https://github.com/…
ttngu207 Jul 6, 2024
dcfd880
Merge pull request #197 from ttngu207/datajoint-spikeinterface
ttngu207 Jul 6, 2024
097d9bb
fix: bugfix spikeinterface extractor name
ttngu207 Jul 30, 2024
6aca235
Merge remote-tracking branch 'upstream/datajoint-spikeinterface' into…
ttngu207 Aug 15, 2024
b6f131b
update: element-interface `main` branch
ttngu207 Aug 15, 2024
84c38d1
Merge branch 'main' into datajoint-spikeinterface
ttngu207 Aug 15, 2024
6b14486
Merge branch 'main' into datajoint-spikeinterface
ttngu207 Aug 15, 2024
c2f2a79
Merge branch 'main' into datajoint-spikeinterface
ttngu207 Aug 15, 2024
726a2bb
Merge branch 'main' into datajoint-spikeinterface
ttngu207 Aug 15, 2024
5dc5953
Merge branch 'main' into datajoint-spikeinterface
ttngu207 Aug 16, 2024
b510cb8
Merge pull request #200 from ttngu207/datajoint-spikeinterface
kushalbakshi Aug 19, 2024
ccd23fc
rearrange(all): major refactor of modules
ttngu207 Sep 10, 2024
0eef1cb
rearrange: remove the `ecephys_spike_sorting` flow
ttngu207 Sep 10, 2024
c2bd5ad
chore: clean up diagrams
ttngu207 Sep 10, 2024
4971108
fix: use tempfile.TemporaryDirectory
ttngu207 Sep 19, 2024
a975353
Merge remote-tracking branch 'upstream/main' into dev_new_main
ttngu207 Sep 19, 2024
63df4cd
format: black
ttngu207 Sep 19, 2024
3ab94dd
Merge branch 'main' into datajoint-spikeinterface
ttngu207 Sep 20, 2024
dc8bac1
Merge branch 'main' into dev_new_main
ttngu207 Sep 20, 2024
9056db8
Merge branch 'main' of https://github.com/datajoint/element-array-ephys
kushalbakshi Sep 20, 2024
de49729
Merge branch 'dev_new_main' of https://github.com/ttngu207/element-ar…
kushalbakshi Sep 20, 2024
800d060
Update docs for new ephys release
kushalbakshi Sep 20, 2024
463ec52
feat(spike_sorting): handle cases when no units/spikes are found
ttngu207 Oct 8, 2024
451571d
feat(spike_sorting): update downstream ephys tables ingestion when NO…
ttngu207 Oct 8, 2024
b1104ce
fix(spike_sorting): create empty `sorting_analyzer` folder when no un…
ttngu207 Oct 9, 2024
bb27fd0
Merge branch 'datajoint-spikeinterface' into dev_new_main
ttngu207 Oct 10, 2024
40dc3c0
Merge pull request #13 from kushalbakshi/dev_new-main-docs
ttngu207 Oct 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/test.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
name: Test
on:
push:
branches:
- main
pull_request:
branches:
- main
workflow_dispatch:
jobs:
devcontainer-build:
Expand Down Expand Up @@ -31,4 +35,3 @@ jobs:
run: |
python_version=${{matrix.py_ver}}
black element_array_ephys --check --verbose --target-version py${python_version//.}

8 changes: 4 additions & 4 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ exclude: (^.github/|^docs/|^images/)

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
rev: v4.5.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
Expand All @@ -16,7 +16,7 @@ repos:

# black
- repo: https://github.com/psf/black
rev: 22.12.0
rev: 24.2.0
hooks:
- id: black
- id: black-jupyter
Expand All @@ -25,15 +25,15 @@ repos:

# isort
- repo: https://github.com/pycqa/isort
rev: 5.11.2
rev: 5.13.2
hooks:
- id: isort
args: ["--profile", "black"]
description: Sorts imports in an alphabetical order

# flake8
- repo: https://github.com/pycqa/flake8
rev: 4.0.1
rev: 7.0.0
hooks:
- id: flake8
args: # arguments to configure flake8
Expand Down
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,21 @@
Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
[Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.


## [1.0.0] - 2024-09-10

+ Update - No longer support multiple variation of ephys module, keep only `ephys_no_curation` module, renamed to `ephys`
+ Update - Remove other ephys modules (e.g. `ephys_acute`, `ephys_chronic`) (moved to different branches)
+ Update - Add support for `SpikeInterface`
+ Update - Remove support for `ecephys_spike_sorting` (moved to a different branch)
+ Update - Simplify the "activate" mechanism

## [0.4.0] - 2024-08-16

+ Add - support for SpikeInterface version >= 0.101.0 (updated API)
+ Add - feature for memoization of spike sorting results (prevent duplicated runs)


## [0.3.5] - 2024-08-16

+ Fix - Improve `spikeglx` loader in extracting neuropixels probe type from the meta file
Expand Down
4 changes: 3 additions & 1 deletion element_array_ephys/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
from . import ephys_acute as ephys
from . import ephys

ephys_no_curation = ephys # alias for backward compatibility
Loading
Loading