Whisper with word-level timestamps

[dgoryeo]

Hi @MahmoudAshraf97 , thanks again for great work. I was wondering: is there a way for this implmentation to take advantage of word-level timestamps in the latest version of Whisper to improve output accuracy?
Thanks

[MahmoudAshraf97]

Hi @dgoryeo , the reason I didn't rely on whisper timestamps was because they are inaccurate not because it didn't provide word timestamps as this had solutions by other contributors, it's fairly easy to do this though as long as the result from whisper has the same format as whisperX, the rest of the code should need minimal changes but this needs user testing to verify the quality of results for both approaches

[dgoryeo]

Thanks for the quick reply @MahmoudAshraf97 . Would you have any plans to add Japanese to your supported languages? I am quite keen on the approach you've implemented --it has all elements that I think make a good quality output, but the alignment step is not as accurate for Japanese. That's actually the genesis of my interest in word-level timestamps, with a hope can make the solution more universal.

[MahmoudAshraf97]

I'm thinking of using whisper word timestamps for the languages that are not supported by whisperX, it will not be the most accurate diarization but it's better than no diarization at all

[MahmoudAshraf97]

@dgoryeo Hi, whisper timestamps are now used for languages that are not supported by whisperX

[dgoryeo]

Updated:

Hi @MahmoudAshraf97 , I tried the colab notebooks. I first used the notebook in Readme, then I noticed the newer notebook in the code directory:

• Notebook in the Readme: notebook1
• Notebook in the code: notebook2

With notebook2 (which I guess is the latest one, right?), I cannot pass the following import/install error:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
[<ipython-input-12-99c1ac25fe61>](https://localhost:8080/#) in <cell line: 11>()
      9 import librosa
     10 import soundfile
---> 11 from nemo.collections.asr.models.msdd_models import NeuralDiarizer
     12 from deepmultilingualpunctuation import PunctuationModel
     13 import re

5 frames
[/usr/local/lib/python3.10/dist-packages/nemo/core/classes/exportable.py](https://localhost:8080/#) in <module>
     16 
     17 import torch
---> 18 from pytorch_lightning.core.module import _jit_is_scripting
     19 
     20 from nemo.core.classes import typecheck

ModuleNotFoundError: No module named 'pytorch_lightning.core.module'

---------------------------------------------------------------------------

I also tried to run the Readme notebook (notebook1), but I get the following error when executing the line alignment_model, metadata = whisperx.load_align_model(language_code=whisper_results["language"], device=device) :

OSError                                   Traceback (most recent call last)
[/usr/lib/python3.10/urllib/request.py](https://localhost:8080/#) in do_open(self, http_class, req, **http_conn_args)
   1347             try:
-> 1348                 h.request(req.get_method(), req.selector, req.data, headers,
   1349                           encode_chunked=req.has_header('Transfer-encoding'))

21 frames
OSError: [Errno 99] Cannot assign requested address

During handling of the above exception, another exception occurred:

URLError                                  Traceback (most recent call last)
[/usr/lib/python3.10/urllib/request.py](https://localhost:8080/#) in do_open(self, http_class, req, **http_conn_args)
   1349                           encode_chunked=req.has_header('Transfer-encoding'))
   1350             except OSError as err: # timeout error
-> 1351                 raise URLError(err)
   1352             r = h.getresponse()
   1353         except:

URLError: <urlopen error [Errno 99] Cannot assign requested address>

Any pointers how to work around the errors? Thanks.