Different output when changing last few samples of input

[EliasLum]

Hey there,

I recently tested some models using asteroid. At some point I figured out that if i change the last few samples of my input audio, this affects the whole output (even the first samples). This was a bit puzzling, since i was using a model that was supposably causal in time and had no time trespassing component (e.g. SuDoRMRF).

I am wondering why this is happening, i was testing several models and on all models the same happens (see attached code). Could this be due to normalisation inside the model?

I tested this with some models. In the code below, there is no model state loaded, but the behaviour is the same, if a state is loaded.

import torch
import numpy as np
import matplotlib.pyplot as plt
from asteroid.models.conv_tasnet import ConvTasNet
from asteroid.models.lstm_tasnet import LSTMTasNet as TasNet
from asteroid.models.dprnn_tasnet import DPRNNTasNet
from asteroid.models.sudormrf import SuDORMRFImprovedNet as SuDO
from asteroid.models.dptnet import DPTNet
from asteroid.models.dcunet import DCUNet
from asteroid.models.dccrnet import DCCRNet
import copy

sr = 16000
model = DCUNet('DCUNet-10', fix_length_mode ="pad", sample_rate=sr)
#model = SuDO(n_src=1, sample_rate=sr, upsampling_depth=1)
#model = DCCRNet(architecture="DCCRN-CL", sample_rate=sr)
#model = DPTNet(n_src=1, sample_rate=sr)
#model = DPRNNTasNet(n_src=1, sample_rate=sr)
#model = TasNet(n_src=1, sample_rate=sr)
#model = ConvTasNet(n_src=1, sample_rate=sr)

model1 = copy.copy(model)
model2 = copy.copy(model)

# create random input
audio = torch.rand(1,1,56000)
input1 = np.copy(audio)
input2 = np.copy(audio)
# change last elements of one of the input
input2[0,0,-1000:] = torch.rand(1000)

# separate
output1 = model1.separate(input1)
output2 = model2.separate(input2)

fig, ax = plt.subplots(4, figsize=(12,12))
ax[0].plot(output1[0,0,:])
ax[0].set_xlim(0,56000)
ax[0].set_title("output 1")
ax[1].plot(output2[0,0,:])
ax[1].set_xlim(0,56000)
ax[1].set_title("output 2")


ax[2].plot(output1[0,0,:]-output2[0,0,:])
ax[2].set_ylim(-0.02,0.02)
ax[2].set_xlim(0,56000)
ax[2].set_title("diff")


ax[3].plot(output1[0,0,:]/output2[0,0,:])
ax[3].set_ylim(0.5,1.5)
ax[3].set_xlim(0,56000)
ax[3].set_title("quotient")
plt.tight_layout()
plt.show()

[mpariente]

Thanks for opening this subject. Can you add screenshots of the plots please for everyone to see without the code? Thanks!

[EliasLum]

Thanks for the swift answer. Of course, see attached the output of the script executed for the DCUNet model. As shown in the code, the only difference in input1 and input2 are the last 1000 samples

.

[EliasLum]

Okay i think i found the problem. For training purposes most models perform batch normalization which takes into account future values for normalization.
When running files for testing or evaluation I should not use the same model as for training, i should use:
model = mode.eval()

Does that sound right?

[mpariente]

Sorry for the late reply, I'm not sure to understand what's happening, have you gone forward with the debugging? How does it go with model.eval()?

[EliasLum]

No problem.
As mentioned above the model contains batch normalization. At the evaluation stage it is important to set batch normalization and dropout to evaluation-mode as mentioned in the pytorch documentation:

Remember that you must call model.eval() to set dropout and batch normalization layers to evaluation mode before running inference. Failing to do this will yield inconsistent inference results.

The two things that model.eval() or the equivalent model.train(False) automatically take care of are:

• normalization layers use running statistics (e.g. running averages).
• dropout layers are de-activated

So in my case, where I missed to set the model to evaluation-mode the normalization would be calculated based on the whole audio file, which meant that it looked like being non causal -> The output of the model was affected by all future content of the audio file. Switching the model to eval-mode, solves this issue and we get the output as expected.

In any case, bottom line is to always call model.eval() for evaluation purposes, which is a trivial but important detail in hindsight.

[mpariente]

In any case, bottom line is to always call model.eval() for evaluation purposes, which is a trivial but important detail in hindsight.

Yep, important detail indeed !

Thanks for confirming this solved your issue.