Old JobSink secrets lead to processing old events again while new events are lost

[tikr7]

First of all, I really like the new feature JobSink in Knative Eventing because it helps to execute long running processes while being fully compatible with the eventing stack!

Describe the bug
JobSink Secret mounted to the Job is not cleaned up after deleting the Job.
There is even no update on that Secret with new CloudEvent information.
This leads to processing old events again and the new event is lost.

Expected behavior
Either clean up Kubernetes Secret after Kubernetes Job has been deleted.
Or overwrite the existing Kubernetes Secret with the new CloudEvent before it gets executed.

To Reproduce
Lets assume, I trigger 3 Jobs via eventing (names of Jobs are somehow autogenerated):

job-sink-logger
job-sink-logger0
job-sink-logger0xgzzy

Lets clean them up (imagine we have 1000 of those):

kubectl delete job job-sink-logger job-sink-logger0 job-sink-logger0xgzzy

We still have the Secrets with the CloudEvent inside:

kubectl get secret
job-sink-logger
job-sink-logger0
job-sink-logger0xgzzy

Now lets assume, I trigger 3 jobs via eventing again (names of Jobs are somehow autogenerated):

job-sink-logger0
job-sink-logger1z
job-sink-logger3mpyh

Secret job-sink-logger0 was not updated and still has the old CloudEvent, hence the Job job-sink-logger0 processes the old event and the new event never gets processed.

Knative release version v1.15.4 (I assume also v1.16.0)

Additional context
If something is unclear, just let me know.

[tikr7]

@pierDipi could you help here? :)