-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add fuse layers for conv+affine+relu and conv+relu #2842
Add fuse layers for conv+affine+relu and conv+relu #2842
Conversation
|
I'm traveling these days, so it might take some take to have a look. I'll try during this week, though. I also thought about fusing the relu layer, so it's nice so see it here. |
I thought it was |
One other thing, while reviewing this problem, I was looking for the documentation of "disable_duplicative_biases" function and I couldn't find it. Turns out it was left in |
Oh yeah, thanks. That fixes the links in the main docs too :) |
Huh yeah, the GPU version of, Anyway, yeah, looks like your PR is good :) |
Oh, I need to check what's going on there |
This PR extends the fuse_layers method to also fuse convolutional layers followed by relu, and convolutional layers follower by affine followed by relu.
I tested it with
examples/dnn_mmod_face_detection_ex.cpp
and got exactly the same results on the images shown.Here are also some tests for different neural networks from https://github.com/dlibml/dnn:
comparative1: measures how much faster the model run with the new fuse_layers compared to the current fuse_layers.
comparative2: measures how much faster the model run with the new fuse_layers compared to not using fuse_layers.
I left the measurements before the fusion to be sure that the different runs used the same convolutional algorithms in cuDNN. Every time they differed by more than 2% up or down, I ran those tests again.