Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DRAFT] First version of fusion optimizations for transformers #1938

Draft
wants to merge 25 commits into
base: main
Choose a base branch
from
Draft

[DRAFT] First version of fusion optimizations for transformers #1938

wants to merge 25 commits into from
+543 −0

Commits on Oct 23, 2024

  1. Fusions

    @gramalingam
    gramalingam committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    d751d37 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. MultiHeadAttention fusion

    @gramalingam
    gramalingam committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    8c4dff5 View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2024

  1. Merge branch 'main' into rama/fusions

    @gramalingam
    gramalingam committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    5d3c9af View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2024

  1. Move transformers optimization into onnxruntime folder

    @gramalingam
    gramalingam committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    4d3ff90 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2024

  1. Support some SDPA variations

    @gramalingam
    gramalingam committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    4a667f9 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2024

  1. Add variations of rules for SDPA

    @gramalingam
    gramalingam committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    33c3753 View commit details
    Browse the repository at this point in the history

  2. Add attention scale validation

    @gramalingam
    gramalingam committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    404e5c3 View commit details
    Browse the repository at this point in the history

  3. Add validation conditions for rotary embedding

    @gramalingam
    gramalingam committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    e98682f View commit details
    Browse the repository at this point in the history

  4. Add tests

    @gramalingam
    gramalingam committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    001bb59 View commit details
    Browse the repository at this point in the history

  5. Move into new xformers folder

    @gramalingam
    gramalingam committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    40b9052 View commit details
    Browse the repository at this point in the history

  6. Add dropout to optimizer

    @gramalingam
    gramalingam committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    94ce2f3 View commit details
    Browse the repository at this point in the history

  7. Run lint

    @gramalingam
    gramalingam committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    bf3b64a View commit details
    Browse the repository at this point in the history

  8. Undo dropout rewrite rule change

    @gramalingam
    gramalingam committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    0491366 View commit details
    Browse the repository at this point in the history

  9. Add concat test

    @gramalingam
    gramalingam committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    3fb7cd1 View commit details
    Browse the repository at this point in the history

  10. Merge with main

    @gramalingam
    gramalingam committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    f25b669 View commit details
    Browse the repository at this point in the history

  11. Add expand identity optimization

    @gramalingam
    gramalingam committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    73723f0 View commit details
    Browse the repository at this point in the history

  12. Some cleanup

    @gramalingam
    gramalingam committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    bb977ec View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2024

  1. Fix dropout optimization

    @gramalingam
    gramalingam committed Nov 9, 2024
    Configuration menu
    Copy the full SHA
    7f1606f View commit details
    Browse the repository at this point in the history

  2. Some more cleanup

    @gramalingam
    gramalingam committed Nov 9, 2024
    Configuration menu
    Copy the full SHA
    a3e0d1d View commit details
    Browse the repository at this point in the history

  3. Cleanup

    @gramalingam
    gramalingam committed Nov 9, 2024
    Configuration menu
    Copy the full SHA
    0879934 View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2024

  1. Minor fixes

    @gramalingam
    gramalingam committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    a8ac3ee View commit details
    Browse the repository at this point in the history

  2. Add ort check to test

    @gramalingam
    gramalingam committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    044a638 View commit details
    Browse the repository at this point in the history

  3. Testing changes

    @gramalingam
    gramalingam committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    b6f0071 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2024

  1. Merge branch 'main' into rama/fusions

    @gramalingam
    gramalingam committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    b985bb1 View commit details
    Browse the repository at this point in the history

  2. Merge with main

    @gramalingam
    gramalingam committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    0f35c45 View commit details
    Browse the repository at this point in the history