Skip to content

[GraphOptimizer] Fix wrong layout assumption in OptimizeReduceMean? #3499

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shajrawi opened this issue Sep 10, 2019 · 3 comments
Open

[GraphOptimizer] Fix wrong layout assumption in OptimizeReduceMean? #3499

shajrawi opened this issue Sep 10, 2019 · 3 comments

Comments

@shajrawi
Copy link
Contributor

I see the following code in the optimization:

      // In Glow, AvgPool expects NHWC.
      auto *TR1 = F->createTranspose(
          RM->getName().str() + ".transposeNCHW2NHWC", in, NCHW2NHWC);
      auto *AP = F->createAvgPool(RM->getName().str() + ".avgPool", TR1,
                                  kernels, strides, pads);
      auto *TR2 = F->createTranspose(
          RM->getName().str() + ".transposeNHWC2NCHW", AP, NHWC2NCHW);

This looks like a bug to me, the canonical representation in Glow is NHWC. Why do we expect the input to be in NCHW format and doing the two transposes?

@shajrawi
Copy link
Contributor Author

cc @vuzelac-cadence

shajrawi added a commit to shajrawi/glow that referenced this issue Nov 6, 2019
Fixes pytorch#3452

Also Fixes pytorch#3493 and pytorch#3500 GraphOptimizer bugs which were found after adding the layout verifier.

Provides a workaround for the pytorch#3499 issue which was also found via the verifier.

Note: I did not want to break the `enum ConvolutionLayout` introduced in 5074a72, As such, I used it in the verifier / did not change the creation of said nodes.
HOWEVER: We should use the more-generic string-based layout, which I introduce to Transpose node in this commit: it is basically an extendable enum that can be used in the backends without touching the generic code base. as a bonus, it makes differentiation easier: see how it is done for transpose now in `Function *glow::differentiate`.

Getting rid of said enum is a proposed TODO / follow-up.

Also note that some nodes *need* layout requirements, which have been added, namely we need to know the layout for placeholders and constants (obviously) and for reshapes (in case we optimized a transpose into a reshape.

An additional nice-to-have feature of the string-based layout is the wildcard / any-layout option. Some operations, such as data parallel nodes, might accept any layout.

A potential follow-up is to get create a "Solver" that automatically inserts transposes if the layouts do not match, this might greatly simplify the loader: we no longer need to insert transposes based on if we are importing NHWC or NCHW (for example). We just need to annotate the placeholder with the layout information we've get at load-time, and which we "forget" afterwards.

The verifier is useful even without creating said solver, it exposed a couple of bugs which are mentioned in this commit, as such any proposed solvers are not a must-have to demonstrate the usefulness of this commit.
facebook-github-bot pushed a commit that referenced this issue Nov 13, 2019
…#3503)

Summary:
Note: I did not want to break the `enum ConvolutionLayout` introduced in 5074a72, As such, I used it in the verifier / did not change the creation of said nodes.
HOWEVER: We should use the more-generic string-based layout, which I introduce to Transpose node in this commit: it is basically an extendable enum that can be used in the backends without touching the generic code base. as a bonus, it makes differentiation easier: see how it is done for transpose now in `Function *glow::differentiate`.

Getting rid of said enum is a proposed TODO / follow-up.

Also note that some nodes *need* layout requirements, which have been added, namely we need to know the layout for placeholders and constants (obviously) and for reshapes (in case we optimized a transpose into a reshape).

An additional nice-to-have feature of the string-based layout is the wildcard / any-layout option. Some operations, such as data parallel nodes, might accept any layout.

A potential follow-up is to get create a "Solver" that automatically inserts transposes if the layouts do not match, this might greatly simplify the loader: we no longer need to insert transposes based on if we are importing NHWC or NCHW (for example). We just need to annotate the placeholder with the layout information we've get at load-time, and which we "forget" afterwards.

The verifier is useful even without creating said solver, it exposed a couple of bugs which are mentioned in this commit, as such any proposed solvers are not a must-have to demonstrate the usefulness of this commit.

Fixes #3452

Also Fixes #3493 and Fixes #3500 GraphOptimizer bugs which were found after adding the layout verifier.

Provides a workaround for the #3499 issue which was also found via the verifier.
Pull Request resolved: #3503

Test Plan: `ninja test`

Differential Revision: D18357369

Pulled By: shajrawi

fbshipit-source-id: 45f91fbe120b234c2a85879cee9ee0de6c100b50
vdantu pushed a commit to vdantu/glow that referenced this issue Jul 12, 2020
…pytorch#3503)

Summary:
Note: I did not want to break the `enum ConvolutionLayout` introduced in 5074a72, As such, I used it in the verifier / did not change the creation of said nodes.
HOWEVER: We should use the more-generic string-based layout, which I introduce to Transpose node in this commit: it is basically an extendable enum that can be used in the backends without touching the generic code base. as a bonus, it makes differentiation easier: see how it is done for transpose now in `Function *glow::differentiate`.

Getting rid of said enum is a proposed TODO / follow-up.

Also note that some nodes *need* layout requirements, which have been added, namely we need to know the layout for placeholders and constants (obviously) and for reshapes (in case we optimized a transpose into a reshape).

An additional nice-to-have feature of the string-based layout is the wildcard / any-layout option. Some operations, such as data parallel nodes, might accept any layout.

A potential follow-up is to get create a "Solver" that automatically inserts transposes if the layouts do not match, this might greatly simplify the loader: we no longer need to insert transposes based on if we are importing NHWC or NCHW (for example). We just need to annotate the placeholder with the layout information we've get at load-time, and which we "forget" afterwards.

The verifier is useful even without creating said solver, it exposed a couple of bugs which are mentioned in this commit, as such any proposed solvers are not a must-have to demonstrate the usefulness of this commit.

Fixes pytorch#3452

Also Fixes pytorch#3493 and Fixes pytorch#3500 GraphOptimizer bugs which were found after adding the layout verifier.

Provides a workaround for the pytorch#3499 issue which was also found via the verifier.
Pull Request resolved: pytorch#3503

Test Plan: `ninja test`

Differential Revision: D18357369

Pulled By: shajrawi

fbshipit-source-id: 45f91fbe120b234c2a85879cee9ee0de6c100b50
@dspmihai
Copy link

dspmihai commented Aug 4, 2021

Will this code be ever executed? I see in lowerBatchedReduceMeanNode that ReduceMean can only have one axis. @jfix71, @mciprian13 , any idea how it's better to approach this again? I am thinking of lowering ReduceMean to AvgPool in Lower.cpp, if possible. Or maybe lower to multiple ReduceMean, then find the pattern in OptimizeReduceMean (possible error-prone after future optimizations)?

@vuzelac-cadence
Copy link
Contributor

@dspmihai, lowering is backend controlled. For example, we do use this optimization.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants