Skip to content

[AutoDiff] inout parameter differentiation mega-patch. #29956

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[AutoDiff] inout parameter differentiation mega-patch. #29956

wants to merge 2 commits into from

Conversation

dan-zheng
Copy link
Contributor

@dan-zheng dan-zheng commented Feb 20, 2020 •
edited
Loading

tensorflow branch mega-patch for TF-129: inout parameter differentiation.

This patch exists for end-to-end testing. Code will be submitted incrementally.

Current status:

  • Derivative registration works for stdlib functions with inout parameters, like Float.+= and Array.append.
  • End-to-end inout parameter differentiation works with no known issues.

@dan-zheng
[AutoDiff] Type-checking support for inout parameter differentiation.
b7fa3db 
Semantically, an `inout` parameter is both a parameter and a result.

`@differentiable` and `@derivative` attributes now support original functions
with one "semantic result": either a formal result type or an `inout` parameter.

The differential/pullback type of a function with `inout` differentiability
parameters also has `inout` parameters. This is ideal for performance.

Differential typing rules:
- Case 1: no `inout` parameters.
  - Original:     `(T0, T1, ...) -> R`
  - Differential: `(T0.Tan, T1.Tan, ...) -> R.Tan`
- Case 2: original function has a non-wrt `inout` parameter.
  - Original:     `(T0, inout T1, ...) -> Void`
  - Differential: `(T0.Tan, ...) -> T1.Tan`
- Case 3: original function has a wrt `inout` parameter.
  - Original:     `(T0, inout T1, ...) -> R`
  - Differential: `(T0.Tan, inout T1.Tan, ...) -> Void`

Pullback typing rules:
- Case 1: original function has no `inout` parameters.
  - Original: `(T0, T1, ...) -> R`
  - Pullback: `R.Tan -> (T0.Tan, T1.Tan, ...)`
- Case 2: original function has a non-wrt `inout` parameter.
  - Original: `(T0, inout T1, ...) -> Void`
  - Pullback: `(T1.Tan) -> (T0.Tan, ...)`
- Case 3: original function has a wrt `inout` parameter.
  - Original: `(T0, inout T1, ...) -> R`
  - Pullback: `(inout T1.Tan) -> (T0.Tan, ...)`

Resolves TF-1164.
@dan-zheng
[WIP]
831aea2 
Resolves TF-1165: differentiation support for `inout` parameters.
Add derivatives for stdlib functions with `inout` parameters.

Todo: add more tests.
@dan-zheng dan-zheng added the tensorflow This is for "tensorflow" branch PRs. label Feb 20, 2020
@dan-zheng dan-zheng mentioned this pull request Feb 20, 2020
Merged
@compnerd compnerd closed this Feb 22, 2020
@dan-zheng
Copy link
Contributor Author

I accidentally deleted tensorflow branch, which closed this PR. That was not intentional, sorry!
It would be nice to protect tensorflow branch against deletion to prevent this from happening again.

@dan-zheng dan-zheng reopened this Feb 22, 2020
@dan-zheng
Copy link
Contributor Author

Obsolete. Landed in tensorflow branch as #30013.

@dan-zheng dan-zheng closed this Feb 23, 2020
@dan-zheng dan-zheng deleted the autodiff-inout-parameters-tf branch February 23, 2020 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
tensorflow This is for "tensorflow" branch PRs.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dan-zheng @compnerd