Rule for `svdvals`

[mcabbott]

Requested here: https://discourse.julialang.org/t/implementation-of-spectral-normalization-for-machine-learning/76074

The workaround is to call svd(X).S, which is slower forwards. But it looks like the gradient calculation with something like svd_rev((; U=NoTangent(), s=s, V=NoTangnet(), Vt=NoTangent()), NoTangent(), S̄, NoTangent()) is probably fairly efficient, and could easily be extracted to its own rule:

ChainRules.jl/src/rulesets/LinearAlgebra/factorization.jl

Lines 221 to 225 in 3590f94

	function rrule(::typeof(svd), X::AbstractMatrix{<:Real})
	F = svd(X)
	svd_pullback(ȳ) = _svd_pullback(ȳ, F)
	return F, svd_pullback
	end

[sethaxen]

I agree, yes, this should be its own rule. We already did that for Hermitian matrices.

[sethaxen]

ChainRules.jl/src/rulesets/LinearAlgebra/symmetric.jl

Lines 286 to 300 in ffbaa5f

	function rrule(::typeof(svdvals), A::LinearAlgebra.RealHermSymComplexHerm{<:BLAS.BlasReal,<:StridedMatrix})
	λ, back = rrule(eigvals, A)
	S = abs.(λ)
	p = sortperm(S; rev=true)
	permute!(S, p)
	function svdvals_pullback(ΔS)
	∂λ = real.(ΔS)
	invpermute!(∂λ, p)
	∂λ .*= sign.(λ)
	_, ∂A = back(∂λ)
	return NoTangent(), unthunk(∂A)
	end
	svdvals_pullback(ΔS::AbstractZero) = (NoTangent(), ΔS)
	return S, svdvals_pullback
	end

There I ended up having a custom rule. I don't remember why. It may have been faster than using NoTangent().

[sethaxen]

Ah, no, it goes via the rrule for eigvals, which uses the pullback for eigen. So yeah, just using the pullback for svd might be best.