unsigned int getFPReg(const llvm::MachineOperand&): Assertion `Reg >= X86::FP0 && Reg <= X86::FP6 && "Expected FP register!"

[RKSimon]

Bugzilla Link	41668
Version	trunk
OS	Windows NT
CC	@topperc,@RKSimon,@rnk,@rotateright

Extended Description

Reduced from https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=14508

source_filename = "llvm/test/Transforms/LoopVectorize/X86/vectorize-only-for-real.ll"
target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-macosx10.11.0"

define void @accum() #0 {
entry:
  %B6 = frem double 4.940660e-324, 0.000000e+00
  store double %B6, double* undef
  unreachable
}
attributes #0 = { "target-features"="+sse,-avx,-avx2,-sse2" }

llc: /home/simon/LLVM/llvm/lib/Target/X86/X86FloatingPoint.cpp:311: unsigned int getFPReg(const llvm::MachineOperand&): Assertion `Reg >= X86::FP0 && Reg <= X86::FP6 && "Expected FP register!"' failed.

[rotateright]

The underlying problem/workaround on this 1 appears to be that we don't have constant folding for an frem node in SDAG.

This gets folded in IR with -instsimplify for example.

[RKSimon]

SelectionDAG::foldConstantFPMath will handle FREM constant folding, but not if it will cause a APFloat::opDivByZero

[rotateright]

SelectionDAG::foldConstantFPMath will handle FREM constant folding, but not
if it will cause a APFloat::opDivByZero

Ah, the same restriction is on fdiv...and that seems wrong. If it's not a strict node, I think we should always fold it.

[topperc]

The failure here isn't caused by the constants. It's caused by calling fmod with double arguments on an sse1 only target. This fails too

define void @accum(double %x, double %y ) #0 {
entry:
  %B6 = frem double %y, %x
  store double %B6, double* undef
  unreachable
}
attributes #0 = { "target-features"="+sse,-avx,-avx2,-sse2" }

We ended up with an xmm0 to fp0 COPY instruction which isn't supported. I feel like I've seen something like this before.

[rotateright]

The failure here isn't caused by the constants. It's caused by calling fmod
with double arguments on an sse1 only target. This fails too

define void @​accum(double %x, double %y ) #​0 {
entry:
%B6 = frem double %y, %x
store double %B6, double* undef
unreachable
}
attributes #​0 = { "target-features"="+sse,-avx,-avx2,-sse2" }

We ended up with an xmm0 to fp0 COPY instruction which isn't supported. I
feel like I've seen something like this before.

We've definitely seen variations of this before - see related bugs.
But there were/are multiple problems visible in the original example:

1. https://reviews.llvm.org/rL359566
2. https://reviews.llvm.org/D61331
3. The general case with doubles w/o SSE2 (or any FP on x86-64 w/o SSE1).