[New Operator] FusedRowwiseQuantizedSparseLengthsWeightedSumNode

[jfix71]

Description: As noted in #2292, we decided to implement both fused and unfused versions of rowwise-quantized SLWS.

Testing: Added OperatorTests and Caffe2ImporterTests.

Documentation: Added.

Closes #1698

[jfix71]

Note that Caffe2 does not have an explicit "fused Int8" fill/type, so when the Caffe2Importer loads the Constant data input it is loaded as normal Int8QTy. During loading a FusedRowwiseQuantizedSparseLengthsWeightedSumNode, it assumes that its data input was previously loaded as Int8QTy, and then changes the type to Int8FusedQTy instead.

If there are other users of the same Constant data that are expecting it to be Int8QTy then it will have an incorrect type. However I don't expect this to be the case, and it doesn't make sense that other users would be unaware that it's fused data.

[yinghai]

What's the status of this one? Is it ready for review?

[jfix71]

@yinghai Yep!

[yinghai]

LGTM. But I'll let Glow experts accept. :)

[rdzhabarov]

Few questions.

[rdzhabarov]

This is great work!

[yinghai]

Why do we have such a type instead of plain Int8ITy?

[jfix71]

The biggest reason I wanted to explicitly differentiate between normal Int8QTy and Int8FusedQTy is because the shape for Int8FusedQTy is very strange -- it appears to be 8 columns wider than the actual data. This prevents accidental interpretation of fused scales/offsets as instead data, or viewing the input data as an otherwise normal tensor. And I didn't think there was a large downside to adding the new type.

[yinghai]

It should be uint8?

[jfix71]

We subtract OFFSETSHIFT to convert to int8_t in our importer:

glow/lib/Importer/Caffe2ModelLoader.cpp

Lines 1118 to 1127 in a64d64f

	// Although in Caffe2 quantized model, the weights is int8 quantized,
	// the weights is stored in uint8_t format due to that Caffe2 requires
	// the type of input and weights must be the same. Therefore, we need to
	// convert it to int8 by subtracting 128.
	T->reset(ElemKind::Int8QTy, dim, scale, offset - OFFSETSHIFT);
	auto TH = T->getHandle<int8_t>();
	std::string str = dict["values"]->s();
	for (; i < str.size(); i++) {
	TH.raw(i) = ((uint8_t)(str.c_str()[i]) - OFFSETSHIFT);
	}