HashingVectorizer fails when data has None values

[mrocklin]

Description

HashingVectorizer fails when data has None values, such as comes up with missing values. It could be that this is out of scope for HashingVectorizer and that users should handle this separately, but I didn't find a related issue on this topic (surprisingly) so thought I'd bring it up just in case.

Steps/Code to Reproduce

from sklearn.feature_extraction.text import HashingVectorizer
HashingVectorizer().transform(['a', 'b', None, 'c'])

Expected Results

A scipy.sparse matrix with None treated as another value, possibly 0 or maxval for interpretability.

Actual Results

It fails when preprocessing (traceback below) but presumably we would want to short-circuit this before it got to that point.

AttributeError                            Traceback (most recent call last)
<ipython-input-2-52ba2560fd22> in <module>()
----> 1 HashingVectorizer().transform(['a', 'b', None, 'c'])

~/workspace/scikit-learn/sklearn/feature_extraction/text.py in transform(self, X)
    601 
    602         analyzer = self.build_analyzer()
--> 603         X = self._get_hasher().transform(analyzer(doc) for doc in X)
    604         if self.binary:
    605             X.data.fill(1)

~/workspace/scikit-learn/sklearn/feature_extraction/hashing.py in transform(self, raw_X)
    166         indices, indptr, values = \
    167             _hashing_transform(raw_X, self.n_features, self.dtype,
--> 168                                self.alternate_sign)
    169         n_samples = indptr.shape[0] - 1
    170 

~/workspace/scikit-learn/sklearn/feature_extraction/_hashing.pyx in sklearn.feature_extraction._hashing.transform()

~/workspace/scikit-learn/sklearn/feature_extraction/hashing.py in <genexpr>(.0)
    163             raw_X = (_iteritems(d) for d in raw_X)
    164         elif self.input_type == "string":
--> 165             raw_X = (((f, 1) for f in x) for x in raw_X)
    166         indices, indptr, values = \
    167             _hashing_transform(raw_X, self.n_features, self.dtype,

~/workspace/scikit-learn/sklearn/feature_extraction/text.py in <genexpr>(.0)
    601 
    602         analyzer = self.build_analyzer()
--> 603         X = self._get_hasher().transform(analyzer(doc) for doc in X)
    604         if self.binary:
    605             X.data.fill(1)

~/workspace/scikit-learn/sklearn/feature_extraction/text.py in <lambda>(doc)
    306                                                tokenize)
    307             return lambda doc: self._word_ngrams(
--> 308                 tokenize(preprocess(self.decode(doc))), stop_words)
    309 
    310         else:

~/workspace/scikit-learn/sklearn/feature_extraction/text.py in <lambda>(x)
    254 
    255         if self.lowercase:
--> 256             return lambda x: strip_accents(x.lower())
    257         else:
    258             return strip_accents

AttributeError: 'NoneType' object has no attribute 'lower'

Versions

System:
    python: 3.6.4 |Anaconda custom (64-bit)| (default, Jan 16 2018, 18:10:19)  [GCC 7.2.0]
executable: /home/mrocklin/Software/anaconda/bin/python
   machine: Linux-4.13.0-26-generic-x86_64-with-debian-stretch-sid

BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: /home/mrocklin/Software/anaconda/lib
cblas_libs: mkl_rt, pthread

Python deps:
       pip: 9.0.1
setuptools: 40.4.3
   sklearn: 0.21.dev0
     numpy: 1.15.1
     scipy: 1.1.0
    Cython: 0.27.3
    pandas: 0.23.4

[jnothman]

Why are the options of `pd.fillna` and `SimpleImputer(strategy='constant')` and `HashingVectorizer.preprocessor = blah` insufficient?

[ogrisel]

HashingVectorizer is meant to be used as the first stage element of a pipeline

>>> HashingVectorizer().fit_transform(['a', 'b', 'c'])
<3x1048576 sparse matrix of type '<class 'numpy.float64'>'
	with 0 stored elements in Compressed Sparse Row format>

and in particular does not work with a 2 input array:

>>> HashingVectorizer().fit_transform(np.array(['a', 'b', 'c'], dtype=object).reshape(-
... 1, 1))
Traceback (most recent call last):
  File "<ipython-input-28-72984faf1f67>", line 1, in <module>
    HashingVectorizer().fit_transform(np.array(['a', 'b', 'c'], dtype=object).reshape(-1, 1))
  File "/home/ogrisel/code/scikit-learn/sklearn/feature_extraction/text.py", line 628, in fit_transform
    return self.fit(X, y).transform(X)
  File "/home/ogrisel/code/scikit-learn/sklearn/feature_extraction/text.py", line 603, in transform
    X = self._get_hasher().transform(analyzer(doc) for doc in X)
  File "/home/ogrisel/code/scikit-learn/sklearn/feature_extraction/hashing.py", line 168, in transform
    self.alternate_sign)
  File "sklearn/feature_extraction/_hashing.pyx", line 54, in sklearn.feature_extraction._hashing.transform
  File "/home/ogrisel/code/scikit-learn/sklearn/feature_extraction/hashing.py", line 165, in <genexpr>
    raw_X = (((f, 1) for f in x) for x in raw_X)
  File "/home/ogrisel/code/scikit-learn/sklearn/feature_extraction/text.py", line 603, in <genexpr>
    X = self._get_hasher().transform(analyzer(doc) for doc in X)
  File "/home/ogrisel/code/scikit-learn/sklearn/feature_extraction/text.py", line 308, in <lambda>
    tokenize(preprocess(self.decode(doc))), stop_words)
  File "/home/ogrisel/code/scikit-learn/sklearn/feature_extraction/text.py", line 256, in <lambda>
    return lambda x: strip_accents(x.lower())
AttributeError: 'numpy.ndarray' object has no attribute 'lower'

On the other hand SimpleImputer is not meant to work directly on a collection of unvectorized text content:

>>> SimpleImputer(strategy='constant', fill_value='missing').fit_transform(np.array(['a
... ', 'b', 'c'], dtype=object))
Traceback (most recent call last):
  File "<ipython-input-24-2c0810f23443>", line 1, in <module>
    SimpleImputer(strategy='constant', fill_value='missing').fit_transform(np.array(['a', 'b', 'c'], dtype=object))
  File "/home/ogrisel/code/scikit-learn/sklearn/base.py", line 462, in fit_transform
    return self.fit(X, **fit_params).transform(X)
  File "/home/ogrisel/code/scikit-learn/sklearn/impute.py", line 238, in fit
    X = self._validate_input(X)
  File "/home/ogrisel/code/scikit-learn/sklearn/impute.py", line 212, in _validate_input
    raise ve
  File "/home/ogrisel/code/scikit-learn/sklearn/impute.py", line 205, in _validate_input
    force_all_finite=force_all_finite, copy=self.copy)
  File "/home/ogrisel/code/scikit-learn/sklearn/utils/validation.py", line 547, in check_array
    "if it contains a single sample.".format(array))
ValueError: Expected 2D array, got 1D array instead:
array=['a' 'b' 'c'].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

So I believe is no natural way to combine those two in the right order.

The preprocessor option should work, although it might not necessarily be very intuitive for our users.

One could think about making the text vectorizers accept working with 2D input collections (several columns of text at once) but in this case I am not sure what semantics we want for the output feature space: should the output matrix have n_output_features = 1 x vocabulary size (meaning that all the text of all the columns be implicitly concatenated)? That would be unexpected in my opinion. The alternative would be to have one independent vectorizer instance per input text column. A third alternative would be to share the vocabulary of all the columns and have n_output_features = n_input_features * shared vocabulary size.

For reference our example that combines ColumnTransformer with text content passes explicitly a single column per vectorizer manually:

http://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer.html#sphx-glr-auto-examples-compose-plot-column-transformer-py

This might be tedious when we have many text columns and want to apply the same vectorization strategy to all of them.

@jorisvandenbossche might have an opinion on this.

[mrocklin]

Why are the options of pd.fillna and SimpleImputer(strategy='constant')
and HashingVectorizer.preprocessor = blah insufficient?

They're sufficient. I ended up going with pd.fillna after I found that this wasn't doable within HashingVectorizer. This felt like a rough edge though so I decided to raise it. I'd be entirely fine if it gets closed as "won't fix" or "not a bug".

One could think about making the text vectorizers accept working with 2D input collections (several columns of text at once)

I also ran into this

The alternative would be to have one independent vectorizer instance per input text column.

I ended up doing this with a column transformer as you suggest.

[rth]

HashingVectorizer().transform(['a', 'b', None, 'c'])
[...]
I ended up going with pd.fillna after I found that this wasn't doable within HashingVectorizer [..] I'd be entirely fine if it gets closed as "won't fix" or "not a bug".
[...]
A scipy.sparse matrix with None treated as another value, possibly 0 or maxval for interpretability.

Vectorizers assume that the input is string. I think making it accept None or NaN, would be more complicated maintenance wise than running fillna during pre-processing, particularly if the result is going to be the same.

Closing. Please comment in case of disagreement.