Created samples for 'ProduceWordBags' and 'ProduceHashedWordBags' API. #3183
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #3183 +/- ##
==========================================
+ Coverage 72.54% 72.58% +0.04%
==========================================
Files 807 807
Lines 144774 144956 +182
Branches 16208 16212 +4
==========================================
+ Hits 105021 105223 +202
+ Misses 35339 35318 -21
- Partials 4414 4415 +1
|
Ivanidzo4ka
reviewed
Apr 3, 2019
| new TextData(){ Text = "This is an example to compute bag-of-word features using hashing." }, | ||
| new TextData(){ Text = "ML.NET's ProduceHashedWordBags API produces count of Ngrams and hashes it as an index into a vector of given bit length." }, | ||
| new TextData(){ Text = "It does so by first tokenizing text/string into words/tokens then " }, | ||
| new TextData(){ Text = "computing Ngram and hash them to the index given by hash value." }, |
There was a problem hiding this comment.
Ngram [](start = 50, length = 5)
Ngrams ? #Resolved
Ivanidzo4ka
reviewed
Apr 3, 2019
| new TextData(){ Text = "ML.NET's ProduceHashedWordBags API produces count of Ngrams and hashes it as an index into a vector of given bit length." }, | ||
| new TextData(){ Text = "It does so by first tokenizing text/string into words/tokens then " }, | ||
| new TextData(){ Text = "computing Ngram and hash them to the index given by hash value." }, | ||
| new TextData(){ Text = "The hashing schem reduces the size of the output feature vector" }, |
There was a problem hiding this comment.
schem [](start = 52, length = 5)
schema #Resolved
Ivanidzo4ka
reviewed
Apr 3, 2019
|
|
||
| // Expected output: | ||
| // Number of Features: 256 | ||
| // Features: 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 |
There was a problem hiding this comment.
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 [](start = 30, length = 78)
This is kinda anticlimactic.
Maybe change numberOfBits to smaller value to get some results different from zero here? #Resolved
Ivanidzo4ka
reviewed
Apr 3, 2019
| new TextData(){ Text = "ML.NET's ProduceWordBags API produces bag-of-word features from input text." }, | ||
| new TextData(){ Text = "It does so by first tokenizing text/string into words/tokens then " }, | ||
| new TextData(){ Text = "computing Ngram and their neumeric values." }, | ||
| new TextData(){ Text = "Each position in the output vector corresponds to a particular Ngram." }, |
There was a problem hiding this comment.
Ngram [](start = 103, length = 5)
https://en.wikipedia.org/wiki/N-gram
We call it NGram in code because you can't use - in variable, but for explanation maybe we should use n-gram? #Resolved
Ivanidzo4ka
reviewed
Apr 3, 2019
| new TextData(){ Text = "Each position in the output vector corresponds to a particular Ngram." }, | ||
| new TextData(){ Text = "The value at each position corresponds to," }, | ||
| new TextData(){ Text = "the number of times Ngram occured in the data (Tf), or" }, | ||
| new TextData(){ Text = "the inverse of the number of documents contain the Ngram (Idf), or." }, |
There was a problem hiding this comment.
or. [](start = 104, length = 3)
double or, plus dot in the end. I would omit this or. #Resolved
rogancarr
reviewed
Apr 4, 2019
| var transformedDataView = textTransformer.Transform(dataview); | ||
|
|
||
| // Create the prediction engine to get the bag-of-word features extracted from the text. | ||
| var predictionEngine = mlContext.Model.CreatePredictionEngine<TextData, TransformedTextData>(textTransformer); |
There was a problem hiding this comment.
Why predictionEngine rather than TakeRows and ConvertToEnumerable? I would use the one we would recommend people to use to inspect data in practice.
rogancarr
reviewed
Apr 4, 2019
|
|
||
| // Preview of the produced n-grams. | ||
| // Get the slot names from the column's metadata. | ||
| // If the column is a vector column the slot names corresponds to the names associated with each position in the vector. |
There was a problem hiding this comment.
If the column is a vector column [](start = 15, length = 32)
The If clause is confusing here. I would just drop it. #Resolved
rogancarr
approved these changes
Apr 4, 2019
zeahmed
added a commit
to zeahmed/machinelearning
that referenced
this pull request
Apr 8, 2019
ghost
locked as resolved and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related to #1209.