Cluster returning "groups" of 1 row/choice

[emiraglia]

All of the cluster methods return clusters with one row/choice, which takes up processing time and makes using anything beyond ngram-fingerprint nearly impossible for larger sets.

Desktop (please complete the following information):

• Windows 10
• Browser Version: Chrome and Firefox

OpenRefine (please complete the following information):

• Version 3.2

[thadguidry]

First, any use of Clustering feature uses quite a bit of memory.
Try to increase the amount of memory that you allocate to OpenRefine. Follow our guide here:
https://github.com/OpenRefine/OpenRefine/wiki/FAQ:-Allocate-More-Memory

Have you tried different Clustering methods and functions such as kNN and Levenshtein?
Please review our wiki page that explains more in depth how to use our Clustering feature.
https://github.com/OpenRefine/OpenRefine/wiki/Clustering-In-Depth

My suggestion is perhaps to use Levenshtein and then start with an edit distance of 6 (the average length of most English words) and then try increasing or decreasing that edit distance value.

[thadguidry]

You might also be expecting Clustering to find duplicates for you, which it sort of can do, but depending on your use case, you might find our Duplicates facet in the Column menu in what you are looking for.
If you want to treat your entire row (all columns) to look for Duplicates, then you can Concatenate all the columns into a new column and then run the Duplicates facet on that new concatenated column.

[emiraglia]

I've tried other methods and ones that worked in earlier versions are no longer working, even on smaller datasets because I'm getting the results shown above. I have never had earlier versions return thousands of clusters with 1 row/choice in them. If I have a set of 4300 rows, I shouldn't get getting 4100 clusters, right?

I'm not using Clustering to find duplicates. I'm using it to hopefully identify items that should be duplicates, but aren't because of spelling errors or diacritics or other things that the clustering methods can address.

[emiraglia]

Just for reference, when I use the exact same clustering method on the same dataset with version 3.1 I get this:

Which is much more in line with what I'd expect, and uses far less memory.

[thadguidry]

I see no problem in 3.2 where you state that "All of the cluster methods return clusters with one row/choice"

There are clusters found with multiple rows using OpenRefine 3.2
3.1 version had several regressions and in hindsight we should not have released it.
So let's focus on 3.2 and expectations there. Thanks.

So... here's what I get with fingerprint and Levenshtein with just 5 row test set that you can see on the grid additional to the Clustering dialog...

And here's a slightly altered test set grid to make sure Fingerprint is actually doing what it is designed to do per our spec on the Wiki and our code and test cases...

[ostephens]

I see the same problem as @emiraglia - the clustering algorithm should never return a cluster of 1 (no matter what clustering method is used), and this is a behaviour introduced in 3.2 - I think this is a bug that needs fixing

[ostephens]

Testing with a local project
In 3.1 http://127.0.0.1:3333/command/core/compute-clusters?project=2512015923657 returns
[]

In 3.2 http://127.0.0.1:3333/command/core/compute-clusters?project=2512015923657 returns

[[{
    "v": "Research studies in music education",
    "c": 1
}], [{
    "v": "Social marketing quarterly",
    "c": 1
}], [{
    "v": "Journal of adult and continuing education",
    "c": 1
}], [{
    "v": "Big data & society",
    "c": 1
}], [{
    "v": "Giornale di tecniche nefrologiche e dialitiche",
    "c": 1
}], [{
    "v": "Australasian psychiatry",
    "c": 1
}]

etc

[thadguidry]

@ostephens Doh ! Thanks, you guys are absolutely right... a Cluster means >1 :-) For some reason my brain was missing the entire reason for being with Clustering this morning!

Let's see if we can quickly push out a patch and 3.2.1 release for this.
How did our test cases miss this? Let's fix that also. (returned cluster must have >1 row or test fails)

[emiraglia]

Thank you both. Also, not to be annoying since it's not my product and I might just be using terms incorrectly, but when you say:
"returned cluster must have >1 row or test fails"
I just want to be sure that means that the cluster will have multiple choices, not just affected rows. For example, true duplicates will occur in multiple rows, but will have only one "choice" and so shouldn't be considered a cluster, right?

[ostephens]

Agreed - the cluster must contain multiple choices to be validly offered to the user as a cluster

[thadguidry]

@emiraglia it would be even better if.... you supplied us with a small test set of tabular data and the results on the Clustering dialog that you are expecting... that way we can also write a small test for your expectation as well.

use this format in the comment reply to provide the sample data, and Github will expose it as a real table of TSV values that we can copy/paste:

First Header | Second Header
------------ | -------------
Content from cell 1 | Content from cell 2
Content in the first column | Content in the second column

First Header	Second Header
Content from cell 1	Content from cell 2
Content in the first column	Content in the second column