Implement filter, map, reduce

[katcharov]

JAVA-4781

(Javadocs are tentative, as per JAVA-4799)

[jyemin]

The type of the reduced value can be different than the type of the array elements, e.g.

{
    $reduce: {
       input: ["1", "2", "3"],
       initialValue: 0,
       in: { $sum : [{$toInt: "$$this"}, "$$value.sum"] }
 }

This can likely be dealt with by introducing a type parameter to the function definition to represent the reduced type.

[katcharov]

I suspect that we should force this to be expressed as:

ofStringArray("1", "2", "3", "4")
        .map(v -> v.toInt())
        .reduce(of(0), (a, b) -> a.add(b));

Or perhaps as:

ofStringArray("1", "2", "3", "4").addOn(v -> v.toInt());

This last variant is appealing because it hides the default value and presents reduction in a simplified way. If it can be made to work, I would argue for not exposing reduce at all. However, I'm not sure how to get the following to work:

ArrayExpression<IntegerExpression> numbers = ofIntegerArray(1, 2, 3, 4);
numbers.add(); // what interface is "add" on, if we want type safety?

[jyemin]

Can you think of a way to express multiple reductions, so we can support something like:

{
    $reduce: {
       input: ["1", "2", "3"],
       initialValue: {sum: 0, product: 0, concat: ""},
       in: { 
          sum :    { $sum :      [{$toInt: "$$this"}, "$$value.sum"] },
          product: { $multiply : [{$toInt: "$$this"}, "$$value.product"] }
          concat:  { $concat   : ["$$this",           "$$value.concat"] }
        }   
     }
 }

?

[katcharov]

This can be accomplished with the existing method:

assertExpression(BsonDocument.parse("{'sum': 10, 'product': 24, 'concat': '1234'}"),

ofStringArray("1", "2", "3", "4")
        // map:
        .<DocumentExpression>map(v -> ofEmptyDocument()
                .setField("sum", v.toInt())
                .setField("product", v.toInt())
                .setField("concat", v))
        // identity:
        .reduce(ofEmptyDocument()
                        .setField("sum", of(0))
                        .setField("product", of(1))
                        .setField("concat", of("")),
        // reduction:
        (a, b) -> ofEmptyDocument()
                .setField("sum", a.getFieldInteger("sum").add(b.getFieldInteger("sum")))
                .setField("product", a.getFieldInteger("product").multiply(b.getFieldInteger("product")))
                .setField("concat", a.getFieldString("concat").concat(b.getFieldString("concat")))));

Above test passes. However, I am unsure if the server knows how to move a map into a reduction, if such a move is needed to help performance.

Compact with a schema object:

ofStringArray("1", "2", "3", "4")
        .map(v -> new Result(v.toInt(), v.toInt(), v))
        .reduce(new Result(0, 1, ""), (a, b) -> new Result(
                a.getSum().add(b.getSum()),
                a.getProduct().multiply(b.getProduct()),
                a.getConcat().concat(b.getConcat())) ))

Could try to push it to something approximating:

ofStringArray("1", "2", "3", "4").reduce(v -> result()
        .set("sum", v -> v.toInt().add())
        .set("product", v -> v.toInt().multiply())
        .set("concat", v -> v.concat()))

I think I'd need to have a closer look at $group. I wonder if this last variant (which is basically syntax sugar) obscures what is actually going on (a map followed by a reduce). On the other hand, I like that this last variant hides the identity value.

[jyemin]

What MQL does that first bit of code ("with the existing method") generate?

[katcharov]

{
  "$reduce":{
    "input":{
      "$map":{
        "input":[
          "1",
          "2",
          "3",
          "4"
        ],
        "in":{
          "$setField":{
            "field":"concat",
            "input":{
              "$setField":{
                "field":"product",
                "input":{
                  "$setField":{
                    "field":"sum",
                    "input":{
                      "$literal":{
                        
                      }
                    },
                    "value":{
                      "$toInt":"$$this"
                    }
                  }
                },
                "value":{
                  "$toInt":"$$this"
                }
              }
            },
            "value":"$$this"
          }
        }
      }
    },
    "initialValue":{
      "$setField":{
        "field":"concat",
        "input":{
          "$setField":{
            "field":"product",
            "input":{
              "$setField":{
                "field":"sum",
                "input":{
                  "$literal":{
                    
                  }
                },
                "value":0
              }
            },
            "value":1
          }
        },
        "value":""
      }
    },
    "in":{
      "$setField":{
        "field":"concat",
        "input":{
          "$setField":{
            "field":"product",
            "input":{
              "$setField":{
                "field":"sum",
                "input":{
                  "$literal":{
                    
                  }
                },
                "value":{
                  "$add":[
                    {
                      "$getField":{
                        "input":"$$value",
                        "field":"sum"
                      }
                    },
                    {
                      "$getField":{
                        "input":"$$this",
                        "field":"sum"
                      }
                    }
                  ]
                }
              }
            },
            "value":{
              "$multiply":[
                {
                  "$getField":{
                    "input":"$$value",
                    "field":"product"
                  }
                },
                {
                  "$getField":{
                    "input":"$$this",
                    "field":"product"
                  }
                }
              ]
            }
          }
        },
        "value":{
          "$concat":[
            {
              "$getField":{
                "input":"$$value",
                "field":"concat"
              }
            },
            {
              "$getField":{
                "input":"$$this",
                "field":"concat"
              }
            }
          ]
        }
      }
    }
  }
}

[stIncMale]

It does not seem possible to specify the limit via this API. Is this this piece of MQL $filter functionality deliberately omitted?

[katcharov]

I think we should implement this parameter in something like the following way instead:

array.filter(condition).limit(10)

[stIncMale]

What's the plan for this functionality? Given that this PR implements $filter, I'd expect the functionality added in this PR.

[jyemin]

What would array.limit(10) do? I don't see a $limit expression for arrays outside the context of a filter

[katcharov]

@stIncMale MQL has added this parameter for the sake of convenience. This convenience helps programmers writing MQL (with query size), and possibly the server in interpreting MQL. However, it is detrimental for users of our API, because it unnecessarily introduces two ways of doing things, and the mechanism that it introduces is inelegant and bloats our API. So I do not think it should be implemented.

@jyemin I believe that method that would correspond to limit is $firstN. I do not like that the names are inconsistent, and would like to address that, but do not see a clean way to do so.