DRIVERS-2888 Support QE with Client.bulkWrite

source/client-side-encryption/tests/README.md

@@ -565,10 +565,13 @@ First, perform the setup.
 2. Using `client`, drop and create the collection `db.coll` configured with the included JSON schema
     [limits/limits-schema.json](../limits/limits-schema.json).
 
-3. Using `client`, drop the collection `keyvault.datakeys`. Insert the document
+3. Using `client`, drop and create the collection `db.coll2` configured with the included encryptedFields
+    [limits/limits-encryptedFields.json](../limits/limits-encryptedFields.json).
+
+4. Using `client`, drop the collection `keyvault.datakeys`. Insert the document
     [limits/limits-key.json](../limits/limits-key.json)
 
-4. Create a MongoClient configured with auto encryption (referred to as `client_encrypted`)
+5. Create a MongoClient configured with auto encryption (referred to as `client_encrypted`)
 
     Configure with the `local` KMS provider as follows:
 
@@ -580,27 +583,27 @@ First, perform the setup.
 
 Using `client_encrypted` perform the following operations:
 
-1. Insert `{ "_id": "over_2mib_under_16mib", "unencrypted": <the string "a" repeated 2097152 times> }`.
+1. Insert `{ "_id": "over_2mib_under_16mib", "unencrypted": <the string "a" repeated 2097152 times> }` into `coll`.
 
     Expect this to succeed since this is still under the `maxBsonObjectSize` limit.
 
 2. Insert the document [limits/limits-doc.json](../limits/limits-doc.json) concatenated with
-    `{ "_id": "encryption_exceeds_2mib", "unencrypted": < the string "a" repeated (2097152 - 2000) times > }` Note:
+    `{ "_id": "encryption_exceeds_2mib", "unencrypted": < the string "a" repeated (2097152 - 2000) times > }` into `coll`. Note:
     limits-doc.json is a 1005 byte BSON document that encrypts to a ~10,000 byte document.
 
     Expect this to succeed since after encryption this still is below the normal maximum BSON document size. Note, before
     auto encryption this document is under the 2 MiB limit. After encryption it exceeds the 2 MiB limit, but does NOT
     exceed the 16 MiB limit.
 
-3. Bulk insert the following:
+3. Use MongoCollection.bulkWrite to insert the following into `coll`:
 
     - `{ "_id": "over_2mib_1", "unencrypted": <the string "a" repeated (2097152) times> }`
     - `{ "_id": "over_2mib_2", "unencrypted": <the string "a" repeated (2097152) times> }`
 
     Expect the bulk write to succeed and split after first doc (i.e. two inserts occur). This may be verified using
     [command monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.md).
 
-4. Bulk insert the following:
+4. Use MongoCollection.bulkWrite insert the following into `coll`:
 
     - The document [limits/limits-doc.json](../limits/limits-doc.json) concatenated with
         `{ "_id": "encryption_exceeds_2mib_1", "unencrypted": < the string "a" repeated (2097152 - 2000) times > }`
@@ -610,15 +613,33 @@ Using `client_encrypted` perform the following operations:
     Expect the bulk write to succeed and split after first doc (i.e. two inserts occur). This may be verified using
     [command logging and monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.md).
 
-5. Insert `{ "_id": "under_16mib", "unencrypted": <the string "a" repeated 16777216 - 2000 times>`.
+5. Insert `{ "_id": "under_16mib", "unencrypted": <the string "a" repeated 16777216 - 2000 times>` into `coll`.
 
     Expect this to succeed since this is still (just) under the `maxBsonObjectSize` limit.
 
 6. Insert the document [limits/limits-doc.json](../limits/limits-doc.json) concatenated with
-    `{ "_id": "encryption_exceeds_16mib", "unencrypted": < the string "a" repeated (16777216 - 2000) times > }`
+    `{ "_id": "encryption_exceeds_16mib", "unencrypted": < the string "a" repeated (16777216 - 2000) times > }` into `coll`.
 
     Expect this to fail since encryption results in a document exceeding the `maxBsonObjectSize` limit.
 
+7. Use MongoClient.bulkWrite to insert the following into `coll2`:
+
+    - `{ "_id": "over_2mib_3", "unencrypted": <the string "a" repeated (2097152 - 1500) times> }`
+    - `{ "_id": "over_2mib_4", "unencrypted": <the string "a" repeated (2097152 - 1500) times> }`
+
+    Expect the bulk write to succeed and split after first doc (i.e. two inserts occur). This may be verified using
+    [command logging and monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.md).
+
+8. Use MongoClient.bulkWrite to insert the following into `coll2`:
+
+    - The document [limits/limits-qe-doc.json](../limits/limits-qe-doc.json) concatenated with
+        `{ "_id": "encryption_exceeds_2mib_3", "foo": < the string "a" repeated (2097152 - 2000 - 1500) times > }`
+    - The document [limits/limits-qe-doc.json](../limits/limits-qe-doc.json) concatenated with
+        `{ "_id": "encryption_exceeds_2mib_4", "foo": < the string "a" repeated (2097152 - 2000 - 1500) times > }`
+
+    Expect the bulk write to succeed and split after first doc (i.e. two inserts occur). This may be verified using
+    [command logging and monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.md).
+
 Optionally, if it is possible to mock the maxWriteBatchSize (i.e. the maximum number of documents in a batch) test that
 setting maxWriteBatchSize=1 and inserting the two documents `{ "_id": "a" }, { "_id": "b" }` with `client_encrypted`
 splits the operation into two inserts.

source/crud/bulk-write.md

@@ -634,13 +634,6 @@ write concern containing the following message:
 
 > Cannot request unacknowledged write concern and ordered writes
 
-## Auto-Encryption
-
-If `MongoClient.bulkWrite` is called on a `MongoClient` configured with `AutoEncryptionOpts`, drivers MUST return an
-error with the message: "bulkWrite does not currently support automatic encryption".
-
-This is expected to be removed once [DRIVERS-2888](https://jira.mongodb.org/browse/DRIVERS-2888) is implemented.
-
 ## Command Batching
 
 Drivers MUST accept an arbitrary number of operations as input to the `MongoClient.bulkWrite` method. Because the server
@@ -917,6 +910,8 @@ error in this specific situation does not seem helpful enough to require size ch
 
 ## **Changelog**
 
+- 2025-04-23: Removed the requirement to error when QE is enabled.
+
 - 2024-11-05: Updated the requirements regarding the size validation.
 
 - 2024-10-07: Error if `w:0` is used with `ordered=true` or `verboseResults=true`.

source/crud/tests/unified/client-bulkWrite-qe.yml

@@ -0,0 +1,128 @@
+description: client bulkWrite with queryable encryption
+
+schemaVersion: "1.23"
+
+runOnRequirements:
+  - minServerVersion: "8.0"
+    serverless: forbid # Serverless does not support bulkWrite: CLOUDP-256344.
+    csfle: true
+
+createEntities:
+  - client:
+      id: &client0 client0
+      observeEvents:
+        - commandStartedEvent
+        - commandSucceededEvent
+      autoEncryptOpts:
+        keyVaultNamespace: keyvault.datakeys
+        kmsProviders:
+          local:
+            key: Mng0NCt4ZHVUYUJCa1kxNkVyNUR1QURhZ2h2UzR2d2RrZzh0cFBwM3R6NmdWMDFBMUN3YkQ5aXRRMkhGRGdQV09wOGVNYUMxT2k3NjZKelhaQmRCZGJkTXVyZG9uSjFk
+  - database:
+      id: &database0 database0
+      client: *client0
+      databaseName: &database0Name crud-tests
+  - collection:
+      id: &collection0 collection0
+      database: *database0
+      collectionName: &collection0Name coll0
+  - client:
+      id: &client1 client1
+      observeEvents:
+        - commandStartedEvent
+  - database:
+      id: &database1 database1
+      client: *client0
+      databaseName: &database1Name keyvault
+  - collection:
+      id: &collection1 collection1
+      database: *database0
+      collectionName: &collection1Name datakeys
+  - database:
+      id: &database2 database2
+      client: *client1
+      databaseName: &database0Name crud-tests
+  - collection:
+      id: &collection2 collection2
+      database: *database2
+      collectionName: &collection0Name coll0
+
+
+initialData:
+  - databaseName: *database1Name
+    collectionName: *collection1Name
+    documents:
+      - _id: &local_key_id { $binary: { base64: EjRWeBI0mHYSNBI0VniQEg==, subType: "04" } }
+        keyAltNames: ["local_key"]
+        keyMaterial: { $binary: { base64: sHe0kz57YW7v8g9VP9sf/+K1ex4JqKc5rf/URX3n3p8XdZ6+15uXPaSayC6adWbNxkFskuMCOifDoTT+rkqMtFkDclOy884RuGGtUysq3X7zkAWYTKi8QAfKkajvVbZl2y23UqgVasdQu3OVBQCrH/xY00nNAs/52e958nVjBuzQkSb1T8pKJAyjZsHJ60+FtnfafDZSTAIBJYn7UWBCwQ==, subType: "00" } }
+        creationDate: { $date: { $numberLong: "1641024000000" } }
+        updateDate: { $date: { $numberLong: "1641024000000" } }
+        status: 1
+        masterKey: &local_masterkey
+          provider: local
+  - databaseName: *database0Name
+    collectionName: *collection0Name
+    documents: []
+    createOptions:
+      encryptedFields: &encrypted_fields {'fields': [{'keyId': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedInt', 'bsonType': 'int', 'queries': {'queryType': 'equality', 'contention': {'$numberLong': '0'}}}]}
+
+_yamlAnchors:
+  namespace: &namespace "crud-tests.coll0"
+
+tests:
+  - description: client bulkWrite QE replaceOne
+    operations:
+      - object: *collection0
+        name: insertMany
+        arguments:
+          documents:
+            - { _id: 1, encryptedInt: 11 }
+            - { _id: 2, encryptedInt: 22 }
+            - { _id: 3, encryptedInt: 33 }
+      - object: *client0
+        name: clientBulkWrite
+        arguments:
+          models:
+            - replaceOne:
+                namespace: *namespace
+                filter: { encryptedInt: { $eq: 11 } }
+                replacement: { encryptedInt: 44 }
+        expectResult:
+          insertedCount: 0
+          upsertedCount: 0
+          matchedCount: 1
+          modifiedCount: 1
+          deletedCount: 0
+      - object: *collection0
+        name: find
+        arguments:
+          filter: { encryptedInt: 44 }
+        expectResult:
+          - _id: 1
+            encryptedInt: 44
+      - object: *collection2
+        name: find
+        arguments:
+          filter: {}
+        expectResult:
+          - { _id: 1, encryptedInt: { $$type: binData }, __safeContent__: { $$type: array} }
+          - { _id: 2, encryptedInt: { $$type: binData }, __safeContent__: { $$type: array} }
+          - { _id: 3, encryptedInt: { $$type: binData }, __safeContent__: { $$type: array} }
+  - description: client bulkWrite QE with multiple replace fails
+    operations:
+      - object: *client0
+        name: clientBulkWrite
+        arguments:
+          models:
+            - replaceOne:
+                namespace: *namespace
+                filter: { encryptedInt: { $eq: 11 } }
+                replacement: { encryptedInt: 44 }
+            - replaceOne:
+                namespace: *namespace
+                filter: { encryptedInt: { $eq: 22 } }
+                replacement: { encryptedInt: 44 }
+        expectError:
+          # Expect error from mongocryptd or crypt_shared
+          isError: true
+          errorContains: "Only insert is supported in BulkWrite with multiple operations and Queryable Encryption"