INTPYTHON-527 Add Queryable Encryption support

[aclark4life]

Previous attempts and additional context here:

• INTPYTHON-527 Add Queryable Encryption config #318
  • Branched from transaction support and includes only helpers
• INTPYTHON-527 Add queryable encryption support #319
  • In which this bug in libmongocrypt was discovered
  • A fix for that bug in PyMongo is being discussed here.
• INTPYTHON-527 Add Queryable Encryption support #323
  • Before I understood that the shared lib for mongocrypt was included in the pymongocrypt wheel.

[timgraham]

The encryption tests are passing locally for me on Enterprise and on the Atlas VM.

On GitHub actions, this first issue was solved by adding "directConnection": True in DATABASES:

  File "/home/runner/work/django-mongodb-backend/django-mongodb-backend/django_repo/django/db/backends/base/base.py", line 197, in check_database_version_supported
    and self.get_database_version() < self.features.minimum_database_version
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-mongodb-backend/django-mongodb-backend/django_mongodb_backend/base.py", line 235, in get_database_version
    return tuple(self.connection.admin.command("buildInfo")["versionArray"])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/_csot.py", line 125, in csot_wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/database.py", line 926, in command
    with self._client._conn_for_reads(read_preference, session, operation=command_name) as (
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 1864, in _conn_for_reads
    server = self._select_server(read_preference, session, operation)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 1812, in _select_server
    server = topology.select_server(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 409, in select_server
    server = self._select_server(
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 387, in _select_server
    servers = self.select_servers(
              ^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 294, in select_servers
    server_descriptions = self._select_servers_loop(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 344, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: Could not reach any servers in [('3ff0eef351ff', 27017)]. Replica set is configured with internal hostnames or IPs?, Timeout: 30s, Topology Description: <TopologyDescription id: 688e9488ca8c88d98365da45, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('3ff0eef351ff', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('3ff0eef351ff:27017: [Errno -3] Temporary failure in name resolution (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms)')>]>

But this issue remains:

Creating test database for alias 'encrypted' ('test_djangotests-encrypted')...
/home/runner/.local/lib/python3.12/site-packages/pymongo/daemon.py:147: RuntimeWarning: Failed to start mongocryptd: is it on your $PATH?
Original exception: [Errno 2] No such file or directory: 'mongocryptd'
  _silence_resource_warning(_spawn(sys.argv[1:]))
/home/runner/.local/lib/python3.12/site-packages/pymongo/daemon.py:147: RuntimeWarning: Failed to start mongocryptd: is it on your $PATH?
Original exception: [Errno 2] No such file or directory: 'mongocryptd'
  _silence_resource_warning(_spawn(sys.argv[1:]))
  Applying sites.0002_alter_domain_unique... OK
Operations to perform:
  Synchronize unmigrated apps: auth, contenttypes, encryption_, messages, sessions, staticfiles
  Apply all migrations: admin, sites
Synchronizing apps without migrations:
  Creating tables...
    Creating table encryption__appointment
Traceback (most recent call last):
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/encryption.py", line 286, in mark_command
    res = self.mongocryptd_client[database].command(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/_csot.py", line 125, in csot_wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/database.py", line 926, in command
    with self._client._conn_for_reads(read_preference, session, operation=command_name) as (
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 1864, in _conn_for_reads
    server = self._select_server(read_preference, session, operation)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 1812, in _select_server
    server = topology.select_server(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 409, in select_server
    server = self._select_server(
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 387, in _select_server
    servers = self.select_servers(
              ^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 294, in select_servers
    server_descriptions = self._select_servers_loop(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 344, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:27020: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 10.0s, Topology Description: <TopologyDescription id: 688e9f76697ba6965a378048, topology_type: Unknown, servers: [<ServerDescription ('localhost', 27020) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27020: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms)')>]>

[aclark4life]

I'm still unclear on the user workflow. What I envisioned is:

* define model, `makemigrations`, `migrate`

* Put the output of `showencryptedfieldsmap` (which includes keyIds retrieved from server) in `AutoEncryptionOpts(encrypted_fields_map=...)`.

That is correct. We can remove --create if we never want to support this workflow:

• define model
• makemigrations
• Put the output of showencryptedfieldsmap --create (which creates new keys) in AutoEncryptionOpts(encrypted_fields_map=...).
• migrate

In other words, do we support the chicken and the egg or just the egg 😂

[aclark4life]

But this issue remains:

Creating test database for alias 'encrypted' ('test_djangotests-encrypted')...
/home/runner/.local/lib/python3.12/site-packages/pymongo/daemon.py:147: RuntimeWarning: Failed to start mongocryptd: is it on your $PATH?
Original exception: [Errno 2] No such file or directory: 'mongocryptd'
  _silence_resource_warning(_spawn(sys.argv[1:]))
/home/runner/.local/lib/python3.12/site-packages/pymongo/daemon.py:147: RuntimeWarning: Failed to start mongocryptd: is it on your $PATH?
Original exception: [Errno 2] No such file or directory: 'mongocryptd'

Any progress on this? Do we need to ask @blink1073 for help?

[addaleax]

@aclark4life Soooo ... I don't know what you've done around that area so far, but it's great that you're bringing it up, I didn't realize we'd have to still worry about this part because I don't think the PR currently concerns itself at all with how we integrate with crypt_shared/mongocryptd?

So I'll explain the whole story here, knowing that I'm risking telling you things you're alread well aware of 🙂

• For automatic CSFLE/QE, we don't just need the libmongocrypt integration provided by the MongoDB driver, we also need a query analysis engine. This is created from part of the server source code, and comes in two forms:
  1. mongocryptd, a daemon process that can be spawned and run by the driver and talked through via the MongoDB wire protocol. We consider this a legacy solution, but it is fully supported and not officially deprecated for now.
  2. The mongo_crypt_v1 shared library, a dll/shared object that can be loaded by libmongocrypt at runtime.

Operationally, mongocryptd is more difficult to deploy in production applications (you're essentially managing an external daemon process from the driver, which is complex on its own, and potentially sharing this instance between multiple application processes), so while it is currently still supported, we are looking to phase it out in the long run. For backwards compatibility reasons, it is still the default (and fallback) though.

So we generally recommend using the crypt_shared library, which can be achieved by passing crypt_shared_lib_path=... to the driver, and ideally also setting crypt_shared_lib_required=True so that libmongocrypt doesn't fall back to mongocryptd and instead fails if the crypt_shared library could not be loaded.

Given that this is a new feature here, I'd strongly consider setting crypt_shared_lib_required=True and only using the crypt_shared library by default, and never using mongocryptd (or only if the user has explicitly requested it). I've asked our PMs about this in https://mongodb.slack.com/archives/C0406ECL478/p1754488650159029, it feels like the right call to me but I'd like to double-check.

As far as the tests here are concerned themselves, I imagine they're passing locally for you because you do have mongocryptd in your path, so the driver is able to spawn it. Regardless on how you decide regarding mongocryptd, you'll need to install at least one of it or the crypt_shared library in CI, and in the latter case, point the driver to the path to it (happy to provide help with the details of this should you have any questions).

[aclark4life]

Given that this is a new feature here, I'd strongly consider setting crypt_shared_lib_required=True and only using the crypt_shared library by default, and never using mongocryptd (or only if the user has explicitly requested it). I've asked our PMs about this in https://mongodb.slack.com/archives/C0406ECL478/p1754488650159029, it feels like the right call to me but I'd like to double-check.

Can we document our way around this by recommending crypt_shared_lib_path in production but allowing mongocryptd in development? I think that is something we can get Django folks to accept but I don't think enforcing the use of crypt shared in development would go over well.

Also what about bundling that library in the pymongocrypt wheel? It's the convenience of pip install django-mongodb-backend[encryption] that we are after and we lose that with the enterprise download step.

[addaleax]

Can we document our way around this by recommending crypt_shared_lib_path in production but allowing mongocryptd in development? I think that is something we can get Django folks to accept but I don't think enforcing the use of crypt shared in development would go over well.

Maybe, but I think we'd want to have a conversation around what the typical expectations here are. You'll generally want to have development and production environments behave similarly, and you'd still need to have a plan for what to do when mongocryptd does get deprecated eventually (long-term, I think it's fair to expect this to happen). You'll also still be in a position where you need to download and install mongocryptd, the only case in which this requirement goes away is the one where you happen to have the enterprise MongoDB server binaries already ready in your $PATH.

Also what about bundling that library in the pymongocrypt wheel? It's the convenience of pip install django-mongodb-backend[encryption] that we are after and we lose that with the enterprise download step.

This question comes up on a regular basis 🙂 Here's a Slack thread from April, which was one of the last times we spoke about this.

tl;dr: Yes, the setup process for CSFLE/QE is involved, and we'd like to make it easier. Currently, there is a requirement for the user to explicitly acknowledge that they have read and accepted the enterprise license agreement and that they are an Atlas or EA customer. Bundling this library with regular packages that can be installed via a regular package manager command like pip install is therefore something we can't do right now.)

[Jibola]

Overall looks good! I've provided comments and feedback around some documentation and clarifications, but will approve once those are addressed.

[Jibola]

Same as earlier comment.

[aclark4life]

Actually this is annoying, can it go back into fields ?

[timgraham]

What does "annoying" mean? As I said before, it looks out of place to include utility functions in the fields module. I don't think there's a precedent in django.db.fields or django.forms.fields.

[aclark4life]

Oh I just don't like the one-off module with a single function in it. I tried to put it in utils too but had the same circular imports issue.