Set charset utf8mb4 for all dbs

[angonz]

#890 enclosed charset: "utf8mb3" in a {%- if RUN_MYSQL %} block. This was to fix an issue with the upgrade from Olive to Palm.
Later on, the charset was changed to utf8mb4 by #1084. However, the setting was still enclosed in the condition to apply only when the DB is run by Tutor.
As a result, external databases are not open with utf8mb4 charset.

[DawoudSheraz]

@Danyal-Faheem do we even need this? Mysql 8.4 default charset is utf8mb4 (https://dev.mysql.com/doc/refman/8.4/en/charset.html).

[angonz]

I am not sure, but our database runs in a 8.0.mysql_aurora.3.05.2 and we had problems when this was not set, even though the charset and collation was properly set in all logical dbs and tables.

We noted this when migrating the forum db from MongoDB to MySQL. Many entries had emojis in the title and/or body, and the migration script raised a database error when the DB was open without this option. After adding this option, it worked perfectly well.

[DawoudSheraz]

The problem with enforcing this for external databases is that we do not have control over which MySQL version is being used. This option is the default for Mysql v8+. If someone is running a prior version, that can cause issues.
Since openedx-auth patch is provided, that patch can be used to override these settings.

@Danyal-Faheem Please correct me if I misquoted anything.

DATABASES:
  default:
    ENGINE: "django.db.backends.mysql"
    HOST: "{{ MYSQL_HOST }}"
    PORT: {{ MYSQL_PORT }}
    NAME: "{{ OPENEDX_MYSQL_DATABASE }}"
    USER: "{{ OPENEDX_MYSQL_USERNAME }}"
    PASSWORD: "{{ OPENEDX_MYSQL_PASSWORD }}"
    ATOMIC_REQUESTS: true
    OPTIONS:
      init_command: "SET sql_mode='STRICT_TRANS_TABLES'"
      charset: "utf8mb4"

[DawoudSheraz]

@angonz Hi, does this need further action? Thanks

[MoisesGSalas]

Hi @DawoudSheraz, I want to support Andrés point a little bit. In one of our recent upgrades to sumac we faced the exact same issue.

I'm not quite sure why, but the MySQL client used by Django is defaulting to utf8mb3, so even though the tables in the database are using the correct collation and charset, the connection is wrong.

I would argue that these setting values shouldn't be about external or internal databases, but rather about the database version. If I'm using the same DB as tutor then all the defaults should work the same. If I'm not, then it should be my responsibility to adjust accordingly.

[DawoudSheraz]

Django defaulted the charset to utf8mb4 in Django v5.2 https://docs.djangoproject.com/en/5.2/releases/5.2/#database-backends.

Since this can be a potentially breaking change, this should be merged into main instead of release. Merging in the current release can impact any existing users who are using external DBs and are not using the tutor version.