Restore UNIQUE_CHECKS mysql variable to its original value when done …

[hostep]

…with changing the database structure. This prevents potentially inserting duplicated values while manipulating data in the setup:upgrade command.

Description (*)

See #32283
When approved, please please please consider backporting this to Magento 2.3.7, this is too important to be ignored for 2.3 in my opinion.

According to Magento core devs I spoke to on Slack in the past few days, there is an underlying problem where connections should be closed and opened again between the schema manipulation and the data manipulation in order to reset these mysql variables. But that's not happening.
So this is only a workaround, but since the same workaround was implemented for the FOREIGN_KEY_CHECKS and others, it makes sense to also do this for UNIQUE_CHECKS until the root cause is found, since it's an easy fix which should not cause issues.

I have no intention to write automated tests for this change, I already spend more then 10 hours on dealing with the consequences of this issue and then an hour on finding an appropriate solution, I can't invest more time in adding automated tests, sorry!

Related Pull Requests

Fixed Issues (if relevant)

1. Fixes Mysql UNIQUE_CHECKS is disabled for data manipulation while it should only be disabled for schema manipulations #32283: Mysql UNIQUE_CHECKS is disabled for data manipulation while it should only be disabled for schema manipulations
2. Fixes Attribute value was duplicated after upgrade magento from 2.1 to 2.3 #28326: Attribute value was duplicated after upgrade magento from 2.1 to 2.3

Manual testing scenarios (*)

1. In some random RecurringData setup script, add some debugging code to verify the value of the UNIQUE_CHECKS variable, the RecurringData script makes it easy to test. For example, in the Magento\SalesSequence\Setup\RecurringData::install method, add:

        $connection = $setup->getConnection();
        var_dump('*** In RecurringData', $connection->query('SHOW VARIABLES LIKE "UNIQUE_CHECKS"')->fetch());

2. Run bin/magento setup:upgrade

Before this PR, it outputs:

string(20) "*** In RecurringData"
array(2) {
  ["Variable_name"]=>
  string(13) "unique_checks"
  ["Value"]=>
  string(3) "OFF"
}

After this PR, it outputs:

string(20) "*** In RecurringData"
array(2) {
  ["Variable_name"]=>
  string(13) "unique_checks"
  ["Value"]=>
  string(2) "ON"
}

Questions or comments

Contribution checklist (*)

• Pull request has a meaningful description of its purpose
• All commits are accompanied by meaningful commit messages
• All new or changed code is covered with unit/integration tests (if applicable)
• All automated tests passed successfully (all builds are green)

[m2-assistant]

Hi @hostep. Thank you for your contribution
Here are some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

• @magento give me test instance - deploy test instance based on PR changes
• @magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

• @magento run all tests - run or re-run all required tests against the PR changes
• @magento run <test-build(s)> - run or re-run specific test build(s)
  For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

1. Database Compare
2. Functional Tests CE
3. Functional Tests EE,
4. Functional Tests B2B
5. Integration Tests
6. Magento Health Index
7. Sample Data Tests CE
8. Sample Data Tests EE
9. Sample Data Tests B2B
10. Static Tests
11. Unit Tests
12. WebAPI Tests
13. Semantic Version Checker

You can find more information about the builds here

ℹ️ Please run only needed test builds instead of all when developing. Please run all test builds before sending your PR for review.

For more details, please, review the Magento Contributor Guide documentation.

⚠️ According to the Magento Contribution requirements, all Pull Requests must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of Pull Requests happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[hostep]

@magento run all tests

[hostep]

@magento run all tests

[magento-engcom-team]

Hi @ihor-sviziev, thank you for the review.
ENGCOM-8825 has been created to process this Pull Request
✳️ @ihor-sviziev, could you please add one of the following labels to the Pull Request?

Label	Description
Auto-Tests: Covered	All changes in Pull Request is covered by auto-tests
Auto-Tests: Not Covered	Changes in Pull Request requires coverage by auto-tests
Auto-Tests: Not Required	Changes in Pull Request does not require coverage by auto-tests

[ihor-sviziev]

@magento run Static Tests

[ihor-sviziev]

@sivaschenko @sidolov @gabrieldagama, it seems like p1 priority should be set for this issue. What do you think?

Note: it would be great to coved this PR with some integration/setup-integration test, but not sure if it will be easy

[sivaschenko]

Milestone set to 2.5 as this change is backwards incompatible.

Is it possible to cover the issue with integration tests (i.e. adding a test module trying to install duplicated data)?

[hostep]

@sivaschenko: I strongly disagree with the BiC statement.

You don't want people to have corrupt databases right? Rows in a database table should be unique according to the unique key which is set on the table. Currently Magento doesn't protect your database from this by disabling unique_checks during bin/magento setup:upgrade.
This is very bad.

I would like to ask you to reconsider, thanks!

[sivaschenko]

@hostep @ihor-sviziev thanks for bringing up the concern!

Is there any functional impact of duplicated DB entries except MySQL dump/import?
Can the issue be reproduced on vanilla Magento, or does it require customization?

[hostep]

Is there any functional impact of duplicated DB entries except MySQL dump/import?

Not as far as I could see.
But this has implications on how people deal with databases. Often shopowners decide to move to a different hosting company and need to migrate the database. Or it is decided to split a shared web and database server running in the same server, to 2 separate servers, requiring you to migrate the database to that other server.
Or just plain simply for developers who want to debug something on their local instances with a copy of a database from the server, requiring again you to be able to migrate the database. There are probably more examples, but performing mysqldump/import is part of a lot of different workflows and having a corrupt dump is not fun to deal with.

Can the issue be reproduced on vanilla Magento, or does it require customization?

This was explained in #32283, but short summary: I've seen 2 separate instances of this same bug happening, it was both times triggered by this script part of core Magento. This patch only executes when you upgrade from Magento < 2.2.0 to Magento 2.2.0 or higher. In our case we ran into this when upgrading a shop from 2.1.11 to 2.3.6 and a different shop from 2.1.16 to 2.3.6-p1. From what I could determine, the ON duplicate KEY UPDATE statement was being ignored because UNIQUE_CHECKS was disabled when that patch was being executed.
I don't know if there are more data patches in Magento using ON duplicate KEY UPDATE, but core Magento patches are not the only concern.
In theory you can trigger the same problem using a custom module which tries to do something similar where it is expected that duplicated entries are not being inserted due to the unique key on the table. Since Magento is a framework, custom modules should be considered as well in scope of this issue and not only Magento core code in my opinion.

Maybe good to know, but I can tell you with 100% certainty that this PR here fixes the issue we had on 1 of those two shops, because I had a way to reproduce the problem on the staging environment of that shop.
For the second shop, it was too late to be tested, we only noticed it once the upgrade had been rolled out to production and then had to clean up the database manually after it already became corrupt.

[ihor-sviziev]

This one might be also related, but not sure #29804

[sivaschenko]

Thank you for the detailed explanation @hostep . I will change the milestone to 2.4 and launch an internal approval process for BIC. In case the BIC will be approved - we'll be able to deliver it to 2.4.

[hostep]

Thanks @sivaschenko: please also consider backporting to 2.3 if possible 🙂

[sivaschenko]

Internal approval request: https://jira.corp.magento.com/browse/MC-41144

[hostep]

@sivaschenko: is there any sort of progress on this?

I accidentally bumped into another report for the same problem, so I'm definitely not the only one who ran into this problem

(And did my comment in that other ticket for some reason moved this from "to approve" to "ready for testing"? That's wierd)

[sivaschenko]

@hostep approval is in progress

[ihor-sviziev]

I moved back to "to approve" status.
@sivaschenko, are there any updates on approval?

[hostep]

Hi folks, asking again after 2 months. PR has prio 1, still no movement, any idea what's blocking this and can I help out somehow?

[ihor-sviziev]

@gabrieldagama @sivaschenko @sidolov could you help with it?

[mebbopakko]

Any update regarding this PR?
I encountered the problem in an upgrade from 2.1.0 to 2.4.2-p1 and fortunately solved it with @hostep solution.

We have now more and more clients coming with upgrade requests and I think this should be fixed asap.

[ihor-sviziev]

@kandy maybe you can move forward approval for such important fix?

[kandy]

Unfortunately, I'm not part of the Community Engineering team. I think the best person to ping is @sidolov.
For me all looks pretty straightforward

[magento-engcom-team]

Hi @Den4ik, thank you for the review.
ENGCOM-8825 has been created to process this Pull Request

[m2-assistant]

Hi @hostep, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

[ihor-sviziev]

Finally, it got merged! 🎉