-
Notifications
You must be signed in to change notification settings - Fork 9.4k
Allows API pagination to return blank pages #12475
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Such change would be pretty confusing.
That confuses even more.
Changing class behavior with some switches/flags is never a good idea. There was already such discussion in #9255. From the issue you mentioned I see that API returns total amount of records:
Could you please add more details why you need to access invalid page at all? From the limitations of patch you described it seems like there is some bug with last page determination in your code but pagination works fine with underflow. Why not fix invalid pagination instead of hacking collection class? |
I agree that this should, ideally, be universal. In an ideal world, this behavior wouldn't be specifically hard-coded into the Collection class. Unfortunately, it is hard-coded, and there's already a system built around that assumption. If you would prefer that this change affect all collections, that is a simple enough change to make and I will do that. However, this change preserves backward-compatibility and only addresses the undesirable behavior from within the API.
This limit is hard-coded into the Zend_Db_Select class, and it deviates from how MariaDB and MySQL behave. At least this protects users and programmers from bad input, though — MariaDB and MySQL will return an error for any
You have members of this class which do exactly this. Of course, this wouldn't be necessary if the Collection class behaved like the underlying SQL database and returned an empty array instead of returning the wrong data.
It's not that "pagination works fine with underflow". A request for Page -1 should also return an empty array, IMO, but this behavior is hard-coded into the Zend_Db_Select class, and that's a very low-level class to modify for what should be a localized problem. Rather than detecting the error and returning an empty set, the design decision was to default to the first page. This is also predictable because a negative page-number is always, always, always invalid. A positive page-number is not always invalid. These also aren't limitations of the patch. They're the related behaviors that I did not touch. I think that Page -1 returning data from Page 1 is just as wrong as Page 1 returning data from Page 7. I'm just a reasonable person, and I understand why a system would enforce a lower-limit of "Page 1" in a paginated system.
Because this is the highest-level in the call-stack where such a change makes sense, and the alternative is to edit every API endpoint individually. The odd behavior originates from the collection class because the collection class deviates in behavior from the underlying database. An empty array is a valid response to a query that returns no results. A populated array is not a valid response to a query that returns no results. |
@stevethedev, thanks for detailed explanation,
Which is most likely Magento 1 legacy code and does not prove flags should ever be used ;) Ok, so we would display standard message "No products matching your selection".
Aha, okiz. Please share your thoughts in underlying issue regarding consistent fix VS proper error returned in API. |
I agree. I've pushed up a new version that addresses this issue in the API instead of in the Collection.
I would prefer to receive an empty array from the API. An empty array unambiguously communicates two things:
This update simplifies communication with the API. The specific use-case where the current behavior becomes a problem is if we use API Paging and JavaScript Promises as a way to throttle requests to a Magento 2 server. This significantly simplifies the way that content is read from the API: // Node.js v8.9.1
async function getItems(endpoint, batchSize = 100) {
const items = [];
let page = 0;
let batch = null;
do {
batch = await requestPage(endpoint, ++page, batchSize);
items.push(...batch);
} while (batch.length);
return items;
}
I think that my newest updates to the Pull Request probably address this issue on the API level. My primary concern is how Magento interacts with the world "outside" of Magento. My main complaint is that the Magento API returns content in a way that makes consumption needlessly complex. |
$searchBuilder->setPageSize(1); | ||
$searchBuilder->setCurrentPage(2); | ||
$pageSize = 1; | ||
$expectedPageCount = ceil(count($expectedResult) / $pageSize); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not a good idea to put logic into test. Why this integration test is even affected by API-only changes?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
$searchBuilder = $this->_objectManager->create(\Magento\Framework\Api\SearchCriteriaBuilder::class);
ah, ok, so this seems to be widely used in the system besides API.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. The API's code is used outside of the main API.
I've also found, while diagnosing the failed tests, that two assumptions exist in several places:
- Page
< 1
should retrieve Page 1. I could not find the source of this assumption. I set the API to assume that any page number less than 1 should be set to Page 1 to avoid breaking existing behavior. We had previously discussed underflow, and this abides by that discussion. - Page
max + 1
should return Pagemax
. In this instance, the test was passing because Page 2 of 1 was returning Page 1. In the AddressRepositoryTest, there are tests that should return only 1 page, but the test always sets "Page 2". A side-effect of the behavior this PR addresses is that the tests could pass if any number of addresses were returned (because Page 2 of 1 would return Page 1, and Page 2 of 3 would return Page 2). This test now assumes that it should retrieve the last page, which seemed to be the original intent of the test.
I haven't fixed all of the failures from the integration tests, yet, but all of the issues I've found recently are because of the two points I listed above.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
$pageSize = 1;
$expectedPageCount = ceil(count($expectedResult) / $pageSize);
is it different from
$expectedPageCount = count($expectedResult);
?
- There is no place between WebAPI and search builder which could be changed to not affect integration tests at all but only WebAPI?
- If so, maybe it makes more sense to affect Web UI pagination as well - by changing underlying collection logic and nothing more?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
{code} is it different from {code}
They are effectively the same. The first one is just for clarity, but could be easily updated to the second one. Would you prefer the 1-line version over the 2-line version?
- There is no place between the WebAPI and search builder which could be changed to not affect integration tests at all but only WebAPI?
These changes invalidate the assumption that (2 of 1) === (1 of 1)
. Any tests that make this assumption at the API level (such as this one) will need to be updated regardless of where the fix is applied.
- If so, maybe it makes more sense to affect Web UI pagination as well - by changing underlying collection logic and nothing more?
This is an option, but it would affect more of the system. This change would be at ./lib/internal/Magento/Framework/Data/Collection.php:239
:
public function getCurPage($displacement = 0)
{
if ($this->_curPage + $displacement < 1) {
return 1;
/* These lines would be removed
} elseif ($this->_curPage + $displacement > $this->getLastPageNumber()) {
return $this->getLastPageNumber();
*/
} else {
return $this->_curPage + $displacement;
}
}
However, tests that assume that Page 2 of 1 should return Page 1 would still need to be updated. That's the logic being changed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, tests like this would need to go:
public function testGetCurPage()
{
$this->_model->setCurPage(10);
$this->assertEquals(1, $this->_model->getCurPage());
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, this test data would need to change:
public function getPriceFormatDataProvider()
{
return [
['en_US', ['decimalSymbol' => '.', 'groupSymbol' => ',']],
['de_DE', ['decimalSymbol' => ',', 'groupSymbol' => '.']],
['de_CH', ['decimalSymbol' => '.', 'groupSymbol' => '\'']], //<--- Right Here
['uk_UA', ['decimalSymbol' => ',', 'groupSymbol' => ' ']]
];
}
That value should be ’
, not \'
. This test should not be passing right now.
@@ -328,7 +328,7 @@ public function testSearchAddresses($filters, $filterGroup, $filterOrders, $expe | |||
} | |||
|
|||
$searchBuilder->setPageSize(1); | |||
$searchBuilder->setCurrentPage(2); | |||
$searchBuilder->setCurrentPage(count($expectedResult)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change is necessary because some of the tests only expect 1 page back. This change ensures that tests that expect 1 page do not request the second page.
$this->_model->setCurPage(0); | ||
$this->assertEquals(1, $this->_model->getCurPage()); | ||
|
||
$this->_model->setCurPage(1); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change removes the assumption that Page 10 of 1 should return Page 1, but preserves the assumption that Pages < 1 return Page 1.
Has there been any further movement on this PR? |
@stevenvdp not yet, we need to understand whether a deeper fix (right on collection) is better. No changes needed from your side currently. |
really wait for merge this PR |
Closed due to inactivity. @stevethedev , Please feel free to port this pull request to 2.3-develop branch as 2.2-develop does not accept improvement anymore (see the porting section of the contributor guide for more details). Also please sign our Contributor License Agreement before we can accept your contributions. |
That's all you need to know about m2. The whole approach to business in one thing. |
@springimport this is not about m2, this is my personal fault actually. Raised this problem once in Slack without much attention, will raise again this week, luckily we now have an |
@orlangur Nice. |
the issue still exists |
Description
This update prevents the API pagination from always returning data after the last page results. Prior to this patch, the
searchCriteria[currentPage]
value would always return the last page that would have returned content, rather than the page that was requested (i.e. given a data set of 10 items, and a page-length of 10, pages [-∞ .. ∞] would all return 10 items).Collection Collection::setIsPageLimited(bool = true)
.Collection Collection::setIsPageLimited(false)
bool Collection::getIsPageLimited()
.Fixed Issues (if relevant)
Manual testing scenarios