Skip to content

ext4 regression causes huge load #5097

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
lategoodbye opened this issue Jul 19, 2022 · 6 comments
Closed

ext4 regression causes huge load #5097

lategoodbye opened this issue Jul 19, 2022 · 6 comments

Comments

@lategoodbye
Copy link
Contributor

lategoodbye commented Jul 19, 2022

Describe the bug

Recently users of Arch Linux ARM complained about huge load on IO task like unpacking bigger tar archives starting with Linux 5.18. Because the error descriptions was mixed with SD cards issues it was hard to identify and i suspect the ext4 partition must also be in a "used" state.

Last week i found a good scenario to bisect the issue. Here is the related upstream bug report:
https://marc.info/?l=linux-ext4&m=165815090631395&w=3

Unfortunately the offending change 3c65b73 has been backported to Linux 5.15.33, but a revert of the commit should fix the problem.

Steps to reproduce the behaviour

  • find a SD card with Raspberry Pi OS a used ext4 rootfs
  • make sure Linux kernel 5.15.33 (RPI and mainline are affected) is installed
  • run "sudo rpi-update"

Expected behavior:

  • update is download and installed
  • ACT LED goes back to normal after update

Observed behavior:

  • update hangs during download and abort or takes very long
  • ACT LED stays green even after abort

Device (s)

Raspberry Pi 4 Mod. B

System

Kernel version: 5.15.55

Logs

No response

Additional context

No response

pelwell added a commit that referenced this issue Jul 19, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
pelwell added a commit that referenced this issue Jul 19, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
pelwell added a commit that referenced this issue Jul 19, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
@pelwell
Copy link
Contributor

pelwell commented Jul 19, 2022

Thanks - that's reverted on the current branches.

@pelwell pelwell closed this as completed Jul 20, 2022
popcornmix added a commit to raspberrypi/firmware that referenced this issue Jul 22, 2022
kernel: dtoverlays: Add nohdmi options to vc4-kms-v3d overlays
raspberrypi/linux#5099

kernel: overlays: Make more overlays runtime-capable
See: raspberrypi/linux#5101

kernel: overlays: Mark more overlays as Pi4-specific

kernel: Revert ext4: make mb_optimize_scan performance mount option work with extents
See: raspberrypi/linux#5097

kernel: configs: Enable IIO software trigger modules
See: raspberrypi/linux#4984

kernel: configs: Enable IP_VS_IPV6 (for loadbalancing)
See: raspberrypi/linux#2860

kernel: configs: Enable CEPH_FS=m
See: raspberrypi/linux#2916

firmware: arm_loader: initramfs over NVME fix
See: #1731
popcornmix added a commit to raspberrypi/rpi-firmware that referenced this issue Jul 22, 2022
kernel: dtoverlays: Add nohdmi options to vc4-kms-v3d overlays
raspberrypi/linux#5099

kernel: overlays: Make more overlays runtime-capable
See: raspberrypi/linux#5101

kernel: overlays: Mark more overlays as Pi4-specific

kernel: Revert ext4: make mb_optimize_scan performance mount option work with extents
See: raspberrypi/linux#5097

kernel: configs: Enable IIO software trigger modules
See: raspberrypi/linux#4984

kernel: configs: Enable IP_VS_IPV6 (for loadbalancing)
See: raspberrypi/linux#2860

kernel: configs: Enable CEPH_FS=m
See: raspberrypi/linux#2916

firmware: arm_loader: initramfs over NVME fix
See: raspberrypi/firmware#1731
popcornmix pushed a commit that referenced this issue Aug 1, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Aug 1, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Aug 1, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
herrnst pushed a commit to herrnst/linux-raspberrypi that referenced this issue Aug 7, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: raspberrypi#5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Aug 15, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Aug 15, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Aug 15, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
pelwell added a commit that referenced this issue Aug 18, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Aug 23, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Aug 23, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Aug 23, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
herrnst pushed a commit to herrnst/linux-raspberrypi that referenced this issue Aug 23, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: raspberrypi#5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Aug 23, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
herrnst pushed a commit to herrnst/linux-raspberrypi that referenced this issue Aug 25, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: raspberrypi#5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Aug 31, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Aug 31, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Sep 5, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Sep 5, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
@lategoodbye
Copy link
Contributor Author

Hi, just a short update on this issue after a lot of work. There is a patch series which hopefully will be applied soon:
https://lore.kernel.org/all/[email protected]/

@popcornmix
Copy link
Collaborator

Thanks for info. I'm guessing it will be backported to linux-5.15.y and I'll know to revert the revert when the conflict comes.

popcornmix pushed a commit that referenced this issue Sep 12, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Sep 12, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Sep 26, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
herrnst pushed a commit to herrnst/linux-raspberrypi that referenced this issue Sep 28, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: raspberrypi#5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Oct 3, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Oct 5, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Oct 12, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Oct 12, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
herrnst pushed a commit to herrnst/linux-raspberrypi that referenced this issue Oct 12, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: raspberrypi#5097

Signed-off-by: Phil Elwell <[email protected]>
herrnst pushed a commit to herrnst/linux-raspberrypi that referenced this issue Oct 12, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: raspberrypi#5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Oct 17, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Oct 17, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Oct 17, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
herrnst pushed a commit to herrnst/linux-raspberrypi that referenced this issue Oct 17, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: raspberrypi#5097

Signed-off-by: Phil Elwell <[email protected]>
mj22226 pushed a commit to mj22226/linux that referenced this issue Oct 18, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: raspberrypi/linux#5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Oct 19, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Oct 20, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Oct 21, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Oct 24, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Oct 25, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Oct 26, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Oct 31, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Oct 31, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Nov 6, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Nov 7, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
popcornmix pushed a commit that referenced this issue Nov 7, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: #5097

Signed-off-by: Phil Elwell <[email protected]>
@lategoodbye
Copy link
Contributor Author

lategoodbye commented Nov 11, 2022

@popcornmix No, need to revert this patch in rpi-5.15.y anymore. The real fixes has been backported at the end of September:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/fs/ext4/mballoc.c?h=v5.15.78&id=fd8b829195495a0b3d6b3cc88c3b77f2c9c115e4

popcornmix added a commit that referenced this issue Nov 12, 2022
…work with extents""

This reverts commit e05a335.

The revert should no longer be necessary after upstream:
ext4: avoid unnecessary spreading of allocations among groups

See: #5097
Signed-off-by: Dom Cobley <[email protected]>
@popcornmix
Copy link
Collaborator

Reverted the revert. Thanks for heads up.

herrnst pushed a commit to herrnst/linux-raspberrypi that referenced this issue Nov 26, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: raspberrypi#5097

Signed-off-by: Phil Elwell <[email protected]>
herrnst pushed a commit to herrnst/linux-raspberrypi that referenced this issue Dec 3, 2022
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: raspberrypi#5097

Signed-off-by: Phil Elwell <[email protected]>
herrnst pushed a commit to herrnst/linux-raspberrypi that referenced this issue Jan 2, 2023
…h extents"

This reverts commit 3c65b73.

The reverted commit is likely to also be reverted upstream, but with a
performance regression like this I don't want to wait for that.

See: raspberrypi#5097

Signed-off-by: Phil Elwell <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants