[5.3.y] pi4 bcm2835-dma issues with kernel 5.3.y on aarch64

[asavah]

Describe the bug
Currently kernel 5.3.y as built from this repo by using bcm2711_defconfig has broken bcm2835-dma

To reproduce
Build the kernel from rpi-5.3.y branch with ARCH=arm64.
Observe dmesg output.
Observe framebuffer not working.
Observe mmc working in PIO mode.

Expected behaviour
bcm2835-dma and by consequence fb works.

Actual behaviour
bcm2835-dma probe fails at the very beginning.
no framebuffer present.
several dma related failures can be seen in dmesg (see below)

System

• Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW
  pi4 4GB
• Which OS and version (cat /etc/rpi-issue)?
  my own, unimportant
• Which firmware version (vcgencmd version)?

Sep 20 2019 18:17:38
Copyright (c) 2012 Broadcom
version 438bd10f818f4fcb6fbe27aaec6b2ac3de84a2de (clean) (release) (start_x)

• Which kernel version (uname -a)?
  5.3.0

Logs
dmesg from 5.3.y boot
https://pastebin.com/0MHc46f1

Additional context
To narrow the search let me highlight a few entries from dmesg

[    0.205191] bcm2835-dma: probe of fe007000.dma failed with error -5
[    1.046967] pci 0000:00:00.0: failed to get arch_dma_ops
[    1.053954] bcm2708_fb soc:fb: FB found 1 display(s)
[    1.054663] bcm2708_fb soc:fb: Couldn't allocate a DMA channel
[    1.054719] bcm2708_fb soc:fb: probe failed, err -19
[    1.584368] mmc-bcm2835 fe300000.mmcnr: mmc_debug:0 mmc_debug2:0
[    1.584380] mmc-bcm2835 fe300000.mmcnr: mmc-bcm2835: Unable to initialise DMA channel. Falling back to PIO

While with 4.19.y everything works as expected

[    0.047945] bcm2835-dma fe007000.dma: DMA legacy API manager at (____ptrval____), dmachans=0x1

[pelwell]

This regression appeared in rpi-4.20.y. There has been a lot of churn in the DMA support, and some of my forward porting has had the unintended effect of causing SWIOTLB to be ignored when it come to deciding if DMA is supported.

[pelwell]

Reverting 66bdb14 (messy, because the file involved has moved) in rpi-4.20.y is one fix/workaround. However, swiotlb_dma_ops doesn't exist in rpi-5.0.y and later so another (as yet undetermined) tactic is needed.

[lategoodbye]

AFAIU @vianpl is working on this issue:
https://lore.kernel.org/lkml/[email protected]/T/

Never test this series with rpi-5.3.

[pelwell]

That series looks interesting, but from a quick reading it's not a complete solution.

I think the main problem arises from the fact that swiotlb use has been pushed down under the dma_direct case, but dma_direct_supported doesn't know that. It seems as though the driver wanting to use DMA has to claim that it can address all the RAM and rely on dma_direct_map_page calling dma_direct_possible (which calls dma_capable) to decide whether or not to use swiotlb_map, which is counter-intuitive.

[vianpl]

Hi @pelwell and @lategoodbye, here's my take on the issue, haven't had the time to test it though:

With e127ac0 you fixed the ZONE_DMA32 size for the RPi4 yet didn't inform dma-direct about the DMA mask limitation. Specially since ZONE_DMA32 implicitly means 32bit DMA masks are supported. This patch should fix it:

diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c
index 706113c6bebc..643f479c12dd 100644
--- a/kernel/dma/direct.c
+++ b/kernel/dma/direct.c
@@ -397,7 +397,7 @@ int dma_direct_supported(struct device *dev, u64 mask)
        if (IS_ENABLED(CONFIG_ZONE_DMA))
                min_mask = DMA_BIT_MASK(ARCH_ZONE_DMA_BITS);
        else
-               min_mask = DMA_BIT_MASK(32);
+               min_mask = DMA_BIT_MASK(30);
 
        min_mask = min_t(u64, min_mask, (max_pfn - 1) << PAGE_SHIFT);

Obviously it's pretty hacky and if it really fixes the issue, I suggest reverting e127ac0 and back-porting my series (which should be straightforward).

[pelwell]

Thanks, that makes sense (lowering the bar rather than jumping higher...) and does seem to fix the issue. I've pushed this to 5.2 and 5.3 for now and will back-port your changes at a later date after a bit more testing.

[vianpl]

Glad it helped 🙂

[asavah]

@pelwell @vianpl
Huge thanks for the quick fix.

[    0.000000]   DMA32    [mem 0x0000000000000000-0x000000003fffffff]
[    0.000000]   DMA32 zone: 3072 pages used for memmap
[    0.000000]   DMA32 zone: 0 pages reserved
[    0.000000]   DMA32 zone: 196608 pages, LIFO batch:63
[    0.021715] DMA: preallocated 1024 KiB pool for atomic allocations
[    0.058578] bcm2835-dma fe007000.dma: DMA legacy API manager at (____ptrval____), dmachans=0x1
[    0.820685] brcm-pcie fd500000.pcie: dmabounce: initialised - 32768 kB, threshold 0x00000000c0000000
[    0.862603] pci 0000:00:00.0: failed to get arch_dma_ops
[    0.924070] unimac-mdio unimac-mdio.-19: DMA mask not set
[    1.390898] Using Buffer DMA mode
[    1.391431] WARN::dwc_otg_hcd_init:1043: FIQ DMA bounce buffers: virt = ffffff80131a1000 dma = 0x00000000ecd10000 len=9024
[    1.409003] mmc-bcm2835 fe300000.mmcnr: DMA channel allocated
[    1.470610] mmc0: SDHCI controller on fe340000.emmc2 [fe340000.emmc2] using ADMA

All is good now.

[pelwell]

No problem - I just do what I'm told.