Replace queue with linked list #21

willmmiles · 2025-02-05T14:16:27Z

Replace the bounded queue with a linked list and "condvar" implementation, and replace the closed_slots system with double indirection via AsyncClient's own memory. This allows the system to correctly handle cases where it is not possible to allocate a new event while guaranteeing that the client's onDisconnect() will be run to free up any other related resources.

Key changes:

Replaces FreeRTOS fixed-size queue with a dynamic allocation list;
Removes CONFIG_ASYNC_TCP_QUEUE_SIZE queue size limit;
Makes AsyncClient's intrusive list support optional via CONFIG_ASYNCTCP_HAS_INTRUSIVE_LIST
Replace the closed slot system with a double indirection on AsyncClient's own _pcb member. Once initialized, this member is written only by the LwIP thread, preventing most races; and the AsyncClient object itself is never deleted on the LwIP thread.
Internal callbacks are made private

This draft rebases the changes from willmmiles/AsyncTCP:master to this development line. As this project is moving faster than I can keep up with, in the interests of making this code available for review sooner, I have performed only minimal testing on this port. It is likely there is a porting error somewhere.

Known issues as of this writing:

AsyncClient::operator=(const AsyncClient&) assignment operator is removed. The old code had implemented this operation with an unsafe partial move sematic, leaving the old object holding a dangling reference to the pcb. It's not clear to me what should be done here - copies of AsyncClient are not generally meaningful.
Poll coaelscing is not yet implemented. This could be done with a list walk, as before, or via storing pending poll event pointers in the AsyncClient objects.
Move construction and assignment is not yet implemented. It will require careful interlocking with the LwIP and async threads.
I'm about 85% confident there's still a race somewhere when an AsyncClient is addressed from a third task (eg. from an Arduino loop(), not LwIP or the async task). The fact that LwIP reserves the right to invalidate tcp_pcbs on its thread at any time after tcp_err makes this extremely challenging to get both strictly correct and performant. Core operations that pass to the LwIP thread are safe, but I think state reads (state(), space(), etc.) are still risky.
If strict memory bounds are required, a soft queue size limit could be reimplemented, with the caveat that each client's _end_event can ignore the limit to ensure onDisconnect gets run.

Future work:

lwip_tcp_event_packet_t::dns should be unbundled to a separate event object type from the rest of the lwip_tcp_event_packet_t variants. It's easily twice the size of any of the others; this will reduce the memory footprint for normal operation.
Event coaelscing could be considered for other types -- recv and send also permit sensible aggregation.

Replace the bounded queue with a linked list and condvar implementation, and replace the closed_slots system with double indirection via AsyncClient's own memory. This allows the system to correctly handle cases where it is not possible to allocate a new event and still ensure that the client's `onDisconnect()` will be queued for execution.

mathieucarbou · 2025-02-05T14:33:41Z

Wow! Thanks a lot!
@vortigont and @me-no-dev will surely have a look!

mathieucarbou · 2025-02-05T15:06:13Z

I tested this implementation in ESPAsyncWebServer:

autocannon -c 16 -w 16 -d 20 http://192.168.4.1 => works, about same perf
autocannon -c 64 -w 32 -d 20 -t 30 http://192.168.4.1 => works fine, no errors, but some requests dropped, aI see a lot of:

[327202][E][AsyncTCP.cpp:1535] _accept(): TCP ACCEPT FAIL

sse perf: for i in {1..16}; do ( count=$(gtimeout 30 curl -s -N -H "Accept: text/event-stream" http://192.168.4.1/events 2>&1 | grep -c "^data:"); echo "Total: $count events, $(echo "$count / 4" | bc -l) events / second" ) & done;

slower (about 50-100 events per second less) but i I did not see any queue overflow

slow requests (SlowChunkResponse.ino) works
request continuation works
and no memory leak for all, while when I run those tests on main, the heap decreases a lot.
websocket also works

So to me, this is a hug improvement over what we have.

mathieucarbou · 2025-02-05T15:08:33Z

src/AsyncTCP.h

 #ifndef CONFIG_ASYNC_TCP_MAX_ACK_TIME
 #define CONFIG_ASYNC_TCP_MAX_ACK_TIME 5000
 #endif

+#ifndef CONFIG_ASYNCTCP_HAS_INTRUSIVE_LIST
+#define CONFIG_ASYNCTCP_HAS_INTRUSIVE_LIST 1


what is the use of that ?

First porting error - both the name and the implementation are bad. :(

There's a strange feature in the original code: AsyncClient has an intrusive list integrated (the prev/next pointers), but it's unclear as to why or what it's useful for. I removed it in my branch to save RAM as nobody in WLED (AsyncWebServer, AsyncMQTT) actually uses it for anything. I was trying to make it conditional, but default on. Will fix.

(default on for backwards compatibility, in case there is someone using it)

Well, I don' see any reason to keep this linked list... Do you @me-no-dev ? This is not even used.

mathieucarbou · 2025-02-05T17:34:43Z

@willmmiles : fyi as seen with @me-no-dev we will merge #19 first, do a v3.3.4, then focus on reviewing / merging you PR.
And we will release a new 4.0.0 once merged.
So you can remove the operator overrides, and prev/next fields that are anyway unused internally.
Thanks 👍

willmmiles · 2025-02-05T18:08:57Z

@willmmiles : fyi as seen with @me-no-dev we will merge #19 first, do a v3.3.4, then focus on reviewing / merging you PR. And we will release a new 4.0.0 once merged. So you can remove the operator overrides, and prev/next fields that are anyway unused internally. Thanks 👍

Sounds good, will do!

The semantics of this operation were error-prone; remove it to ensure safety.

vortigont · 2025-02-06T06:00:57Z

I had just a short glimpse here, an interesting approach indeed. But also have a lot of things to help me to understand.
First I like the idea of giving up the slot system, but replacing the rtos queue with a poor-man's linked list brings all the burden of it's integrity into lib's code, which is - sync/locking, length management, etc...

first thing first - how the list size is controlled there? Does it have any limits or it can grow as much as resources are there?

willmmiles · 2025-02-06T15:38:41Z

First I like the idea of giving up the slot system, but replacing the rtos queue with a poor-man's linked list brings all the burden of it's integrity into lib's code, which is - sync/locking, length management, etc...
first thing first - how the list size is controlled there? Does it have any limits or it can grow as much as resources are there?

This draft makes no attempt to limit the queue length, so long as there's heap available. A strictly fixed size queue is impractical because we must ensure that disconnection events cannot be dropped, or resources will leak. It's possible to add a soft upper limit on non-critical events, but it didn't seem to be worth the extra complexity (or having to explain an arbitrary resource limit that's independent of what the heap can actually service). Implementing event coalescence for poll, recv, and sent events will put a functional upper bound on the queue size as well, based on the number of open connections.

The rationale to replace the queue breaks down as:

(Correctness) There exist events that must never be dropped, so maintaining a strict upper limit requires an extremely complicated dance to invalidate accepted events (and abort yet more connections!) if the queue should fill up
(Correctness) The "purge events on close" sequence requires an external lock on the queue, or there's a potential risk that events will be re-inserted out of order if new events arrive during the operation. Using the RTOS queue means paying the locking penalty twice (since it has an internal lock).
(Correctness) Available list libraries in std will crash if allocation fails.
(Performance) The "purge events on close" requires locking the queue only once, instead of 2*N times
(Performance) The local queue implementation minimizes the amount of code space required -- it is significantly smaller than std::forward_list
(Simplicity) The "purge events on close" becomes trivial code.
(Simplicity) Once an event has been allocated, it is guaranteed that it can be added to the queue; this eliminates all the code paths that have to free the new event should the queue turn out to be full.

I'm usually the first to recommend using library implementations for classic data structures; ultimately I judged that the maintenance burden for the limited requirements here was less than the cost of bringing in some external library, given that std is unusable for my use case (specifically: not crashing when low on memory). If you know of a suitable alternative library (especially one that allows pre-allocation of items before locking the queue) and you're happy with bringing in a new dependency, I've otherwise got no objections.

Replacing the close_slot system is otherwise straightforward (it's nothing more than strict ownership of the pcb* by the LwIP thread), but it is contingent on guaranteed disconnect event delivery. I did originally implement these as separate changes, but since it wasn't going to merge cleanly with the development line from my old fork, I judged it wasn't worth trying to break it down in to separate commits.

mathieucarbou · 2025-02-06T15:53:08Z

@willmmiles : fyi asynctcp is released and we'll do an asyncwebserver release tomorrow with the current v3.3.5 of asynctcp.

So we are good to refresh this PR and have time to reviewed / merged.

When i tested with autocannon: autocannon -c 64 -w 32 -d 20 -t 30 http://192.168.4.1, or different variants at 16 connections or more, I always found this implementation more stable, compared to the current one in main, which as these issues:

memory leak, and a lot! Impossible to run twice the same test
and logs some error logs regarding inability to allocate, ack timeout, etc

I didn't test the client part yet. Will do later.

No reason for this to require a function call.

Use new(std::nothrow) instead.

If any of the TCP callbacks fails to allocate, return ERR_MEM.

mathieucarbou · 2025-02-06T20:11:44Z

src/AsyncTCP.cpp

+    tcp_recv(pcb, &_tcp_recv);
+    tcp_sent(pcb, &_tcp_sent);
+    tcp_err(pcb, &_tcp_error);
+    tcp_poll(pcb, &_tcp_poll, 1);


maybe use a constant here ? it was CONFIG_ASYNC_TCP_POLL_TIMER before

# 2 on the merge failure list!

Fixed in 2321755

mathieucarbou · 2025-02-06T20:17:09Z

src/AsyncTCP.cpp

+  tcpip_api_call(_tcp_connect_api, (struct tcpip_api_call_data *)&msg);
+  return msg.err == ESP_OK;


there is an issue with the client part which is not working (testing with the Client.ino).

connect() returns false from here.

I've changed the code to get the error:

tcpip_api_call(_tcp_connect_api, (struct tcpip_api_call_data *)&msg); if(msg.err != ERR_OK) { log_e("tcpip_api_call error: %d", msg.err); } return msg.err == ESP_OK;

[ 1305][E][AsyncTCP.cpp:791] _connect(): tcpip_api_call error: -16

-16 is the invalid arg error.

Hope that helps!

And there's merge failure #3 - I missed an underscore (and put the lines in the wrong place) when importing the locking code.

Fixed in 26e49f6

Should be using the config setting.

We can store the newly allocated handle directly in the object member.

We don't plan on writing to it - might as well save the copying.

willmmiles · 2025-02-07T00:59:33Z

I've put a prototype event coalescing branch at https://github.com/willmmiles/AsyncTCP/tree/replace-queue-coalesce . My basic tests seem to work, but I don't yet have a good test case to really exercise the new logic.

vortigont · 2025-02-07T05:17:14Z

@willmmiles agreed to most of your points above, with proper coalescing code the events chain size cap won't be that critical. And the benefits of correctness does worth the efforts on proper locking code. Good job indeed!

I'm not sure I understood your point on why std::fwd_list is not usable here, list node could be created with proper alloc exception handling, then locking the queue and moving the node to list chain. I do not see much problem here, but on the second thought it might be a good choice to use simplified list structs code here. std containers in esp32 is linked against a quite large lib that can't be stripped down and affects fw size considerably. On one hand we already use those containers in AsyncWebServer code, so it will bring cxx lib anyway, but on the other hand if we consider AsyncTCP as a standalone lib that might be used elsewhere it would be wise to minimize the dependencies that considerably affects resulting fw size. Point taken!

me-no-dev · 2025-02-07T06:32:33Z

Hi folks! Just trying to understand the issues that this PR is hoping to fix. Is it just the fact that events might be missed, because Queue is overflown, or are there also other things that are being fixed also? Please excuse my ignorance. I've been out of the game for a bit :)

mathieucarbou · 2025-02-07T09:45:34Z

Hi folks! Just trying to understand the issues that this PR is hoping to fix. Is it just the fact that events might be missed, because Queue is overflown, or are there also other things that are being fixed also? Please excuse my ignorance. I've been out of the game for a bit :)

Here are the problematic behaviour that we currently have and that this implementation seems to solve

Memory leak: currently asynctcp has one or many memory leaks which can be easily seen by running the Perf example with autocannon running like 32 concurrent connections. A big amount of heap is lost, and the test cannot be run twice: MCU crashes because cannot allocate
Running several concurrent connections with asynctcp also creates some ack timeout I saw sometimes and memory errors linked to the above.
SSE events lost: currently, when running 16 concurrent SSE connections, the queue fills up quickly and discards some events, while I do not see events discarded with this implementation. The send rate with this implementation is a little higher so that might be the reason

Plus all the valid points Will explained above. I think it makes sense to inclue his changes following the many efforts he did in stabilising this library in the case of WLED.

@willmmiles : I will retest your changes and the replace-queue-coalesce branch also, we have 2 use cases.

This simplies the queue operations.

willmmiles · 2025-02-17T16:44:17Z

I've found the race: if the async thread is in the middle of processing an event that results in destruction of the AsyncClient (fin, or a sent that completes a transaction) when an a _tcp_error comes in, the _end_event gets freed before dequeuing and bad things happen. I have a fix in mind but it'll take another day or two to get enough hours to fully validate it.

For clarity of implementation

Eliminates any possibility that a client may have its end event queued twice. LwIP should prevent this anyways, but it never hurts to be safe.

mathieucarbou · 2025-02-24T09:38:32Z

src/AsyncTCP.cpp

+  // The associated pcb is now invalid and will soon be deallocated
+  // We call on the preallocated end event from the client object
+  lwip_tcp_event_packet_t *e = AsyncClient_detail::invalidate_pcb(*client);
+  assert(e);


could e be null here if client._end_event is null following a bad allocation ?

_tcp_error isn't bound to the pcb unless the end_event could be allocated.

Ensure that the discard event is run once and exactly once for each client.

It's unclear why this might be necessary - LwIP oughtn't be triggering new callbacks after the error callback; but just to be on the safe side, make sure we have no dangling references in the pcb we are releasing. Related to the changes in me-no-dev#31.

The existing TCP_MUTEX_(UN)LOCK macros are not reentrant nor safe to call on code that might get run on the LwIP thread. Convert to a C++ lock_guard-style object, which is safer. cleans up some code paths, and correctly supports AsyncClient's constructor.

This gives access to AsyncClient internals, preparing for O(1) coalescing.

This is preparing to expand the guard scope to include other client object members for event coaelscense.

If one of these events is pending on a client, subsequent callbacks can be safely rolled together in to one event.

It's redundant in this design; we block forever

Add a feature to use the LwIP lock for the queue lock. This reduces the load on the LwIP thread, but might have the async thread wait longer.

Saves a little space when CONFIG_ASYNC_TCP_QUEUE_LWIP_LOCK is set

willmmiles · 2025-03-01T03:46:05Z

This seems to be properly stable for me, now - the race is resolved and I've put in extra checking with the discard callback to catch any as yet unconsidered cases for double-free. I've also merged an equivalent to #31, to be on the safe side; as I understood the docs, LwIP shouldn't trigger any more callbacks after tcp_err, but it doesn't hurt (much) to do the extra cleanup to ensure our own invariants are met.

I pulled the queue code out to a separate file to make it easier to inspect. I actually tried a version that used std::unique_ptr<> to validate the event pointer management; it functioned, though at a code size penalty and performance regression. :( I don't love that the header is exposed to users; it's easy enough to move the public header to "include" for platformio, but I don't currently have test environments for Arduino or ESP-IDF package management to ensure they'll also pick it up. Would it be better to embed it back in the cpp file?

Rebased coalesce support is rebased on replace-queue-coalesce. I tried three different locking approaches: a "double lock" approach where it locks to check for coalescing, releases it to malloc, and again to enqueue; a "lock over whole handler" approach (tip default), and a "just use the LwIP lock" (CONFIG_ASYNC_TCP_QUEUE_LWIP_LOCK) option. My test case results suggested that all of these came with some performance penalty, and the larger lock scopes seemed worse - somewhat surprising as I would've thought the semphore acquire/release would dominate. I have not yet tried using atomics instead of locking for poll and sent; and it may be that recv coalescence isn't worth bothering with (though there's an simplification to be had there in eliminating the need for _free_event).

Is there anything else I should look at?

mathieucarbou · 2025-03-01T07:35:08Z

@willmmiles thank you very much!
Yes if you could please rebase on main, then I will retest the PR carefully and @me-no-dev and @vortigont will need to take time to review it.

Instead of pointing to the chain, point to the event. Fixes handling of cases with nonzero error events mid-sequence. This requires clarifying the _remove_events/invalidate_pcb.

willmmiles · 2025-03-16T20:46:04Z

Unfortunately it wasn't really practical to rebase - you've adopted some of the fixes directly, and merging via rebase was just a nightmare of conflicts. This branch does too much all at once to keep up with the ongoing development and maintaince in the main line given my time limitations. I did some manual merges in my development line with coalescing support, which were painful enough that I'm not sure if they're worth repeating.

For the sake of discussion, I've pushed the full coalescence development line here in this draft for discussion. The line now has several different approaches to coalescing:

"double locking" -- taking the queue lock, coalescing if possible; otherwise releasing the queue lock and going through the existing logic. Advantage is the queue lock is held for the smallest amount of time; disadvantage is needing to lock twice on the new event path.
"single locking" -- holding the queue lock over the entire handler scope, resulting in fewer lock operations. Load testing suggests this is a performance loss compared to the double locking solution.
"single locking using LwIP lock" -- Dead end. Only really viable on newer platforms. Slower still than explicit single locking.
Atomics, viable for poll and sent but not recv. No noticable performance change from "single locking" solution; costs about 250 bytes of code compared to the "single locking" system.

Personally I think either double locking or atomics are the way to go. The value of recv coalescence is IMO somewhat debatable; it might save queuing some events, but adds a risk of unfairness/starvation as the event handlers might end up processing more data at once.

If there's no hurry, what I think might be bet the best way to go is to do a "logical rebase" from main, one key change at a time, to make it easier to review each step in isolation. I'll leave this PR open if there's any value in testing the end state as-is. The basic change sequence I suggest is:

Removal of intrusive list
Shallow queue replacement with minimal/no logic changes
Close slots removal (fixes possible race with re-used slot)
end_event preallocation (fixes possible case of unqueuable error event resulting in object leak)
Any other safety-related changes and/or validation
Additional coalescence, if useful.

Does this make sense?

vortigont · 2025-03-20T06:11:50Z

@willmmiles sounds reasonable, indeed the differences with main branch is quite complex and I'm not sure what would be more challenging - merge your changes into main or port upstream changes back to your branch.
As for coalescing - a large field for discussions :) let's take it piece by piece. I'm here to help with merging if you you can split/delegate tasks probably.

mathieucarbou assigned willmmiles Feb 5, 2025

mathieucarbou requested a review from a team February 5, 2025 14:32

mathieucarbou reviewed Feb 5, 2025

View reviewed changes

This comment was marked as off-topic.

Sign in to view

willmmiles added 3 commits February 5, 2025 19:30

Fix pre-commit suggestions

148ee43

Remove AsyncClient intrusive list

673fbc9

Remove AsyncClient::operator=(const AsyncClient&)

29d7ee5

The semantics of this operation were error-prone; remove it to ensure safety.

willmmiles added 5 commits February 6, 2025 17:26

Explicitly indicate non-copyability

7e0725a

Inline AsyncClient::operator==

5fdb222

No reason for this to require a function call.

Say goodbye to malloc

f9191a9

Use new(std::nothrow) instead.

Consistently return ERR_MEM

3caa027

If any of the TCP callbacks fails to allocate, return ERR_MEM.

Merge remote-tracking branch 'upstream/main'

affbd22

mathieucarbou reviewed Feb 6, 2025

View reviewed changes

willmmiles added 4 commits February 6, 2025 17:58

Fix tcp_poll timer

2321755

Should be using the config setting.

Fix merge error in _connect

26e49f6

We can store the newly allocated handle directly in the object member.

AsyncClient::_connect: Pass addr by const reference

f3a5256

We don't plan on writing to it - might as well save the copying.

Revert incorrect debug prints

50e0086

Use double indirection for _async_queue_tail

9d3dec1

This simplies the queue operations.

willmmiles mentioned this pull request Feb 17, 2025

Directly reset callbacks without going through lwip functions (fix #30) #31

Merged

willmmiles added 5 commits February 17, 2025 19:01

Break queue out to class

2df8999

For clarity of implementation

Gift end event to queue

96aee4b

Eliminates any possibility that a client may have its end event queued twice. LwIP should prevent this anyways, but it never hurts to be safe.

simple_intrusive_list: Use typedefs

b862fbc

simple_intrusive_list: Add validation call

28ab534

Expand _remove_events_for debug

e763c29

mathieucarbou reviewed Feb 24, 2025

View reviewed changes

willmmiles added 11 commits February 24, 2025 14:58

Clarify discard execution

bbd42d0

Ensure that the discard event is run once and exactly once for each client.

simple_intrusive_list: Add pragma once

74dd111

Teardown pcb on invalidate

6de4a1d

It's unclear why this might be necessary - LwIP oughtn't be triggering new callbacks after the error callback; but just to be on the safe side, make sure we have no dangling references in the pcb we are releasing. Related to the changes in me-no-dev#31.

Move TCP callbacks in to AsyncClient_detail

c8efcff

This gives access to AsyncClient internals, preparing for O(1) coalescing.

Raise queue mutex guard scope to callbacks

c424ea3

This is preparing to expand the guard scope to include other client object members for event coaelscense.

Coalesce poll, recv, and sent events

8926f30

If one of these events is pending on a client, subsequent callbacks can be safely rolled together in to one event.

Refactor to minimize queue locking

97ca3f8

Remove guard test

c4a3de0

It's redundant in this design; we block forever

Add CONFIG_ASYNC_TCP_QUEUE_LWIP_LOCK

8c1a823

Add a feature to use the LwIP lock for the queue lock. This reduces the load on the LwIP thread, but might have the async thread wait longer.

Embed global mutex in queue_mutex_guard

152fb19

Saves a little space when CONFIG_ASYNC_TCP_QUEUE_LWIP_LOCK is set

willmmiles added 4 commits March 1, 2025 17:10

Merge remote-tracking branch 'upstream/main' into replace-queue-coalesce

997a256

Alternate approach to recv coalesce

91726a6

Instead of pointing to the chain, point to the event. Fixes handling of cases with nonzero error events mid-sequence. This requires clarifying the _remove_events/invalidate_pcb.

Use atomics instead of locking for sent+poll

ea2c3e4

Merge upstream/main into replace-queue-coalesce

1eb5ccb

willmmiles mentioned this pull request Apr 16, 2025

Code cleanup prior to queue replacement #50

Merged

willmmiles mentioned this pull request May 7, 2025

Replace queue v2 part2 #58

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace queue with linked list #21

Replace queue with linked list #21

willmmiles commented Feb 5, 2025

mathieucarbou commented Feb 5, 2025

mathieucarbou commented Feb 5, 2025

mathieucarbou Feb 5, 2025

willmmiles Feb 5, 2025

willmmiles Feb 5, 2025

mathieucarbou Feb 5, 2025

This comment was marked as off-topic.

mathieucarbou commented Feb 5, 2025

willmmiles commented Feb 5, 2025

vortigont commented Feb 6, 2025

willmmiles commented Feb 6, 2025

mathieucarbou commented Feb 6, 2025 •

edited

Loading

mathieucarbou Feb 6, 2025

willmmiles Feb 6, 2025 •

edited

Loading

willmmiles Feb 7, 2025

mathieucarbou Feb 6, 2025

willmmiles Feb 7, 2025

willmmiles commented Feb 7, 2025

vortigont commented Feb 7, 2025

me-no-dev commented Feb 7, 2025

mathieucarbou commented Feb 7, 2025 •

edited

Loading

willmmiles commented Feb 17, 2025

mathieucarbou Feb 24, 2025

willmmiles Feb 24, 2025

willmmiles commented Mar 1, 2025

mathieucarbou commented Mar 1, 2025 •

edited

Loading

willmmiles commented Mar 16, 2025

vortigont commented Mar 20, 2025

		tcpip_api_call(_tcp_connect_api, (struct tcpip_api_call_data *)&msg);
		return msg.err == ESP_OK;

Replace queue with linked list #21

Are you sure you want to change the base?

Replace queue with linked list #21

Conversation

willmmiles commented Feb 5, 2025

mathieucarbou commented Feb 5, 2025

mathieucarbou commented Feb 5, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as off-topic.

mathieucarbou commented Feb 5, 2025

willmmiles commented Feb 5, 2025

vortigont commented Feb 6, 2025

willmmiles commented Feb 6, 2025

mathieucarbou commented Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

willmmiles Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

willmmiles commented Feb 7, 2025

vortigont commented Feb 7, 2025

me-no-dev commented Feb 7, 2025

mathieucarbou commented Feb 7, 2025 • edited Loading

willmmiles commented Feb 17, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

willmmiles commented Mar 1, 2025

mathieucarbou commented Mar 1, 2025 • edited Loading

willmmiles commented Mar 16, 2025

vortigont commented Mar 20, 2025

mathieucarbou commented Feb 6, 2025 •

edited

Loading

willmmiles Feb 6, 2025 •

edited

Loading

mathieucarbou commented Feb 7, 2025 •

edited

Loading

mathieucarbou commented Mar 1, 2025 •

edited

Loading