[audio-server] Audio issues using alsa playback backend (drain error + sound is cut) #203

cburghardt · 2019-10-05T13:39:37Z

What's the issue

I just upgraded my system (Raspi 4 with Stretch packages) to 0.64. Unfortunately the error
ALSA function 'snd_pcm_drain' failed with error 'EIO: I/O error'
is back. None of the known workarounds fix it.
Once I downgraded to 0.63.3 the problem disappears.

Expected
Same behavior as 1.2.2

Version

1.3.0 (0.64.0)

Environment

Raspberry Pi 4 - Debian Stretch
Sound card - Raspberry pi sound card

[workaround edit by @cpoisson ]

Workaround 1 - Back to 0.63.3

Downgrade snips-tts and snips-audio-server to 1.2.2 (0.63.3)

sudo apt install snips-audio-server=0.63.3
sudo apt install snips-tts=0.63.3

Workaround 2 - Back to portaudio

Switch back to portaudio for playback.

Add portaudio_playback = "default" to the [snips-audio-server] section of your /etc/snips.toml configuration file.
sudo systemctl restart snips-audio-server

Workaround 3 - bcm2835

If you are using the pi 3 or pi 4 bcm2835 embedded sound card.

Switch back to portaudio
Use the following configuration for playback

pcm.!default {
    type asym
    playback.pcm {
        type plug
        slave.pcm "hw:0,0"
        slave.rate 48000  # Does not work if this is removed.
    }
    capture.pcm {
        type plug
        slave.pcm "hw:1,0"   # Configure the correct hw:Id,SubId for your capture device
    }
}

Workaround 4 : Delegate to dmix and dsnoop

Delegate to dmix to make sure that the sound is played well using alsa

pcm.!default {
    type asym
    playback.pcm "playback"
    capture.pcm "capture"
}

pcm.playback {
    type plug
    slave.pcm "dmixed"
}

pcm.capture {
    type plug
    slave.pcm "array"
}

pcm.dmixed {
    type dmix
    ipc_key 555555
    ipc_key_add_uid yes
    ipc_perm 0666
    slave.pcm "hw:1,0"  # Set the correct hw:Id,SubId for your playback device
}

pcm.array {
    type dsnoop
    ipc_key 666666
    ipc_key_add_uid yes
    ipc_perm 0666
    slave.pcm "hw:1,0"  # Set the correct hw:Id,SubId for your capture device
}

This configuration has been tested on the following devices

Audio device	Playback	Capture	Details
`bcm2835` (raspberry pi)	❌		Does not work. playback is stuck waiting for `/dev/snd/timer` (known issue)
`Jabra PHS002W`	✅	✅
`UGREEN USB Sound card`	✅	✅
`PS Eye Camera`		✅
`Respeaker PiHAT 2 mics`	✅	✅
`Respeaker 6 Microphones`	❌	✅	Playback does not work. The default configuration installed by the drivers works correctly tough

Related Issue

[audio-server] Output sound like TTS is cut quickly when it starts playing for some configurations #201 : Sound partial playback (duplicated)
[Platform][Audio-Server][PortAudio] Cannot play sound sampled at 22050Hz when playback is set at 16000Hz #148 : Other portaudio issue (22050 samples/s issue)

The text was updated successfully, but these errors were encountered:

cpoisson · 2019-10-07T07:30:49Z

Thanks for your ticket @cburghardt,

Quick question, Which sound card are you using as your main sound playback interface?

Can you post as well your asound.conf configuration.

Many thanks for your feedbacks

cburghardt · 2019-10-07T08:07:26Z

I use the onboard sound card of the raspberry.

pcm.jack {                  
        type hw             
        card 0              
        device 0            
        rate 16000          
} 

pcm.!default {
  type asym
   playback.pcm {
     type plug
     slave.pcm "jack"
   }
   capture.pcm {
     type plug
     slave.pcm "hw:1,0"
   }
}

cpoisson · 2019-10-07T15:08:29Z

Ok,

So the issue is indeed related to #201

Using alsa, the ring buffer parameters setup done by the audio server is incorrect, and seems to induce multiple issues:

snips-audio-server is deadlocked (probably waiting for the snd_pcm_drain (all audio samples are played) and need a restart.
sound is sometimes cut (work / does not work, depends on your luck + sample rate)

This issue seems to appears with different behavior depending on your sound card.

Pi 3 Embedded sound card

pi@snips-pi-qa-assistant:~ $ cat /proc/asound/card0/pcm0p/sub0/hw_params
access: MMAP_INTERLEAVED
format: S16_LE
subformat: STD
channels: 1
rate: 48000 (48000/1)
period_size: 512
buffer_size: 65536

Jabra

pi@snips-pi-qa-assistant:~ $ cat /proc/asound/card1/pcm0p/sub0/hw_params
access: MMAP_INTERLEAVED
format: S16_LE
subformat: STD
channels: 2
rate: 48000 (48000/1)
period_size: 256
buffer_size: 262144

Typical parameters induced by aplay

period_size: 6000; buffer_size: 24000 @ 48000Hz
period_size: 4000; buffer_size: 16000 @ 22050Hz
period_size: 2000; buffer_size: 8000 @ 16000Hz

What seems to work
Using the following configuration and portaudio backend. Things seems to work again.

pcm.!default {
    type asym
    playback.pcm {
        type plug
        slave.pcm "hw:0,0" # the pi3 sound card
        slave.rate 48000 # to avoid port audio sampling issues

    }
    capture.pcm {
        type plug
        slave.pcm "hw:2,0" # A usb microphone plugged on the my raspberry pi.
    }
}

Which give me the following hw parameters when playing a sound sample with the audio-server.

pi@snips-pi-qa-assistant:~ $ cat /proc/asound/card0/pcm0p/sub0/hw_params
access: MMAP_INTERLEAVED
format: S16_LE
subformat: STD
channels: 2
rate: 48000 (48000/1)
period_size: 836
buffer_size: 4184

The buffer size is back to a reasonable value. The sound issues disappears.

(I found some documentation for the one interested)

cpoisson · 2019-10-07T15:22:28Z

@marcteyssier, @cburghardt

I gave a try to your configuration (coming from the forum I suppose), using portaudio.

snips-audio-server -v --portaudio_playback=default

Seems to work correclty I don't see any errors.

Can you patch your /etc/snips.toml with portaudio_playback = "default" under [snips-audio-server] section, to give a shot to this workaround (1.3.0 only) ?

cburghardt · 2019-10-07T15:53:19Z

So I upgraded to 0.64 and applied the changes. Result:
caused by: an error occured in the audio pipeline: ErrorMessage { msg: "Couldn't find a PortAudio device with the name default" }

cpoisson · 2019-10-08T07:28:30Z

Ok, that's another one to figure out.

I also tried to delegate things to dmix. Works great with the piHAT 2 (provided the user using the sound is in the audio group)

But as soon as I switch on the raspberry pi even aplay deadlock. After an inspection with strace I think I stumbled on this known issue

marcteyssier · 2019-10-08T08:47:05Z

Hi, this work for me with my respeaker mic 4 array.
I heard all the sentence.

Thank you for your help

cpoisson · 2019-10-08T13:58:00Z

Regarding the bcm2835 (raspi sound card) it is really impossible to use it with dmix on my side. The thing just stay stuck.

Otherwise the following asound.conf configuration works perfectly on the hw available on my desk using the default alsa playback.

/etc/asound.conf


pcm.!default {
    type asym
    playback.pcm "playback"
    capture.pcm "capture"
}

pcm.playback {
    type plug
    slave.pcm "dmixed"
}

pcm.capture {
    type plug
    slave.pcm "array"
}

pcm.dmixed {
    type dmix
    ipc_key 555555
    ipc_key_add_uid yes
    ipc_perm 0666
    slave.pcm "hw:5,0"
}

pcm.array {
    type dsnoop
    ipc_key 666666
    ipc_key_add_uid yes
    ipc_perm 0666
    slave.pcm "hw:5,0"
}

Hardware tested

Audio device	Playback	Capture	Details
`bcm2835` (raspberry pi)	❌		playback is stuck waiting for `/dev/snd/timer` (known issue)
`Jabra PHS002W`	✅	✅
`UGREEN USB Sound card`	✅	✅
`PS Eye Camera`		✅
`Respeaker PiHAT 2 mics`	✅	✅

cburghardt · 2019-10-08T14:39:38Z

So how do I convince the audio server to use portaudio if that seems to work?
My main unit just uses the onboard raspi sound card for playback. Recording is done via satellites.

cpoisson · 2019-10-08T16:27:52Z

The test above was done using alsa for playback and capture.

I had the following configuration working on my side using portaudio on the bcm2835, it should match your needs

pcm.!default {
    type asym
    playback.pcm {
        type plug
        slave.pcm "hw:0,0"
        slave.rate 48000
    }
}

Test it along

sudo -u _snips snips-audio-server -v --disable-capture --portaudio_playback="default"

Fun fact, it stops working if you remove slave.rate 48000, with the same error as yours.

[18:27:16.007495] ERROR:snips_audio_server_hermes::player: An error happened while trying to play some audio: could not play "c38d0537-611b-4611-a81a-9862f2701d81"
 -> caused by: an error occured in the audio pipeline: ErrorMessage { msg: "Couldn\'t find a PortAudio device with the name default" }
 -> caused by: Couldn't find a PortAudio device with the name default

cburghardt · 2019-10-09T06:18:13Z

Result:

[08:17:03.272191] DEBUG:snips_audio_server_hermes::player: Playing bytes EWQUWP1YISAA0LP8
Expression 'alsa_snd_pcm_hw_params_set_period_size_near( pcm, hwParams, &alsaPeriodFrames, &dir )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 923
Expression 'alsa_snd_pcm_hw_params_set_period_size_near( pcm, hwParams, &alsaPeriodFrames, &dir )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 923
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.front
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround21
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround21
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround40
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround41
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround50
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround51
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround71
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.iec958
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.iec958
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.iec958
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib pcm_hw.c:1822:(_snd_pcm_hw_open) Invalid value for card
ALSA lib pcm_hw.c:1822:(_snd_pcm_hw_open) Invalid value for card
[08:17:03.321778] ERROR:snips_audio_server_hermes::player: An error happened while trying to play some audio: could not play "EWQUWP1YISAA0LP8"
 -> caused by: an error occured in the audio pipeline: ErrorMessage { msg: "Couldn\'t find a PortAudio device with the name default" }
 -> caused by: Couldn't find a PortAudio device with the name default

cpoisson · 2019-10-09T12:45:26Z

🤔 ok, now I'm running short of ideas.

Can you show me the content of aplay -l on your device?

This configuration works on 3 out of 3 Pi 3 boards using version 0.64.0

Here are the logs I obtain.

:~ $ snips-audio-server -v --disable-capture --portaudio_playback=default
[14:39:28.521619] DEBUG:snips_common_cli::cli: conf lookup: snips-audio-server.no_exit_on_all_panics -> Ok(None)
[14:39:28.521984] DEBUG:snips_common_cli::cli: conf lookup: snips-common.no_exit_on_all_panics -> Ok(None)
...
[14:39:36.319702] DEBUG:snips_audio_server_hermes::player: Playing bytes dummysessionid
Expression 'alsa_snd_pcm_hw_params_set_period_size_near( pcm, hwParams, &alsaPeriodFrames, &dir )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 923
Expression 'alsa_snd_pcm_hw_params_set_period_size_near( pcm, hwParams, &alsaPeriodFrames, &dir )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 923
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.front
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround21
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround21
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround40
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround41
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround50
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround51
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround71
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.iec958
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.iec958
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.iec958
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib pcm_asym.c:106:(_snd_pcm_asym_open) capture slave is not defined
[14:39:36.365710] INFO :snips_audio_portaudio            : Playing "dummysessionid" using output "default", wav spec: WavSpec { channels: 2, sample_rate: 22050, bits_per_sample: 16, sample_format: Int }
[14:39:36.372966] DEBUG:snips_audio_portaudio            : Padding output with 23 frames
[14:39:36.952786] INFO :snips_audio_portaudio            : Playing of "dummysessionid" finished
[14:39:36.954156] DEBUG:snips_audio_server_hermes::player: Finished Playing bytes dummysessionid

You should not have those following lines in your log

ALSA lib pcm_hw.c:1822:(_snd_pcm_hw_open) Invalid value for card
ALSA lib pcm_hw.c:1822:(_snd_pcm_hw_open) Invalid value for card

Looks like a configuration issue to me but I prefer to confirm that with your device information (aplay -l)

cburghardt · 2019-10-09T12:56:29Z

It's a pi 4 maybe that's the difference.

**** List of PLAYBACK Hardware Devices ****
card 0: ALSA [bcm2835 ALSA], device 0: bcm2835 ALSA [bcm2835 ALSA]
  Subdevices: 7/7
  Subdevice #0: subdevice #0
  Subdevice #1: subdevice #1
  Subdevice #2: subdevice #2
  Subdevice #3: subdevice #3
  Subdevice #4: subdevice #4
  Subdevice #5: subdevice #5
  Subdevice #6: subdevice #6
card 0: ALSA [bcm2835 ALSA], device 1: bcm2835 IEC958/HDMI [bcm2835 IEC958/HDMI]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 0: ALSA [bcm2835 ALSA], device 2: bcm2835 IEC958/HDMI1 [bcm2835 IEC958/HDMI1]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

Alelelex · 2019-10-09T15:01:06Z

Hi, I have the same problem here. I tried a lot of solution without success.
My error is a bit different is "sam watch" :
[Audio Server] reported an error on site default : An error happened while trying to play some audio

It doesn't work better by downgrading snips-tts and snips-audio-server to version 0.63.3.

On hardware side I tried with a Jabra PHS001U and a Respeaker 6mics with the same result for both.

Let me know if you need more informations.

cpoisson · 2019-10-09T15:13:47Z

@Alelelex It could be a very different issue. Let's try not to clutter this issue and deal that on the forum

cpoisson · 2019-10-10T11:18:52Z

@cburghardt

I could grab a Pi 4 on debian buster.

pi@snips-pi-qa-test:~ $ lsb_release -a
No LSB modules are available.
Distributor ID:	Raspbian
Description:	Raspbian GNU/Linux 10 (buster)
Release:	10
Codename:	buster

I do confirm that the following configuration is working with snips-audio-server 0.64.0 using portaudio backend for playback

pcm.!default {
    type asym
    playback.pcm {
        type plug
        slave.pcm "hw:0,0"
        slave.rate 48000
    }
}

(same test command)

 sudo -u _snips snips-audio-server -v --disable-capture --portaudio_playback=default

SvenSchiffner · 2019-10-11T08:49:48Z

I can confirm that the workaround works for me. But with portaudio I have to remove capture.pcm from asound.conf. Otherwise I also get the PortAudio default device error.

cburghardt · 2019-10-11T09:23:02Z

Thanks a lot @sven5s - that was the key. With that modification it works.

cpoisson · 2019-10-11T09:51:52Z

@cburghardt

Nice to heart that!

I'll keep it open to follow up this issue until we release a definitive fix.

cpoisson · 2019-10-17T14:59:58Z

For the record,

Trying to fix this issue we still stumbled on AudioError(OtherError(Error("snd_pcm_drain", Sys(EIO)) could not drain pcm)) using the bcm2835 embedded sound card.

It seems that there is a kernel known issue about that => raspberrypi/linux#2983

The current version is 4.19.66, rolling back to kernel version 4.19.36 or upgrading to 4.19.79 fix the issue on my side.

sudo rpi-update c50c65e52bad69e8fa7d6576d9e2399eade7faf6 => 4.19.36
sudo rpi-update 985bc5353e4f5fe5a11c8b6c4c646dc7165bbc21 => 4.19.79
Please read rpi-update documentation before using it.

I guess this particular issue will be solved in a next official kernel update.

thorstenMueller · 2019-10-22T14:02:29Z

I can confirm that the downgrade on 4.19.36 works for my setup (all snips components running 0.64.0, raspberry 3b running stetch).
As user of matrix voice i was unable to build kernel modules because of missing kernel headers for .36 kernel version.
After using "rpi-source" building matrix voice modules works again.

Andergraw · 2019-10-24T12:52:56Z

I was having issues with cutted sounds and unstable audio server and upgrading kernel to 4.19.79 solved them all.
Thanks!

cpoisson added the sub: audio-server Related to the audio server label Oct 7, 2019

cpoisson self-assigned this Oct 7, 2019

cpoisson added the type: bug Not working as expected label Oct 7, 2019

cpoisson added this to the Snips Flow - Next milestone Oct 7, 2019

cpoisson changed the title ~~[audio-server] alsa issue back after upgrade to 0.64~~ [audio-server] Multiple audio issues using alsa playback backend (drain error + sound is cut) Oct 7, 2019

cpoisson mentioned this issue Oct 7, 2019

[audio-server] Output sound like TTS is cut quickly when it starts playing for some configurations #201

Closed

cpoisson mentioned this issue Oct 8, 2019

[audio-server] Plop / Click artefact at the beginning of playback sound #198

Closed

cpoisson changed the title ~~[audio-server] Multiple audio issues using alsa playback backend (drain error + sound is cut)~~ [audio-server] Audio issues using alsa playback backend (drain error + sound is cut) Oct 8, 2019

cburghardt closed this as completed Oct 11, 2019

cpoisson reopened this Oct 11, 2019

cpoisson mentioned this issue Oct 17, 2019

Issue with latest snips version respeaker/seeed-voicecard#193

Closed

cpoisson mentioned this issue Nov 12, 2019

Snips-audio-server creates 90%+ Cpu loads even on idle in combination with respeaker gpio mics #213

Closed

[audio-server] Audio issues using alsa playback backend (drain error + sound is cut) #203

[audio-server] Audio issues using alsa playback backend (drain error + sound is cut) #203

Comments

cburghardt commented Oct 5, 2019 • edited by cpoisson Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

cpoisson commented Oct 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cburghardt commented Oct 7, 2019

Uh oh!

cpoisson commented Oct 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cpoisson commented Oct 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cburghardt commented Oct 7, 2019

Uh oh!

cpoisson commented Oct 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marcteyssier commented Oct 8, 2019

Uh oh!

cpoisson commented Oct 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cburghardt commented Oct 8, 2019

Uh oh!

cpoisson commented Oct 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cburghardt commented Oct 9, 2019

Uh oh!

cpoisson commented Oct 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cburghardt commented Oct 9, 2019

Uh oh!

Alelelex commented Oct 9, 2019

Uh oh!

cpoisson commented Oct 9, 2019

Uh oh!

cpoisson commented Oct 10, 2019

Uh oh!

SvenSchiffner commented Oct 11, 2019

Uh oh!

cburghardt commented Oct 11, 2019

Uh oh!

cpoisson commented Oct 11, 2019

Uh oh!

cpoisson commented Oct 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thorstenMueller commented Oct 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Andergraw commented Oct 24, 2019

Uh oh!

cburghardt commented Oct 5, 2019 •

edited by cpoisson

Loading

cpoisson commented Oct 7, 2019 •

edited

Loading

cpoisson commented Oct 7, 2019 •

edited

Loading

cpoisson commented Oct 7, 2019 •

edited

Loading

cpoisson commented Oct 8, 2019 •

edited

Loading

cpoisson commented Oct 8, 2019 •

edited

Loading

cpoisson commented Oct 8, 2019 •

edited

Loading

cpoisson commented Oct 9, 2019 •

edited

Loading

cpoisson commented Oct 17, 2019 •

edited

Loading

thorstenMueller commented Oct 22, 2019 •

edited

Loading