Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crowsnest Setting WLAN1 Network Interface DOWN #276

Closed
zimmertr opened this issue Oct 24, 2024 · 20 comments
Closed

Crowsnest Setting WLAN1 Network Interface DOWN #276

zimmertr opened this issue Oct 24, 2024 · 20 comments
Labels
bug Something isn't working

Comments

@zimmertr
Copy link

What happened

I recently flashed my printer to run Klipper which is driven by a Raspberry Pi 5. The Pi is installed beneath the printer in it's electronics housing which results in poor wireless reception. So I purchased a USB wireless adapter with an antenna that sits outside of the printer. This works great, however, I've been chasing down a bug where the Pi would randomly disconnect from my network when I would interact with Mainsail, Fluidd, or KlipperScreen. I believe the ultimate catalyst if the bug in all of these cases is Crowsnest.

The Internal wireless chip is wlan0 and the USB wireless adapter is wlan1. I run Raspberry Pi OS Lite which uses NetworkManager. I've defined a connection profile for wlan1 but not wlan0. So, on boot, wlan1 comes up and gets an IP from my router through a static ip reservation.

This all works great, and I can use Mainsail, KlipperScreen, etc just fine. But as soon as I try and load any UI element that loads the webcam, the wlan1 interface gets set DOWN and networking breaks. Locally on KlipperScreen, Klipper continues to function just fine. So it's not crashing or anything. The networking just breaks.

I poked through the code and I'm not sure where this could be happening, but I suspect it has to do with how ustreamer binds to a NIC. I don't see any configuration options documented that allow this to be configurable.

It does eventually recover and wlan0 comes back up. I'll share some STDOUT from ping showing this in the Additional information section below. But it takes minutes.

Here is a screen recording demos the bug occurring around 1:40: https://www.youtube.com/watch?v=WBQlUoP5Dfc

What did you expect to happen

Crowsnest to support multiple network interfaces, not default to the first one, and to work with any of them

How to reproduce

Run Klipper, Mainsail, Crowsnest on a device with multiple wireless NICs and a USB Webcam. Have the first NIC DOWN and the second NIC UP. Load the webcam and observe the second NIC be set to DOWN.

Additional information

Crownest Config

[crowsnest]
log_path: ~/printer_data/logs/crowsnest.log
log_level: debug
delete_log: false
no_proxy: false

[cam 1]
mode: mjpg
port: 8080
device: /dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_809E7D1F-video-index0
resolution: 1280x720
max_fps: 60
v4l2ctl: focus_automatic_continuous=0,focus_absolute=70,brightness=100,contrast=100,saturation=100,sharpness=200,backlight_compensation=1
custom_flags: --format=yuyv

Network Manager Connection Config

[connection]
id=preconfigured
uuid=d2d19ac7-9b76-4a81-9243-863e09772559
type=wifi
interface-name=wlan1

[wifi]
mode=infrastructure
ssid=Hubble
hidden=false

[ipv4]
method=auto

[ipv6]
addr-gen-mode=default
method=auto

[proxy]

[wifi-security]
>REDACTED<

ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 2c:cf:67:5e:63:12 brd ff:ff:ff:ff:ff:ff
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 2c:cf:67:5e:63:13 brd ff:ff:ff:ff:ff:ff
4: wlan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 54:07:7d:85:9e:5e brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.170/24 brd 192.168.30.255 scope global dynamic noprefixroute wlan1
       valid_lft 7197sec preferred_lft 7197sec
    inet6 fe80::970a:645c:f90b:1371/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

Crowsnest Debug Log

[10/24/24 13:21:50] crowsnest: crowsnest - A webcam Service for multiple Cams and Stream Services.
[10/24/24 13:21:50] crowsnest: Version: v4.1.9-1-gd75a3ae
[10/24/24 13:21:50] crowsnest: Prepare Startup ...
[10/24/24 13:21:50] crowsnest: INFO: Host information:
[10/24/24 13:21:50] crowsnest: Host Info: Distribution: Debian GNU/Linux 12 (bookworm)
[10/24/24 13:21:50] crowsnest: Host Info: Kernel: Linux 6.6.51+rpt-rpi-2712 aarch64
[10/24/24 13:21:50] crowsnest: Host Info: Model:  Raspberry Pi 5 Model B Rev 1.0
[10/24/24 13:21:50] crowsnest: Host Info: Available CPU Cores: 4
[10/24/24 13:21:50] crowsnest: Host Info: Available Memory: 8245552 kB
[10/24/24 13:21:50] crowsnest: Host Info: Diskspace (avail. / total): 104G / 117G
[10/24/24 13:21:50] crowsnest: INFO: Checking Dependencies
[10/24/24 13:21:52] crowsnest: Dependency: 'crudini' found in /usr/bin/crudini.
[10/24/24 13:21:52] crowsnest: Dependency: 'find' found in /usr/bin/find.
[10/24/24 13:21:52] crowsnest: Dependency: 'xargs' found in /usr/bin/xargs.
[10/24/24 13:21:52] crowsnest: Dependency: 'ustreamer' found in bin/ustreamer/src/ustreamer.bin.
[10/24/24 13:21:53] crowsnest: Version Control: ustreamer is up to date. (v6.10)
[10/24/24 13:21:53] crowsnest: INFO: Print Configfile: '/home/tj/printer_data/config/crowsnest.conf'
[10/24/24 13:21:53] crowsnest:		[crowsnest]
[10/24/24 13:21:53] crowsnest:		log_path: ~/printer_data/logs/crowsnest.log
[10/24/24 13:21:53] crowsnest:		log_level: debug
[10/24/24 13:21:53] crowsnest:		delete_log: false
[10/24/24 13:21:53] crowsnest:		no_proxy: false
[10/24/24 13:21:53] crowsnest:
[10/24/24 13:21:53] crowsnest:		[cam 1]
[10/24/24 13:21:53] crowsnest:		mode: mjpg
[10/24/24 13:21:53] crowsnest:		port: 8080
[10/24/24 13:21:53] crowsnest:		device: /dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_809E7D1F-video-index0
[10/24/24 13:21:53] crowsnest:		resolution: 1280x720
[10/24/24 13:21:53] crowsnest:		max_fps: 60
[10/24/24 13:21:53] crowsnest:		v4l2ctl: focus_automatic_continuous=0,focus_absolute=70,brightness=100,contrast=100,saturation=100,sharpness=200,backlight_compensation=1
[10/24/24 13:21:53] crowsnest:		custom_flags: --format=yuyv
[10/24/24 13:21:53] crowsnest: INFO: Detect available Devices
[10/24/24 13:21:55] crowsnest: INFO: Found 1 total available Device(s)
[10/24/24 13:21:55] crowsnest: INFO: Found 1 available v4l2 (UVC) camera(s)
[10/24/24 13:21:55] crowsnest: /dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_809E7D1F-video-index0 -> /dev/video0
[10/24/24 13:21:55] crowsnest: Supported Formats:
[10/24/24 13:21:55] crowsnest:		[0]: 'YUYV' (YUYV 4:2:2)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 640x480
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 160x90
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 160x120
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 176x144
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 320x180
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 320x240
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 352x288
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 432x240
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 640x360
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 800x448
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 800x600
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 864x480
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 960x720
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 1024x576
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 1280x720
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 1600x896
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 1920x1080
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 2560x1472
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.500s (2.000 fps)
[10/24/24 13:21:55] crowsnest:		[1]: 'MJPG' (Motion-JPEG, compressed)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 640x480
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 160x90
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 160x120
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 176x144
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 320x180
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 320x240
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 352x288
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 432x240
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 640x360
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 800x448
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 800x600
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 864x480
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 960x720
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 1024x576
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 1280x720
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 1600x896
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest:		Size: Discrete 1920x1080
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.033s (30.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.042s (24.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.050s (20.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.067s (15.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.100s (10.000 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.133s (7.500 fps)
[10/24/24 13:21:55] crowsnest:		Interval: Discrete 0.200s (5.000 fps)
[10/24/24 13:21:55] crowsnest: Supported Controls:
[10/24/24 13:21:55] crowsnest:
[10/24/24 13:21:55] crowsnest:		User Controls
[10/24/24 13:21:55] crowsnest:
[10/24/24 13:21:55] crowsnest:		brightness 0x00980900 (int)    : min=0 max=255 step=1 default=128 value=128
[10/24/24 13:21:55] crowsnest:		contrast 0x00980901 (int)    : min=0 max=255 step=1 default=128 value=128
[10/24/24 13:21:55] crowsnest:		saturation 0x00980902 (int)    : min=0 max=255 step=1 default=128 value=128
[10/24/24 13:21:55] crowsnest:		white_balance_automatic 0x0098090c (bool)   : default=1 value=1
[10/24/24 13:21:55] crowsnest:		gain 0x00980913 (int)    : min=0 max=255 step=1 default=0 value=0
[10/24/24 13:21:55] crowsnest:		power_line_frequency 0x00980918 (menu)   : min=0 max=2 default=2 value=2 (60 Hz)
[10/24/24 13:21:55] crowsnest:		0: Disabled
[10/24/24 13:21:55] crowsnest:		1: 50 Hz
[10/24/24 13:21:55] crowsnest:		2: 60 Hz
[10/24/24 13:21:55] crowsnest:		white_balance_temperature 0x0098091a (int)    : min=2000 max=6500 step=1 default=4000 value=4000 flags=inactive
[10/24/24 13:21:55] crowsnest:		sharpness 0x0098091b (int)    : min=0 max=255 step=1 default=128 value=128
[10/24/24 13:21:55] crowsnest:		backlight_compensation 0x0098091c (int)    : min=0 max=1 step=1 default=0 value=0
[10/24/24 13:21:55] crowsnest:
[10/24/24 13:21:55] crowsnest:		Camera Controls
[10/24/24 13:21:55] crowsnest:
[10/24/24 13:21:55] crowsnest:		auto_exposure 0x009a0901 (menu)   : min=0 max=3 default=3 value=3 (Aperture Priority Mode)
[10/24/24 13:21:55] crowsnest:		1: Manual Mode
[10/24/24 13:21:55] crowsnest:		3: Aperture Priority Mode
[10/24/24 13:21:55] crowsnest:		exposure_time_absolute 0x009a0902 (int)    : min=3 max=2047 step=1 default=250 value=250 flags=inactive
[10/24/24 13:21:55] crowsnest:		exposure_dynamic_framerate 0x009a0903 (bool)   : default=0 value=1
[10/24/24 13:21:55] crowsnest:		pan_absolute 0x009a0908 (int)    : min=-36000 max=36000 step=3600 default=0 value=0
[10/24/24 13:21:55] crowsnest:		tilt_absolute 0x009a0909 (int)    : min=-36000 max=36000 step=3600 default=0 value=0
[10/24/24 13:21:55] crowsnest:		focus_absolute 0x009a090a (int)    : min=0 max=250 step=5 default=0 value=0 flags=inactive
[10/24/24 13:21:55] crowsnest:		focus_automatic_continuous 0x009a090c (bool)   : default=1 value=1
[10/24/24 13:21:55] crowsnest:		zoom_absolute 0x009a090d (int)    : min=100 max=500 step=1 default=100 value=100
[10/24/24 13:21:55] crowsnest: Try to start configured Cams / Services...
[10/24/24 13:21:55] crowsnest: INFO: Configuration of Section [cam 1] looks good. Continue ...
[10/24/24 13:21:55] crowsnest: V4L2 Control: Device: [cam 1]
[10/24/24 13:21:55] crowsnest: V4L2 Control: Options: focus_automatic_continuous=0,focus_absolute=70,brightness=100,contrast=100,saturation=100,sharpness=200,backlight_compensation=1
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl:
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: User Controls
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl:
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: brightness 0x00980900 (int)    : min=0 max=255 step=1 default=128 value=100
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: contrast 0x00980901 (int)    : min=0 max=255 step=1 default=128 value=100
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: saturation 0x00980902 (int)    : min=0 max=255 step=1 default=128 value=100
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: white_balance_automatic 0x0098090c (bool)   : default=1 value=1
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: gain 0x00980913 (int)    : min=0 max=255 step=1 default=0 value=0
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: power_line_frequency 0x00980918 (menu)   : min=0 max=2 default=2 value=2 (60 Hz)
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: 0: Disabled
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: 1: 50 Hz
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: 2: 60 Hz
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: white_balance_temperature 0x0098091a (int)    : min=2000 max=6500 step=1 default=4000 value=4000 flags=inactive
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: sharpness 0x0098091b (int)    : min=0 max=255 step=1 default=128 value=200
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: backlight_compensation 0x0098091c (int)    : min=0 max=1 step=1 default=0 value=1
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl:
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: Camera Controls
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl:
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: auto_exposure 0x009a0901 (menu)   : min=0 max=3 default=3 value=3 (Aperture Priority Mode)
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: 1: Manual Mode
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: 3: Aperture Priority Mode
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: exposure_time_absolute 0x009a0902 (int)    : min=3 max=2047 step=1 default=250 value=250 flags=inactive
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: exposure_dynamic_framerate 0x009a0903 (bool)   : default=0 value=1
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: pan_absolute 0x009a0908 (int)    : min=-36000 max=36000 step=3600 default=0 value=0
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: tilt_absolute 0x009a0909 (int)    : min=-36000 max=36000 step=3600 default=0 value=0
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: focus_absolute 0x009a090a (int)    : min=0 max=250 step=5 default=0 value=70
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: focus_automatic_continuous 0x009a090c (bool)   : default=1 value=0
[10/24/24 13:21:55] crowsnest: DEBUG: v4l2ctl: zoom_absolute 0x009a090d (int)    : min=100 max=500 step=1 default=100 value=100
[10/24/24 13:21:56] crowsnest: DEBUG: Value is now: focus_absolute=70
[10/24/24 13:21:56] crowsnest: Starting ustreamer with Device /dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_809E7D1F-video-index0 ...
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: Parameters: --host 127.0.0.1 -p 8080 -d /dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_809E7D1F-video-index0 --device-timeout=2 -m MJPEG --encoder=HW -r 1280x720 -f 60 --allow-origin=* --static /home/tj/crowsnest/ustreamer-www --format=yuyv
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.143          ] -- Starting PiKVM uStreamer 6.10 ...
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.143          ] -- HTTP: Enabling the file server: /home/tj/crowsnest/ustreamer-www
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.143          ] -- HTTP: Listening HTTP on [127.0.0.1]:8080
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.143          ] -- HTTP: Starting eventloop ...
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.143          ] -- Using V4L2 device: /dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_809E7D1F-video-index0
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.143          ] -- CAP: Using capture type: single-planar
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.143          ] -- CAP: Using input channel: 0
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.145          ] -- CAP: Using resolution: 1280x720
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.145          ] -- CAP: Using format: YUYV
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.148          ] -- CAP: Using HW FPS: 60 -> 10 (coerced)
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.148          ] -- CAP: Using IO method: MMAP
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.153          ] -- CAP: Requested 5 device buffers, got 5
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.562          ] -- CAP: Capturing started
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.562          ] -- Switching to CPU encoder: the input format is not (M)JPEG ...
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.562          ] -- Using JPEG quality: 80%
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.562          ] -- Creating pool JPEG with 4 workers ...
[10/24/24 13:21:56] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO  [30.563          ] -- Capturing ...
[10/24/24 13:21:58] crowsnest:  ... Done!

STDOUT from Ping During Failure and Eventual Recovery

...
64 bytes from 192.168.30.170: icmp_seq=1072 ttl=64 time=9.270 ms
64 bytes from 192.168.30.170: icmp_seq=1073 ttl=64 time=7.934 ms
64 bytes from 192.168.30.170: icmp_seq=1074 ttl=64 time=7.209 ms
64 bytes from 192.168.30.170: icmp_seq=1075 ttl=64 time=3.168 ms
64 bytes from 192.168.30.170: icmp_seq=1076 ttl=64 time=7.359 ms
64 bytes from 192.168.30.170: icmp_seq=1077 ttl=64 time=3.297 ms
64 bytes from 192.168.30.170: icmp_seq=1078 ttl=64 time=2.795 ms
64 bytes from 192.168.30.170: icmp_seq=1079 ttl=64 time=2.651 ms
64 bytes from 192.168.30.170: icmp_seq=1080 ttl=64 time=3.431 ms
64 bytes from 192.168.30.170: icmp_seq=1081 ttl=64 time=2.528 ms
64 bytes from 192.168.30.170: icmp_seq=1082 ttl=64 time=1.986 ms
64 bytes from 192.168.30.170: icmp_seq=1083 ttl=64 time=3.369 ms
64 bytes from 192.168.30.170: icmp_seq=1084 ttl=64 time=7.924 ms
64 bytes from 192.168.30.170: icmp_seq=1085 ttl=64 time=3.169 ms
64 bytes from 192.168.30.170: icmp_seq=1086 ttl=64 time=2.258 ms
64 bytes from 192.168.30.170: icmp_seq=1087 ttl=64 time=3.103 ms
64 bytes from 192.168.30.170: icmp_seq=1088 ttl=64 time=5.401 ms
64 bytes from 192.168.30.170: icmp_seq=1089 ttl=64 time=4.739 ms
64 bytes from 192.168.30.170: icmp_seq=1090 ttl=64 time=2.706 ms # <------ At this point I load the webcam
Request timeout for icmp_seq 1093
Request timeout for icmp_seq 1094
Request timeout for icmp_seq 1095
Request timeout for icmp_seq 1096
Request timeout for icmp_seq 1097
Request timeout for icmp_seq 1098
Request timeout for icmp_seq 1099
Request timeout for icmp_seq 1100
Request timeout for icmp_seq 1101
Request timeout for icmp_seq 1102
Request timeout for icmp_seq 1103
Request timeout for icmp_seq 1104
Request timeout for icmp_seq 1105
Request timeout for icmp_seq 1106
Request timeout for icmp_seq 1107
Request timeout for icmp_seq 1108
Request timeout for icmp_seq 1109
Request timeout for icmp_seq 1110
Request timeout for icmp_seq 1111
Request timeout for icmp_seq 1112
Request timeout for icmp_seq 1113
Request timeout for icmp_seq 1114
Request timeout for icmp_seq 1115
Request timeout for icmp_seq 1116
Request timeout for icmp_seq 1117
Request timeout for icmp_seq 1118
Request timeout for icmp_seq 1119
Request timeout for icmp_seq 1120
Request timeout for icmp_seq 1121
Request timeout for icmp_seq 1122
Request timeout for icmp_seq 1123
Request timeout for icmp_seq 1124
Request timeout for icmp_seq 1125
Request timeout for icmp_seq 1126
Request timeout for icmp_seq 1127
Request timeout for icmp_seq 1128
Request timeout for icmp_seq 1129
Request timeout for icmp_seq 1130
Request timeout for icmp_seq 1131
Request timeout for icmp_seq 1132
Request timeout for icmp_seq 1133
Request timeout for icmp_seq 1134
Request timeout for icmp_seq 1135
Request timeout for icmp_seq 1136
Request timeout for icmp_seq 1137
Request timeout for icmp_seq 1138
Request timeout for icmp_seq 1139
Request timeout for icmp_seq 1140
Request timeout for icmp_seq 1141
Request timeout for icmp_seq 1142
Request timeout for icmp_seq 1143
Request timeout for icmp_seq 1144
Request timeout for icmp_seq 1145
Request timeout for icmp_seq 1146
Request timeout for icmp_seq 1147
Request timeout for icmp_seq 1148
Request timeout for icmp_seq 1149
Request timeout for icmp_seq 1150
Request timeout for icmp_seq 1151
Request timeout for icmp_seq 1152
Request timeout for icmp_seq 1153
Request timeout for icmp_seq 1154
Request timeout for icmp_seq 1155
Request timeout for icmp_seq 1156
Request timeout for icmp_seq 1157
Request timeout for icmp_seq 1158
Request timeout for icmp_seq 1159
Request timeout for icmp_seq 1160
Request timeout for icmp_seq 1161
Request timeout for icmp_seq 1162
Request timeout for icmp_seq 1163
Request timeout for icmp_seq 1164
Request timeout for icmp_seq 1165
Request timeout for icmp_seq 1166
Request timeout for icmp_seq 1167
Request timeout for icmp_seq 1168
Request timeout for icmp_seq 1169
Request timeout for icmp_seq 1170
Request timeout for icmp_seq 1171
Request timeout for icmp_seq 1172
Request timeout for icmp_seq 1173
Request timeout for icmp_seq 1174
Request timeout for icmp_seq 1175
Request timeout for icmp_seq 1176
Request timeout for icmp_seq 1177
Request timeout for icmp_seq 1178
Request timeout for icmp_seq 1179
Request timeout for icmp_seq 1180
Request timeout for icmp_seq 1181
Request timeout for icmp_seq 1182
Request timeout for icmp_seq 1183
Request timeout for icmp_seq 1184
Request timeout for icmp_seq 1185
Request timeout for icmp_seq 1186
Request timeout for icmp_seq 1187
Request timeout for icmp_seq 1188
Request timeout for icmp_seq 1189
Request timeout for icmp_seq 1190
Request timeout for icmp_seq 1191
Request timeout for icmp_seq 1192
Request timeout for icmp_seq 1193
Request timeout for icmp_seq 1194
Request timeout for icmp_seq 1195
Request timeout for icmp_seq 1196
Request timeout for icmp_seq 1197
Request timeout for icmp_seq 1198
Request timeout for icmp_seq 1199
Request timeout for icmp_seq 1200
Request timeout for icmp_seq 1201
Request timeout for icmp_seq 1202
Request timeout for icmp_seq 1203
Request timeout for icmp_seq 1204
Request timeout for icmp_seq 1205
Request timeout for icmp_seq 1206
Request timeout for icmp_seq 1207
Request timeout for icmp_seq 1208
Request timeout for icmp_seq 1209
Request timeout for icmp_seq 1210
Request timeout for icmp_seq 1211
Request timeout for icmp_seq 1212
Request timeout for icmp_seq 1213
Request timeout for icmp_seq 1214
Request timeout for icmp_seq 1215
Request timeout for icmp_seq 1216
Request timeout for icmp_seq 1217
Request timeout for icmp_seq 1218
Request timeout for icmp_seq 1219
Request timeout for icmp_seq 1220
Request timeout for icmp_seq 1221
Request timeout for icmp_seq 1222
Request timeout for icmp_seq 1223
Request timeout for icmp_seq 1224
Request timeout for icmp_seq 1225
Request timeout for icmp_seq 1226
Request timeout for icmp_seq 1227
Request timeout for icmp_seq 1228
Request timeout for icmp_seq 1229
ping: sendto: No route to host
Request timeout for icmp_seq 1230
ping: sendto: Host is down
Request timeout for icmp_seq 1231
ping: sendto: Host is down
Request timeout for icmp_seq 1232
ping: sendto: Host is down
Request timeout for icmp_seq 1233
ping: sendto: Host is down
Request timeout for icmp_seq 1234
ping: sendto: Host is down
Request timeout for icmp_seq 1235
ping: sendto: Host is down
Request timeout for icmp_seq 1236
ping: sendto: Host is down
Request timeout for icmp_seq 1237
ping: sendto: Host is down
Request timeout for icmp_seq 1238
ping: sendto: Host is down
Request timeout for icmp_seq 1239
ping: sendto: Host is down
Request timeout for icmp_seq 1240
ping: sendto: Host is down
Request timeout for icmp_seq 1241
ping: sendto: Host is down
Request timeout for icmp_seq 1242
ping: sendto: Host is down
Request timeout for icmp_seq 1243
ping: sendto: Host is down
Request timeout for icmp_seq 1244
ping: sendto: Host is down
Request timeout for icmp_seq 1245
Request timeout for icmp_seq 1246
64 bytes from 192.168.30.170: icmp_seq=1226 ttl=64 time=21903.111 ms # <- Approximately 150 pings later, it recovers with massive latency
64 bytes from 192.168.30.170: icmp_seq=1227 ttl=64 time=20899.944 ms
64 bytes from 192.168.30.170: icmp_seq=1228 ttl=64 time=19898.018 ms
64 bytes from 192.168.30.170: icmp_seq=1229 ttl=64 time=18894.773 ms
64 bytes from 192.168.30.170: icmp_seq=1230 ttl=64 time=17890.825 ms
64 bytes from 192.168.30.170: icmp_seq=1247 ttl=64 time=840.429 ms
64 bytes from 192.168.30.170: icmp_seq=1248 ttl=64 time=3.870 ms
64 bytes from 192.168.30.170: icmp_seq=1249 ttl=64 time=3.149 ms
64 bytes from 192.168.30.170: icmp_seq=1250 ttl=64 time=2.947 ms
64 bytes from 192.168.30.170: icmp_seq=1251 ttl=64 time=5.378 ms
64 bytes from 192.168.30.170: icmp_seq=1252 ttl=64 time=6.148 ms
@zimmertr zimmertr added the bug Something isn't working label Oct 24, 2024
@mryel00
Copy link
Member

mryel00 commented Oct 24, 2024

That's some interesting bug. But as you already noticed, Crowsnest is basically doing nothing, that could create this problem. I recommend to get Nginx, Mainsail and Crowsnest out of the equation and test it only with ustreamer. So run the following command:

/home/tj/crowsnest/bin/ustreamer/ustreamer --host 0.0.0.0 -p 8080 -d /dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_809E7D1F-video-index0 --device-timeout=2 --encoder=HW -r 1280x720 -f 10 --allow-origin=* --format=yuyv

Then connect to http://<ip.of.your.pi>:8080/?action=stream and see if you can reproduce the issue like that.

@zimmertr
Copy link
Author

Hm, yeah. You're totally right. It does seem like an upstream issue with ustreamer. I can replicate the bug by launching it directly instead of through Crownest.

Here's another screen recording: https://www.youtube.com/watch?v=3VNb6MwUPKQ

@mryel00
Copy link
Member

mryel00 commented Oct 24, 2024

Can you try iperf? Just to see, if it's really ustreamer or just some problem at higher bandwidth usage

@zimmertr
Copy link
Author

zimmertr commented Oct 24, 2024

I cannot bring the network interface down withiperf, so it does seem unique to ustreamer. Here's another screen recording: https://www.youtube.com/watch?v=lghwhJxn3MU

I'll try and test against the newest version as you suggested on Discord and report back after lunch.

What's also interesting, is that I am not observing the same behavior when I run curl locally against the endpoint. Instead of using my browser. Nothing breaks. However, if I run curl against the endpoint from outside the Pi, for example from my Macbook, it breaks networking. So this bug only occurs when the traffic originates outside of localhost?

curl hektor.sol.milkyway:8080/stream --output /dev/null

@zimmertr
Copy link
Author

Unfortunately it occurs with version 6.16 as well. Here is another screen recording showing that as well as the note I made above about it only failing when the traffic originates from outside of the device itself. https://www.youtube.com/watch?v=QSUJxR5HU0g

I'll go ahead and file a different issue with ustreamer. You can close this one if you want. Or leave it open to track the status of the other. In either case, I'll comment a link to it here once I've opened it.

@zimmertr
Copy link
Author

Issue posted: pikvm/ustreamer#293

@rackrick
Copy link
Member

rackrick commented Oct 25, 2024

What's also interesting, is that I am not observing the same behavior when I run curl locally against the endpoint.

Do you use localhost or 127.0.0.1 for that? These addresses point to loopback apdapter which is a virtual interface.
So you don't have the same way the packages go as through your external wifi card.

@zimmertr
Copy link
Author

I used the FQDN which resolves to the 192.168.30.170 IP Address. I just repeated the experiment and I can see that ustreamer logs the connection attempt as originating from that IP too. But I wouldn't be surprised if Linux is somehow redirecting it to go through the loopback interface.

Screenshot 2024-10-25 at 9 13 49 AM

@rackrick
Copy link
Member

Thanks for your information.
Well I wouldn't surprised either if Linux does something like that.

Anyways did you check the syslog for entries of the usb device disconnecting or something related?

@zimmertr
Copy link
Author

zimmertr commented Oct 25, 2024

No, but I did notice that dmesg was continuously dumping the following error message:

brcmfmac: brcmf_set_channel: set chanspec fail, reason -52

Screenshot 2024-10-25 at 11 14 55 AM

This is interesting, because brcmfmac is the driver used by the Raspberry Pi's internal wlan0 NIC

Screenshot 2024-10-25 at 11 15 31 AM

I set up a second NetworkManager connection profile for wlan0 and enabled it and rebooted. Then those errors stopped getting logged.

Screenshot 2024-10-25 at 11 17 12 AM

Afterwards, I went ahead and started up a ustreamer server again and connected to it over 192.168.30.170 via wlan1 as usual and it did not crash!

However, using bmon, I can see that wlan0 is the one transferring data continuously, NOT wlan1. Despite the fact that I am hitting the ustreamer server over wlan1's IP Address.

Screenshot 2024-10-25 at 11 29 36 AM

Like, I'm happy things are working right now. But something is still wrong. ustreamer/crowsnest is only seemingly functional when it can use wlan0. And even when I connect to the Raspberry Pi over wlan1, I still see that wlan0 is being used to transfer data. This is problematic because the wireless reception of that NIC is poor, and I can only get 7-11FPS over the 720p stream. Which is what originally led to me purchasing the USB wireless adapter to begin with.

Screenshot 2024-10-25 at 11 20 26 AM

@rackrick
Copy link
Member

Right above the brcmfmac spam you can see your wifi stick disconnecting and reconnecting to the usb bus. Also you can see a tx timeout.
When did you check dmesg? After a crash?

@zimmertr
Copy link
Author

zimmertr commented Oct 25, 2024

I think that dmesg output was from immediately after startup. I don't see the log indicating a disconnection nor a tx timeout. I do see it mention limiting tx power. I presumed that had to do with my Ubiquiti AP's Power settings but I could be wrong.

It was previously difficult to get crash information. The crash only occurs if wlan0 is down. In these cases, once wlan1 also goes down the device is no longer accessible on the network. Additionally, the Raspberry Pi 5 does not have an HDMI output, it just has a DSI header.

However, I've just attached a USB keyboard to the Pi, set a password for my user, turned off the KlipperScreen SystemD unit, and now I can access a TTY and log in without networking.

Give me a while and I will trigger a crash and record of a video of the screen to gather more data.

@zimmertr
Copy link
Author

zimmertr commented Oct 26, 2024

Okay, I've gotten to the bottom of this. Mostly.

I recorded a long video of the DSI screen and then when I started to watch it over I realized none of this was necessary at all I could simply use journalctl --dmesg -b -1 to get the last boot's log message. Then I remembered that not even that was necessary, because the networking starts to work again after ~20 minutes. So, with wlan0 down, I SSHd into the device, started a ustreamer log, hit it with my browser and caused a crash. Then I waited until it started to respond to pings again and pulled a log from the device.

In this, I saw various messages repeating like the following examples:

[ 129.573465] usb usb1-port1: over-current change #1
[ 129.708999] usb 1-1: USB disconnect, device number 2
[ 131.309504] usb 4-1: reset SuperSpeed USB device number 10 using xhci-hcd

This led to me trying a few things.

I removed the Wireless adapter from the extension cable it was using in case the cable was causing a power/bandwidth problem. That didn't fix anything. Then I removed one unnecessary USB device from my Pi (A Noctua fan) to see if that might help with reducing power consumption. That's didn't fix anything either.

It was at that point I was starting to worry that I messed something up between when I wired my printer's power supply to a buck converter and then eventually the Raspberry Pi. And I was over/under volting the device. I'm a software engineer, but do not have any real experience with electrical engineering.

On a whim, I decided to change to a new USB Wireless Adapter that just came in the mail. And the problem went away entirely. To rule out any power issues, I ran stress --cpu 4 and maxxed out all four cores on the device for a minute or two until lm-sensors started showing the temps creeping towards 80°C when I backed it off. None of the USB devices or anything ever disconnected, and the Pi never turned off.

So I'm inclined to believe it's an issue with the mt7921u driver. So, anyone landing here from Google in the future. Don't use anything with that driver. Including, in this case, a NETGEAR Nighthawk (A8000) - AXE3000. The random, and significantly cheaper, Nineplus adapter works without issue. And is actually performing BETTER than the NETGEAR one was.


A huge thank you @mryel00 and @rackrick for patiently trying to help me troubleshoot this over the past day. Now I can finally finish up this guide and put the printer back together :)

https://github.com/zimmertr/Ender-6-Klipper-Guide/blob/a749396826e695c7e971c4320f91e869832bcc71/README.md

@zimmertr
Copy link
Author

zimmertr commented Oct 26, 2024

Still only getting like 7-11fps at 720p though. So I still need to figure that out. Not sure why it's been capped at 10 by ustreamer.

...
[10/25/24 17:37:17] crowsnest: [cam 1]
[10/25/24 17:37:17] crowsnest: mode: mjpg
[10/25/24 17:37:17] crowsnest: port: 8080
[10/25/24 17:37:17] crowsnest: device: /dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_809E7D1F-video-index0
[10/25/24 17:37:17] crowsnest: resolution: 1280x720
[10/25/24 17:37:17] crowsnest: max_fps: 30
...
[10/25/24 17:37:19] crowsnest: Size: Discrete 1280x720
[10/25/24 17:37:19] crowsnest: Interval: Discrete 0.033s (30.000 fps)
[10/25/24 17:37:19] crowsnest: Interval: Discrete 0.042s (24.000 fps)
[10/25/24 17:37:19] crowsnest: Interval: Discrete 0.050s (20.000 fps)
[10/25/24 17:37:19] crowsnest: Interval: Discrete 0.067s (15.000 fps)
[10/25/24 17:37:19] crowsnest: Interval: Discrete 0.100s (10.000 fps)
[10/25/24 17:37:19] crowsnest: Interval: Discrete 0.133s (7.500 fps)
[10/25/24 17:37:19] crowsnest: Interval: Discrete 0.200s (5.000 fps)
....
[10/25/24 17:37:20] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO [55.348 ] -- CAP: Using HW FPS: 30 -> 10 (coerced)

@zimmertr
Copy link
Author

zimmertr commented Oct 26, 2024

Okay, managed to get up to about 15fps for my Logitech C920 with this configuration:

[crowsnest]
log_path: ~/klipper_logs/crowsnest.log
log_level: verbose
delete_log: false
no_proxy: false

[cam 1]
mode: ustreamer
port: 8080
device: /dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_809E7D1F-video-index0
resolution: 800x600
max_fps: 30
v4l2ctl: focus_automatic_continuous=0,focus_absolute=70,brightness=100,contrast=100,saturation=100,sharpness=200,backlight_compensation=1
custom_flags: --format=yuyv

I gather from the Discord forums that the recommended solution to further improve FPS would be to use WebRTC which requires camera-streamer. However, these issues indicate to me that's not supported on the Raspberry Pi 5 do to a lack of hardware encoders.

So I suspect this is the best possible outcome at this time. I don't exactly understand why I cannot reach higher resolutions/fps though as this does not appear to be a bandwidth constraint. bmon currently shows a TX of about 360kb/s while streaming to Mainsail at 800x600@15fps. But iperf shows my Macbook Pro can talk to the Pi at over 5MiB/s.

@meteyou
Copy link
Member

meteyou commented Oct 26, 2024

Higher resolutions and MJPEG (normal ustreamer format) needs a lot more bandwidth to transfer the images (its just multiple jpeg per second. nothing advanced). WebRTC is much more bandwidth optimiced then MJPEG.

@mryel00
Copy link
Member

mryel00 commented Oct 26, 2024

About your fps. You should not use --format=YUYV unless you really need it to run the cam. It's possible that your cam itself doesn't produce more pictures with YUYV. YUYV is not compressed therefore it uses more USB bandwidth. With 800x600 you only get a maximum of 24 fps for your cam. At 1280x720 you only get up to 10 fps with YUYV. Those informations can also be found in the log.
Furthermore you should use the MJPG service in Mainsail, not the Adaptive MJPG.

@zimmertr
Copy link
Author

zimmertr commented Oct 26, 2024

If I remove the --format=yuyv flag, FPS drops to 1-3 at both 800x600 and 1280x720. So I'm afraid this is the best I can do with a Raspberry Pi 5 and a Logitech C920.

@zimmertr
Copy link
Author

Also, I'm starting to think my original issue was not necessarily driver related so much as power related.

Most Raspberry Pis provide enough current to USB peripherals to power most USB devices, including keyboards, mice, and adapters. However, some devices require additional current, including modems, external disks, and high-powered antenna. To connect a USB device with power requirements that exceed the values specified in the table above, connect it using an externally-powered USB hub.
The power requirements of the Raspberry Pi increase as you make use of the various interfaces on the Raspberry Pi. Combined, the GPIO pins can draw 50mA safely; each pin can individually draw up to 16mA. The HDMI port uses 50mA. The Camera Module requires 250mA. USB keyboards and mice can take as little as 100mA or as much as 1000mA. Check the power rating of the devices you plan to connect to the Raspberry Pi and purchase a power supply accordingly. If you’re not sure, use an externally-powered USB hub.

https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#typical-power-requirements

@zimmertr
Copy link
Author

zimmertr commented Oct 27, 2024

Okay, so I've come to learn that the Raspberry Pi 5 requests the input current from the USB-C port when it boots. If no response is received, or it is less than 3A, the USB-C devices are limited to consuming 600mA. However, if it's greater than 3A, they can use up to 1.6A. Since my device is powered by the GPIO pins I was being limited.

This can be fixed by turning this feature off and manually defining the input amperage.

sudo raspi-config
# 4 Performance Options
# P4 USB Current 
# Would you like the USB current limit to be disabled? -> Yes

sudo rpi-eeprom-config --edit
# Add PSU_MAX_CURRENT=5000
# Save and exit the file: Ctrl+X -> Return 

sudo reboot

After this, the NETGEAR wireless USB adapter works just fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants