Basic motion support

[cholcombe973]

I really like how little resources this program uses. What are people using for motion tracking support? I'm thinking setting up a little pipeline where moonfire saves files, some motion tracker detects if there was motion and either saves a clip or deletes the files to save space. Anyone have something like that setup already?

[scottlamb]

I'd call this Moonfire NVR's biggest missing feature. There's no support for on-NVR analytics, and no UI support for analytics period.

I'm using my cameras' (mediocre) built-in analytics with this program, and a (buggy, in progress) UI branch that lets me restrict the view to just times when there's motion.

I certainly want to improve this...

[scottlamb]

Oh, I forgot to mention: there's also this program which can connect to Moonfire NVR and run a TensorFlow Lite model on all the video there. But it's not production-ready in a bunch of ways: it doesn't send the results back to Moonfire NVR, it just runs once and returns (rather than following the live view), and it just runs the (low-resolution) model against the whole frame rather than on a zoomed-in model. frigate's approach of doing motion detection to find the right zoomed-in regions to run the model on gives better results as well as requiring far less inference power. I want to match that.

[tim-seoss]

A few thoughts - OpenCV includes motion detection, and includes OpenCL support for GPU offloading, it might be a cheap way of adding motion detection. Unfortunately it looks like the available Raspberry Pi 0/1/2/3 OpenCL hardware implementation only supports the OpenCL "embedded profile", and OpenCV requires(?) the "full profile", so this might be problematic (I haven't looked in detail yet). For other platforms (including MALI T-600 and later based ARM SoCs) full OpenCL is available and should be usable.

The Raspberry Pi 0/1/2/3 hardware h.264 decoder could be used (via Gstreamer or otherwise) to extract decompressed frames with little CPU use.

OpenCL also has a "Motion Estimation Extension" which is implemented by Intel hardware.

I realise that adding these libraries would go against the goal of reducing non-rust dependencies, but may still be worth considering as reasonably cross-platform methods for adding motion detection with relatively little development. Individual components could be reimplemented in Rust over time where they present possible security issues.

Another approach for motion detection using the Pi's VC4 video encoder hardware which might be a useful reference is here: https://github.com/dickontoo/omxmotion/

A third, and more generally cross-platform approach might be to use OpenGL (ES), but I haven't looked into this.

[scottlamb]

I realise that adding these libraries would go against the goal of reducing non-rust dependencies

That's not a deal-breaker. I definitely still want this feature. My thought has long been that analytics can be a separate process/binary that communicates with the core server via the API. That would allow folks to implement analytics in other languages (Python is super popular for ML and computer vision AFAICT), iterate faster (without even shutting down the main server), use multiple machines if necessary (as analytics are far more resource-intensive than anything else Moonfire does or would do), plug in different approaches to analytics, etc. And it'd let the core server be zero-deps (#160) so people can get started with Moonfire quickly, without restricting what's possible for analytics. It'd be nice to make setting up analytics easy too, but if it has to be a bit more involved (to use a variety of hardware and C/C++ libraries that may impractical to statically link into one deploy-anywhere binary), so be it.

Another possibility would be cargo feature flags, but all else being equal, the separate binary approach appeals to me more.

The big blocker has been my limited time and lack of experience with ML/vision. If you're interested in working on this, I'd be really happy to talk with you more about it.