How to use Intel NCS2 for inference?

[shrinand1996]

I would like to try the Intel Neural Compute stick2 for inference.
Can i run inference on it without any modifications in the OPENVINO inference?

And can i run it on a raspberry pi 4?

Since this library is under openvinotoolkit, i thought it will be optimised for the NCS2.

Before ordering the NCS2, i thought ill ask first if it will work here or not. And if yes, how will it work?

[ashwinvaidya17]

@shrinand1996 you can have a look at all the OpenVINO supported devices here https://docs.openvino.ai/latest/openvino_docs_OV_UG_supported_plugins_Supported_Devices.html. It says that Raspberry Pi 4 has a community plugin but it needs to be compiled from scratch. I don't have a raspi so I cannot confirm if the compiled distribution works with anomalib but I can let you know if it works on NCS2 next week. If it does you can always use NCS2 on raspi.

[shrinand1996]

But do we need the arm plugin to run NCS2? Dont we need the VPU plugin to use NCS2?

[ashwinvaidya17]

No VPU plugin should we fine. But some models after export are quite large in size. I am not sure if they will fit the NCS2 memory. I'll have to test them on it. Otherwise it might not meet your need.

[ashwinvaidya17]

@shrinand1996 I tested with padim, patchcore and stfpm and only padim and stfpm work on my end. I get RuntimeError: Failed to allocate graph: NC_OUT_OF_MEMORY when I try to use patchcore. While patchcore isn't a big model you might get similar issues in a bigger model. But you can always look into quantization. In case you do end up getting NCS2 you need to edit this line

anomalib/anomalib/deploy/inferencers/openvino_inferencer.py

Line 83 in dacf3f4

executable_network = ie_core.load_network(network=network, device_name="CPU")

and replace CPU with MYRIAD