Skip to content

Latest commit

 

History

History
179 lines (151 loc) · 6.67 KB

README.md

File metadata and controls

179 lines (151 loc) · 6.67 KB
Raw

FlexInfer

A flexible Python front-end inference SDK.

Features

  • Flexible

    FlexInfer has a Python front-end, which makes it easy to build a computer vision product prototype.

  • Efficient

    Most of time consuming part of FlexInfer is powered by C++ or CUDA, so FlexInfer is also efficient. If you are really hungry for efficiency and don't mind the trouble of C++, you can refer to CheetahInfer.

License

This project is released under Apache 2.0 license.

Installation

Requirements

  • Linux
  • Python 3.6 or higher
  • TensorRT 7.1.3.4 or higher
  • PyTorch 1.4.0 or higher
  • CUDA 10.2 or higher
  • volksdep 3.2.0 or higher

We have tested the following versions of OS and softwares:

  • OS: Ubuntu 16.04.6 LTS
  • Python 3.6.9
  • TensorRT 7.1.3.4
  • PyTorch 1.6.0
  • CUDA: 10.2
  • volksdep: 3.2.0

Install FlexInfer

  1. If your platform is x86 or x64, you can create a conda virtual environment and activate it.
conda create -n flexinfer python=3.6.9 -y
conda activate flexinfer

  1. Install volksdep following the official instructions

  2. Setup

pip install "git+https://github.com/Media-Smart/flexinfer.git"

Usage

We provide some examples for different tasks.

Throughput benchmark

  • Device: Jetson AGX Xavier
  • CUDA: 10.2
Tasks framework version input shape data type throughput(FPS) latency(ms)
Classification (ResNet18) PyTorch 1.5.0 (1, 3, 224, 224) FP16 172 6.01
TensorRT 7.1.0.16 (1, 3, 224, 224) FP16 754 1.8
Segmentation（U-Net） PyTorch 1.5.0 (1, 3, 513, 513) FP16 15 63.27
tensorrt 7.1.0.16 (1, 3, 513, 513) FP16 29 34.03
Object Detection RetinaNet-R50 PyTorch 1.5.0 (1, 3, 768, 1280) FP16 8 118.79
TensorRT 7.1.0.16 (1, 3, 768, 1280) FP16 15 68.10
TinaFace-R50-FPN-BN PyTorch 1.5.0 (1, 3, 768, 1280) FP16 3 273.60
TensorRT 7.1.0.16 (1, 3, 768, 1280) FP16 6 159.70
Scene Text Recognition (ResNet-CTC) PyTorch 1.5.0 (1, 1, 32, 100) FP16 113 10.75
TensorRT 7.1.0.16 (1, 1, 32, 100) FP16 308 3.55

Media-Smart toolboxes

We provide some toolboxes of different tasks for training, testing and deploying.

Contact

This repository is currently maintained by Yuxin Zou (@Yuxin Zou), Jun Sun(@ChaseMonsterAway), Hongxiang Cai (@hxcai) and Yichao Xiong (@mileistone).