Jie
4 years ago
69 changed files with 1415 additions and 745 deletions
@ -1,261 +0,0 @@
@@ -1,261 +0,0 @@
|
||||
# RKNNToolkit2 OPs Support |
||||
|
||||
Based on RKNN Toolkit2 Version 1.1.0 |
||||
|
||||
## Explanation of terms: |
||||
|
||||
Remarks: |
||||
|
||||
Operators' specifications must meet the remarks' requirements. |
||||
|
||||
Broadcast rule: |
||||
|
||||
- per-layer： |
||||
|
||||
shape(A) = (2, 3, 4, 5), shape(B) = (,), i.e. B is a scalar ==> shape(result) = (2, 3, 4, 5) |
||||
|
||||
- per-channel: |
||||
|
||||
shape(A) = (2, 3, 4, 5), shape(B) = (3,), ==> shape(result) = (2, 3, 4, 5) |
||||
|
||||
shape(A) = (2, 3, 4, 5), shape(B) = (1,3,1,1), ==> shape(result) = (2, 3, 4, 5) |
||||
|
||||
- per-element: |
||||
|
||||
shape(A) = (2, 3, 4, 5), shape(B) = (2,3,4,5) ==> shape(result) = (2, 3, 4, 5) |
||||
|
||||
- other: |
||||
|
||||
shape(A) = (2, 3, 4, 5), shape(B) = (5,) ==> shape(result) = (2, 3, 4, 5) |
||||
|
||||
|
||||
|
||||
## Caffe OPs supported by RKNN Toolkit2 |
||||
|
||||
Caffe protocols RKNN Toolkit2 uses only based on the officially modified protocol of berkeley. |
||||
The protocol based on the official revision of berkeley comes from [berkeley caffe](https://github.com/BVLC/caffe/tree/master/src/caffe/proto 'Berkeley Caffe'), commit hash is 21d0608. On this basis RKNN Toolkit2 have added some OPs. |
||||
Based on this protocol, the list of Caffe OPs supported by RKNN Toolkit2 Version 1.1.0 is as follows: |
||||
|
||||
| **Operators** | **Remarks** | |
||||
| -------------------- | ----------- | |
||||
| BatchNorm |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| bn (BatchNorm + Scale) |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br /> according to https://github.com/TimoSaemann/caffe-segnet-cudnn5| |
||||
| BNLL || |
||||
| Concat |axis: 1,2,3| |
||||
| Convolution |channel: [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: [1, 7]<br />kernels: [1, 8184]<br />pad left/right/top/bottom: [0, 15]<br />group: 1, channel / N <br /><br />| |
||||
| ConvolutionDepthwise|channel:[1, 8192]<br />kernel height/width: [1, 8]<br />stride height/width: [1, 7]<br />kernels: 1<br />pad left/right/top/bottom: [0, 15]| |
||||
| Crop || |
||||
| Deconvolution |channel: [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: 2, 4, 8<br />kernels: [1, 8192]<br />pad left/right/top/bottom: [0, 15]| |
||||
| Dropout || |
||||
| Eltwise |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />support broadcast rule: per-layer/channel/element| |
||||
| Flatten || |
||||
| InnerProduct |channel: [1, 8192]| |
||||
| LRN || |
||||
| Normalize || |
||||
| Permute || |
||||
| Power || |
||||
| Pooling | **AveragePool**:<br />channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]<br /><br />**GlobalAveragePool**:<br />channel: [1, 8192]<br />kernel height/width: [1, 128]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7] <br /><br />**MaxPool/GlobalMaxPool**:<br />channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]<br /><br />**MaxPool**: <br />auto_pad only support NOTSET，ceil_mode only support 0，unsupport dilations | |
||||
| PRelu |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />slope: per-layer/channel| |
||||
| Proposal |batch: 1| |
||||
| Reduction |output dims <= 4| |
||||
| Relu |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| Relu6 |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| Reorg || |
||||
| Reshape || |
||||
| Resize |bilinear; nearest| |
||||
| Reverse || |
||||
| ROIPooling |according to https://github.com/twmht/caffe-pva-faster-rcnn| |
||||
| Scale |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| Sigmoid |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| Slice || |
||||
| Softmax || |
||||
| Split || |
||||
| TanH |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| Tile || |
||||
| Transpose || |
||||
| Upsample |according to https://github.com/SeanQ88/caffe_upsample and https://github.com/TimoSaemann/caffe-segnet-cudnn5| |
||||
|
||||
## ONNX OPs supported by RKNN Toolkit2 |
||||
|
||||
The ONNX version supported by RKNN Toolkit2 is 1.1.0. According to [ONNX official instructions](https://github.com/microsoft/onnxruntime/blob/master/docs/Versioning.md 'ONNX Version Description'), the corresponding ONNX opset version is 12. |
||||
The list of ONNX OPs supported by RKNN Toolkit2 Version 1.1.0 is as follows: |
||||
|
||||
| **Operators** | **Remarks** | |
||||
| ----------------------------- | ----------- | |
||||
| Add |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />support broadcast rule: per-layer/channel/element| |
||||
| AveragePool |channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]| |
||||
| ArgMin || |
||||
| ArgMax || |
||||
| ReduceMin || |
||||
| ReduceMax || |
||||
| BatchNormalization |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| Clip |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| Concat |axis: 1,2,3| |
||||
| Conv |channel: [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: [1, 7]<br />kernels: [1, 8184]<br />pad left/right/top/bottom: [0, 15]<br />dilation: [1, 31]<br />group: 1, channel / N| |
||||
| ConvTranspose |channel: [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: 2, 4, 8<br />kernels: [1, 8192]<br />pad left/right/top/bottom: [0, 15]<br />dilation: [1, 31]<br />group: 1, channel / N| |
||||
| DepthToSpace || |
||||
| Div |support broadcast rule: per-element/other| |
||||
| Flatten || |
||||
| Gather || |
||||
| Gemm |channel: [1, 8192]<br /> One input should be Const| |
||||
| GlobalAveragePool |channel: [1, 8192]<br />kernel height/width: [1, 128]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]| |
||||
| GlobalMaxPool |channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]| |
||||
| Greater |support broadcast rule: per-element/other| |
||||
| HardSigmoid || |
||||
| LeakyRelu |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| Less |support broadcast rule: per-element/other| |
||||
| LpNormalization || |
||||
| LRN || |
||||
| MatMul |channel: [1, 8192]<br />dims: 2| |
||||
| Max |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br /> dims=4| |
||||
| MaxPool |channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]<br />auto_pad only support NOTSET，ceil_mode only support 0，unsupport dilations| |
||||
| MaxRoiPool || |
||||
| MaxUnpool |unsupport pad| |
||||
| Mul |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />support broadcast rule: per-layer/channel/element| |
||||
| Pad |pad value should >= 0; pad dims must be 2 when mode is reflect or edge| |
||||
| Pow || |
||||
| PRelu |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />slope support broadcast rule:: per-layer/channel| |
||||
| ReduceMean |output dims <= 4| |
||||
| ReduceSum |output dims <= 4| |
||||
| Relu |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| Reshape || |
||||
| Resize |bilinear（not support tf_crop_and_resize）; nearest2d| |
||||
| ReverseSequence || |
||||
| RoiAlign |pool type: average| |
||||
| Slice || |
||||
| Softmax || |
||||
| SpaceToDetph || |
||||
| Split || |
||||
| Squeeze || |
||||
| Tanh |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| Tile || |
||||
| Transpose || |
||||
| Upsample (resize) || |
||||
|
||||
## Pytorch OPs supported by RKNN Toolkit2 |
||||
|
||||
The Pytorch version supported by RKNN Toolkit2 is 1.6.0, models generated by other versions may not support. |
||||
The list of Pytorch OPs supported by RKNN Toolkit2 Version 1.1.0 is as follows: |
||||
|
||||
| **Operators** | **Remarks** | |
||||
| ------------------------- | ----------- | |
||||
| aten::_convolution |channel: [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: [1, 7]<br />kernels: [1, 8184]<br />pad left/right/top/bottom: [0, 15]<br />dilation: [1, 31]<br />group: 1, channel / N| |
||||
| aten::add |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />support broadcast rule: per-layer/channel/element | |
||||
| aten::avg_pool2d |channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]| |
||||
| aten::batch_norm |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| aten::cat |axis: 1,2,3| |
||||
| aten::dropout || |
||||
| aten::flatten || |
||||
| aten::leaky_relu |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| aten::matmul |channel: [1, 8192]<br />dims: 2| |
||||
| aten::max_pool2d |channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]<br />auto_pad only support NOTSET，ceil_mode only support 0，unsupport dilations| |
||||
| aten::mean |output dims <= 4| |
||||
| aten::mul |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />support broadcast rule: per-layer/channel/element | |
||||
| aten::pad |pad value should >= 0; pad dims must be 2 when mode is reflect or edge| |
||||
| aten::permute || |
||||
| aten::prelu |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />slope support broadcast rule:: per-layer/channel| |
||||
| aten::relu |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| aten::reshape || |
||||
| aten::sigmoid || |
||||
| aten::slice || |
||||
| aten::softmax || |
||||
| aten::squeeze || |
||||
| aten::sum |output dims <= 4| |
||||
| aten::upsample_bilinear2d || |
||||
| aten::upsample_nearest2d || |
||||
| aten::view || |
||||
|
||||
## TensorFlow OPs supported by RKNN Toolkit2 |
||||
|
||||
The pb files (contain OPs belows) generated by TensorFlow version 1.12 - 1.15 are supported by RKNN Toolkit2. For more information on TensorFlow version compatibility, please refer to [tensorflow official instructions on OP version](https://www.tensorflow.org/guide/versions 'Tensorflow official instructions on OP version') . |
||||
The list of TensorFlow OPs supported by RKNN Toolkit2 is as follows: |
||||
|
||||
| **Operators** | **Remarks** | |
||||
| ---------------------------------- | ----------- | |
||||
| AvgPool |channel: [1, 8192]<br>kernel height/width: [1, 7]<br>stride height/width: [1, 8]<br>pad left/right/top/bottom: [0, 7]| |
||||
| Concat |axis: 1,2,3| |
||||
| Conv2D |channel: [1, 8192]<br>kernel height/width: [1, 31]<br />stride height/width: [1, 7]<br />kernels: [1, 8184]<br />pad left/right/top/bottom: [0, 15]<br />dilation: [1, 31]<br />group: 1, channel / N| |
||||
| DepthToSpace || |
||||
| DepthwiseConv2d |channel:[1, 8192]<br />kernel height/width: [1, 8]<br />stride height/width: [1, 7]<br />kernels: 1<br />pad left/right/top/bottom: [0, 15]| |
||||
| Div |support broadcast rule: per-element/other| |
||||
| Dropout || |
||||
| Flatten || |
||||
| LeakyRelu |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| Less |support broadcast rule: per-element/other| |
||||
| LRN || |
||||
| MatMul |channel: [1, 8192]<br />dims: 2| |
||||
| MaxPool |channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]<br />auto_pad only support NOTSET，ceil_mode only support 0，unsupport dilations| |
||||
| Mean |output dims <= 4| |
||||
| Pad |pad value should >= 0; pad dims must be 2 when mode is reflect or edge| |
||||
| Relu |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| Reshape || |
||||
| ResizeBilinear || |
||||
| ResizeNearestNeighbor || |
||||
| Sigmoid || |
||||
| Slice || |
||||
| Softmax || |
||||
| Softplus || |
||||
| SpaceToDepth || |
||||
| Split || |
||||
| Squeeze || |
||||
| StridedSlice || |
||||
| Tanh |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
| Transpose || |
||||
|
||||
## TensorFlow Lite OPs supported by RKNN Toolkit2 |
||||
RKNN Toolkit2 uses the TF Lite schema commits in link: |
||||
https://github.com/tensorflow/tensorflow/commits/master/tensorflow/lite/schema/schema.fbs |
||||
Commit hash: 0c4f5dfea4ceb3d7c0b46fc04828420a344f7598. |
||||
Because TF Lite schema may not compatible with each other, TF Lite models with older or newer schema may not be loaded successfully. |
||||
The list of TensorFlow Lite OPs supported by RKNN Toolkit2 is as follows: |
||||
|
||||
| **Operators** | **Remarks** | |
||||
|---| ----------- | |
||||
|ADD|channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />support broadcast rule: per-layer/channel/element | |
||||
|AVERAGE_POOL_2D|channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]| |
||||
|CONCATENATION|axis: 1,2,3| |
||||
|CONV_2D_TRANSPOSE|channel: [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: 2, 4, 8<br />kernels: [1, 8192]<br />pad left/right/top/bottom: [0, 15]<br />dilation: [1, 31]<br />group: 1, channel / N| |
||||
|CONV_2D|channel: [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: [1, 7]<br />kernels: [1, 8184]<br />pad left/right/top/bottom: [0, 15]<br />dilation: [1, 31]<br />group: 1, channel / N| |
||||
|DEPTH_TO_SPACE|| |
||||
|DEPTHWISE_CONV_2D|channel:[1, 8192]<br />kernel height/width: [1, 8]<br />stride height/width: [1, 7]<br />kernels: 1<br />pad left/right/top/bottom: [0, 15]| |
||||
|DIV|support broadcast rule: per-element/other | |
||||
|FULLY_CONNECTED|channel: [1, 8192]<br />| |
||||
|GREATER|support broadcast rule: per-element/other| |
||||
|L2_NORMALIZATION|dims: 4| |
||||
|LEAKY_RELU|channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
|LESS|support broadcast rule: per-element/other| |
||||
|LOCAL_RESPONSE_NORMALIZATION|| |
||||
|MAX_POOL_2D|channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]<br />auto_pad only support NOTSET，ceil_mode only support 0，unsupport dilations| |
||||
|MUL|channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />support broadcast rule: per-layer/channel/element | |
||||
|PAD|pad value should >= 0; pad dims must be 2 when mode is reflect or edge| |
||||
|PRELU|channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />slope support broadcast rule:: per-layer/channel| |
||||
|RELU|channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
|RESHAPE|| |
||||
|RESIZE_BILINEAR|| |
||||
|RESIZE_NEAREST_NEIGHBOR|| |
||||
|SOFTMAX|| |
||||
|SPACE_TO_DEPTH|| |
||||
|SPLIT/SPLIT_V|| |
||||
|SQUEEZE|| |
||||
|STRIDED_SLICE|| |
||||
|TANH|channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
|TRANSPOSE|| |
||||
|
||||
## Darknet OPs supported by RKNN Toolkit2 |
||||
The list of Darknet OPs supported by RKNN Toolkit2 Version 1.1.0 is as follows: |
||||
|
||||
| **Operators** | **Remarks** | |
||||
|---| ----------- | |
||||
|add|channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />support broadcast rule: per-layer/channel/element | |
||||
|batchnormalize|channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
|concat|axis: 1,2,3| |
||||
|convolutional|hannel: [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: [1, 7]<br />kernels: [1, 8184]<br />pad left/right/top/bottom: [0, 15]<br />dilation: [1, 31]<br />group: 1, channel / N| |
||||
|depthwise_convolutional|channel:[1, 8192]<br />kernel height/width: [1, 8]<br />stride height/width: [1, 7]<br />kernels: 1<br />pad left/right/top/bottom: [0, 15]| |
||||
|fullconnect|channel: [1, 8192]| |
||||
|leakyrelu|channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]| |
||||
|mish|| |
||||
|pooling|**AveragePool**:<br />channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]<br /><br /> **GlobalAveragePool**:<br /> channel: [1, 8192]<br />kernel height/width: [1, 128]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7] <br /><br /> **MaxPool/GlobalMaxPool**:<br /> channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]<br />MaxPool: <br />auto_pad only support NOTSET，ceil_mode only support 0，unsupport dilations| |
||||
|route|| |
||||
|shortcut|| |
||||
|softmax|| |
||||
|upsampling|| |
||||
@ -0,0 +1,240 @@
@@ -0,0 +1,240 @@
|
||||
# RKNNToolkit2 OPs Support |
||||
|
||||
## Explanation of terms: |
||||
|
||||
Remarks: |
||||
|
||||
Operators' specifications must meet the remarks' requirements. |
||||
|
||||
Broadcast rule: |
||||
|
||||
- per-layer： |
||||
|
||||
shape(A) = (2, 3, 4, 5), shape(B) = (,), i.e. B is a scalar ==> shape(result) = (2, 3, 4, 5) |
||||
|
||||
- per-channel: |
||||
|
||||
shape(A) = (2, 3, 4, 5), shape(B) = (3,), ==> shape(result) = (2, 3, 4, 5) |
||||
|
||||
shape(A) = (2, 3, 4, 5), shape(B) = (1,3,1,1), ==> shape(result) = (2, 3, 4, 5) |
||||
|
||||
- per-element: |
||||
|
||||
shape(A) = (2, 3, 4, 5), shape(B) = (2,3,4,5) ==> shape(result) = (2, 3, 4, 5) |
||||
|
||||
- other: |
||||
|
||||
shape(A) = (2, 3, 4, 5), shape(B) = (5,) ==> shape(result) = (2, 3, 4, 5) |
||||
|
||||
|
||||
## ONNX OPs supported by RKNN Toolkit2 |
||||
|
||||
According to [ONNX official instructions](https://github.com/microsoft/onnxruntime/blob/master/docs/Versioning.md 'ONNX Version Description'), the corresponding ONNX opset version is 12. |
||||
The list of ONNX OPs supported by RKNN Toolkit2 is as follows: |
||||
|
||||
| **Operators** | **Remarks** | |
||||
|-----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
||||
| Add | channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />support broadcast rule: per-layer/channel/element | |
||||
| AveragePool | channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7] | |
||||
| ArgMin || |
||||
| ArgMax || |
||||
| BatchNormalization | channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176] | |
||||
| cast | only support bool/int8/float | |
||||
| Clip | channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176] | |
||||
| Concat | axis: 1,2,3 | |
||||
| Conv | channel: unlimited <br />kernel height/width: [1, 31]<br />stride height/width: [1, 7]<br />kernels: [1, 8184]<br />pad left/right/top/bottom: [0, 15]<br />dilation: [1, 31]<br />group: unlimited | |
||||
| ConvTranspose | channel: unlimited<br />kernel height/width: [1, 31]<br />stride height/width: 2, 4, 8<br />kernels: [1, 8192]<br />pad left/right/top/bottom: [0, 15]<br />dilation: [1, 31]<br />group: unlimited | |
||||
| DepthToSpace || |
||||
| Div | support broadcast rule: per-element/other | |
||||
| Elu || |
||||
| Exp || |
||||
| Flatten || |
||||
| Gather || |
||||
| Gemm | channel: unlimited<br /> One input should be Const | |
||||
| GlobalAveragePool | channel: [1, 8192]<br />kernel height/width: [1, 343]<br /> | |
||||
| GlobalMaxPool | channel: [1, 8192]<br />kernel height/width: [1, 343]<br /> | |
||||
| Greater | support broadcast rule: per-element/other | |
||||
| HardSigmoid || |
||||
| InstanceNormalization || |
||||
| HardSwish || |
||||
| LeakyRelu | channel: unlimited<br />height: [1, 8192]<br />width: [1, 8176] | |
||||
| Less | support broadcast rule: per-element/other | |
||||
| LpNormalization || |
||||
| LRN || |
||||
| LSTM | batchsize: 1 | |
||||
| GRU | batchsize: 1 | |
||||
| MatMul | | |
||||
| Max | channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br /> dims=4 | |
||||
| MaxPool | channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]<br />auto_pad only support NOTSET，ceil_mode only support 0，unsupport dilations | |
||||
| MaxRoiPool || |
||||
| MaxUnpool | unsupport pad | |
||||
| Mul | channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />support broadcast rule: per-layer/channel/element | |
||||
| Pad | pad value should >= 0; pad dims must be 2 when mode is reflect or edge | |
||||
| Pow || |
||||
| PRelu | channel: unlimited<br />height: [1, 8192]<br />width: [1, 8176]<br />slope support broadcast rule:: per-layer/channel | |
||||
| ReduceMax || |
||||
| ReduceMean | output dims <= 4 | |
||||
| ReduceSum | output dims <= 4 | |
||||
| ReduceMin || |
||||
| Relu | channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176] | |
||||
| Reshape || |
||||
| Resize | bilinear（not support tf_crop_and_resize）; nearest2d | |
||||
| ReverseSequence || |
||||
| RoiAlign | pool type: average | |
||||
| Sigmoid || |
||||
| Slice || |
||||
| Softmax || |
||||
| SpaceToDetph || |
||||
| Split || |
||||
| Squeeze || |
||||
| Tanh | channel: unlimited<br />height: [1, 8192]<br />width: [1, 8176] | |
||||
| Tile || |
||||
| Transpose || |
||||
| Upsample (resize) | coordinate_transformation_mode unsupport tf_crop_and_resize | |
||||
|
||||
|
||||
## Caffe OPs supported by RKNN Toolkit2 |
||||
|
||||
Caffe protocols RKNN Toolkit2 uses only based on the officially modified protocol of berkeley. |
||||
The protocol based on the official revision of berkeley comes from [berkeley caffe](https://github.com/BVLC/caffe/tree/master/src/caffe/proto 'Berkeley Caffe'), commit hash is 21d0608. On this basis RKNN Toolkit2 have added some OPs. |
||||
Based on this protocol, the list of Caffe OPs supported by RKNN Toolkit2 is as follows: |
||||
|
||||
| **Operators** | **Remarks** | |
||||
|------------------------|---------------------------------------------------------------------------------------------------------------------------------------------| |
||||
| BatchNorm | same as onnx BatchNormalization | |
||||
| bn (BatchNorm + Scale) | same as onnx BatchNormalization according to https://github.com/TimoSaemann/caffe-segnet-cudnn5 | |
||||
| BNLL || |
||||
| Concat | same as onnx Concat | |
||||
| Convolution | same as onnx Conv | |
||||
| ConvolutionDepthwise | channel:unlimited<br />kernel height/width: [1, 8]<br />stride height/width: [1, 7]<br />kernels: 1<br />pad left/right/top/bottom: [0, 15] | |
||||
| Crop || |
||||
| Deconvolution | same as ConvTranspose | |
||||
| Dropout || |
||||
| Eltwise | channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />support broadcast rule: per-layer/channel/element | |
||||
| Flatten || |
||||
| HardSigmoid || |
||||
| InnerProduct | same as onnx Gemm | |
||||
| LRN | same as onnx LRN | |
||||
| Lstm | same as onnx LSTM according to https://github.com/xmfbit/warpctc-caffe | |
||||
| Normalize || |
||||
| Permute | same as onnx Transpose | |
||||
| Power || |
||||
| Pooling | same as onnx pooling | |
||||
| PRelu | same as onnx PRelu | |
||||
| Proposal | batch: 1 | |
||||
| Reduction | output dims <= 4 | |
||||
| Relu | same as onnx Relu | |
||||
| Relu6 | same as onnx Clip | |
||||
| Reorg || |
||||
| Reshape | same as onnx Reshape | |
||||
| Resize | bilinear; nearest | |
||||
| Reverse || |
||||
| ROIPooling | same as MaxRoiPool according to https://github.com/twmht/caffe-pva-faster-rcnn | |
||||
| Scale | channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176] | |
||||
| Sigmoid | same as onnx Sigmoid | |
||||
| Slice | same as onnx Split | |
||||
| Softmax | same as onnx Softmax | |
||||
| Split | same as onnx Slice | |
||||
| TanH | same as onnx TanH | |
||||
| Tile | same as onnx Tile | |
||||
| Transpose | same as onnx Transpose | |
||||
| Upsample | according to https://github.com/SeanQ88/caffe_upsample and https://github.com/TimoSaemann/caffe-segnet-cudnn5 | |
||||
|
||||
|
||||
## Pytorch OPs supported by RKNN Toolkit2 |
||||
|
||||
The Pytorch version supported by RKNN Toolkit2 is >1.6.0, models generated by other versions may not support. |
||||
The list of Pytorch OPs supported by RKNN Toolkit2 is as follows: |
||||
|
||||
| **Operators** | **Remarks** | |
||||
|---------------------------|------------------------------------| |
||||
| aten::_convolution | same as onnx Conv | |
||||
| aten::add | same as onnx Add | |
||||
| aten::avg_pool2d | same as onnx AveragePool | |
||||
| aten::batch_norm | same as onnx BatchNormalization | |
||||
| aten::cat | same as onnx Concat | |
||||
| aten::chunk || |
||||
| aten::dropout || |
||||
| aten::elu | same as onnx Elu | |
||||
| aten::flatten || |
||||
| aten::hardswish | same as onnx HardSwish | |
||||
| aten::instance_norm | same as onnx InstanceNormalization | |
||||
| aten::layer_norm || |
||||
| aten::leaky_relu | same as onnx LeakyRelu | |
||||
| aten::linear | same as onnx Gemm | |
||||
| aten::matmul | same as onnx MatMul | |
||||
| aten::max_pool2d | same as onnx MaxPool | |
||||
| aten::mean | output dims <= 4 | |
||||
| aten::mul | same as onnx Mul | |
||||
| aten::pad | same as onnx Pad | |
||||
| aten::permute | same as onnx Transpose | |
||||
| aten::prelu | same as onnx PRelu | |
||||
| aten::relu | same as onnx Relu | |
||||
| aten::reshape | | |
||||
| aten::sigmoid | same as onnx Sigmoid | |
||||
| aten::slice | same as onnx Slice | |
||||
| aten::split | same as onnx Split | |
||||
| aten::softmax | same as onnx Softmax | |
||||
| aten::squeeze | same as onnx Squeeze | |
||||
| aten::sum | output dims <= 4 | |
||||
| aten::tanh | same as onnx TanH | |
||||
| aten::upsample_bilinear2d | same as onnx Resize | |
||||
| aten::upsample_nearest2d | same as onnx Resize | |
||||
| aten::view | same as onnx Reshape | |
||||
|
||||
## TensorFlow OPs supported by RKNN Toolkit2 |
||||
|
||||
The pb files (contain OPs belows) generated by TensorFlow version 1.12 - 1.15 for 1.x and 2.3 - 2.5 for 2.x are supported by RKNN Toolkit2. For more information on TensorFlow version compatibility, please refer to [tensorflow official instructions on OP version](https://www.tensorflow.org/guide/versions 'Tensorflow official instructions on OP version') . |
||||
The list of TensorFlow OPs supported by RKNN Toolkit2 is as follows: |
||||
|
||||
| **Operators** | **Remarks** | |
||||
|-----------------------|---------------------------------------------------------------------------------------------------------------------------------------------| |
||||
| Add | same as onnx Add | |
||||
| AvgPool | same as onnx AveragePool | |
||||
| Concat | same as onnx Concat | |
||||
| Conv2D | same as onnx Conv | |
||||
| DepthToSpace || |
||||
| DepthwiseConv2d | channel:unlimited<br />kernel height/width: [1, 8]<br />stride height/width: [1, 7]<br />kernels: 1<br />pad left/right/top/bottom: [0, 15] | |
||||
| Div | same as onnx Div | |
||||
| Dropout || |
||||
| Flatten || |
||||
| LeakyRelu | same as onnx LeakyRelu | |
||||
| Less | same as onnx Less | |
||||
| LRN || |
||||
| MatMul | | |
||||
| MaxPool | same as onnx MaxPool | |
||||
| Mean | output dims <= 4 | |
||||
| Pad | same as onnx Pad | |
||||
| Relu | same as onnx Relu | |
||||
| Reshape || |
||||
| ResizeBilinear || |
||||
| ResizeNearestNeighbor || |
||||
| Sigmoid || |
||||
| Slice || |
||||
| Softmax || |
||||
| Softplus || |
||||
| SpaceToDepth || |
||||
| Split || |
||||
| Squeeze || |
||||
| StridedSlice || |
||||
| Tanh | same as onnx TanH | |
||||
| Transpose || |
||||
|
||||
## Darknet OPs supported by RKNN Toolkit2 |
||||
The list of Darknet OPs supported by RKNN Toolkit2 is as follows: |
||||
|
||||
| **Operators** | **Remarks** | |
||||
|-------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
||||
| add | same as onnx Add | |
||||
| batchnormalize | same as onnx BatchNormalization | |
||||
| concat | same as onnx Concat | |
||||
| convolutional | same as onnx Conv | |
||||
| depthwise_convolutional | channel:unlimited<br />kernel height/width: [1, 8]<br />stride height/width: [1, 7]<br />kernels: 1<br />pad left/right/top/bottom: [0, 15] | |
||||
| fullconnect | | |
||||
| leakyrelu | same as onnx LeakyRelu | |
||||
| mish || |
||||
| pooling | **AveragePool**:<br />channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]<br /><br /> **GlobalAveragePool**:<br /> channel: [1, 8192]<br />kernel height/width: [1, 128]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7] <br /> <br /> **MaxPool/GlobalMaxPool**:<br /> channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]<br /><br />MaxPool: <br />auto_pad only support NOTSET，ceil_mode only support 0，unsupport dilations | |
||||
| route || |
||||
| shortcut || |
||||
| softmax || |
||||
| upsampling || |
||||
@ -0,0 +1,337 @@
@@ -0,0 +1,337 @@
|
||||
# RKNNToolkit2 API Difference With Toolkit1 |
||||
|
||||
## rknn.config |
||||
- Toolkit1: |
||||
|
||||
config(batch_size=100, # abandoned |
||||
caffe_mean_file=None, # abandoned |
||||
dtype='float32', # abandoned |
||||
epochs=-1, # abandoned |
||||
force_gray=None, # abandoned |
||||
input_fitting='scale', # abandoned |
||||
input_normalization=None, # abandoned |
||||
mean_file=None, # abandoned |
||||
model_data_format='zone', # abandoned |
||||
model_quantize=None, # abandoned |
||||
optimize='Default', # abandoned |
||||
quantized_dtype='asymmetric_affine-u8', |
||||
quantized_moving_alpha=0.01, # abandoned |
||||
quantized_algorithm='normal', |
||||
quantized_divergence_nbins=1024, # abandoned |
||||
mmse_epoch=3, # abandoned |
||||
random_brightness=None, # abandoned |
||||
random_contrast=None, # abandoned |
||||
random_crop=None, # abandoned |
||||
random_flip=None, # abandoned |
||||
random_mirror=None, # abandoned |
||||
reorder_channel=None, # abandoned |
||||
restart=False, # abandoned |
||||
samples=-1, # abandoned |
||||
need_horizontal_merge=False, # abandoned |
||||
deconv_merge=True, # abandoned |
||||
conv_mul_merge=True, # abandoned |
||||
quantized_hybrid=False, # abandoned |
||||
output_optimize=0, # abandoned |
||||
remove_tensorflow_output_permute=False, # abandoned |
||||
optimization_level=3, |
||||
target_platform=None, |
||||
mean_values=None, |
||||
std_values=None, |
||||
channel_mean_value=None, # abandoned |
||||
force_builtin_perm=False, # abandoned |
||||
do_sparse_network=True, # abandoned |
||||
merge_dequant_layer_and_output_node=False, # abandoned |
||||
quantize_input_node=False, # abandoned |
||||
inputs_scale_range=None) # abandoned |
||||
|
||||
- Toolkit2: |
||||
|
||||
config(mean_values=None, |
||||
std_values=None, |
||||
quantized_dtype='asymmetric_quantized-8', |
||||
quantized_algorithm='normal', |
||||
quantized_method='channel', # new |
||||
target_platform=None, |
||||
quant_img_RGB2BGR=False, # new |
||||
float_dtype='float16', # new |
||||
optimization_level=3, |
||||
custom_string=None, # new |
||||
output_tensor_type=None) # new |
||||
|
||||
- In addition to the above abandoned/new items, there are other differences: |
||||
|
||||
quantized_dtype: |
||||
toolkit1: asymmetric_affine-u8, dynamic_fixed_point-i8, dynamic_fixed_point-i16 |
||||
toolkit2: asymmetric_quantized-8 |
||||
quantized_algorithm: |
||||
toolkit1: normal(default), mmse, kl_divergence, moving_average |
||||
toolkit2: normal(default), mmse |
||||
target_platform: |
||||
toolkit1: rk1808, rk3399pro, rv1109, rv1126 |
||||
toolkit2: rk3566, rk3568, rk3588 |
||||
|
||||
## rknn.load_tensorflow |
||||
- Toolkit1: |
||||
|
||||
load_tensorflow(tf_pb, |
||||
inputs, |
||||
input_size_list, |
||||
outputs, |
||||
predef_file=None, # abandoned |
||||
mean_values=None, # abandoned |
||||
std_values=None, # abandoned |
||||
size_with_batch=None) # abandoned |
||||
|
||||
- Toolkit2: |
||||
|
||||
load_tensorflow(tf_pb, |
||||
inputs, |
||||
input_size_list, |
||||
outputs) |
||||
|
||||
- In addition to the above abandoned items, there are other differences: |
||||
|
||||
inputs: |
||||
toolkit1: node list (layer name) |
||||
toolkit2: node list (operand name) |
||||
outputs: |
||||
toolkit1: node list (layer name) |
||||
toolkit2: node list (operand name) |
||||
|
||||
## rknn.load_caffe |
||||
- Toolkit1: |
||||
|
||||
load_caffe(model, |
||||
proto, # abandoned |
||||
blobs=None) |
||||
- Toolkit2: |
||||
|
||||
load_caffe(model, |
||||
blobs=None, |
||||
input_name=None) # new |
||||
|
||||
|
||||
## rknn.load_keras |
||||
- Toolkit1: |
||||
|
||||
load_keras(model, convert_engine='Keras') |
||||
- Toolkit2: |
||||
|
||||
Not supported yet! |
||||
|
||||
## rknn.load_pytorch |
||||
- Toolkit1: |
||||
|
||||
load_pytorch(model, |
||||
input_size_list=None, |
||||
inputs=None, # abandoned |
||||
outputs=None, # abandoned |
||||
convert_engine='torch') # abandoned |
||||
- Toolkit2: |
||||
|
||||
load_pytorch(model, |
||||
input_size_list) |
||||
|
||||
## rknn.load_mxnet |
||||
- Toolkit1: |
||||
|
||||
load_mxnet(symbol, params, input_size_list=None) |
||||
- Toolkit2: |
||||
|
||||
Not supported yet! |
||||
|
||||
## rknn.build |
||||
- Toolkit1: |
||||
|
||||
build(do_quantization=True, |
||||
dataset='dataset.txt', |
||||
pre_compile=False, # abandoned |
||||
rknn_batch_size=-1) |
||||
- Toolkit2: |
||||
|
||||
build(do_quantization=True, |
||||
dataset='dataset.txt', |
||||
rknn_batch_size=-1): |
||||
|
||||
|
||||
## rknn.direct_build |
||||
- Toolkit1: |
||||
|
||||
direct_build(model_input, data_input, model_quantize=None, pre_compile=False) |
||||
- Toolkit2: |
||||
|
||||
Not supported yet! |
||||
|
||||
## rknn.hybrid_quantization_step1 |
||||
- Toolkit1: |
||||
|
||||
hybrid_quantization_step1(dataset=None) |
||||
- Toolkit2: |
||||
|
||||
hybrid_quantization_step1(dataset=None, |
||||
rknn_batch_size=-1, # new |
||||
proposal=False, # new |
||||
proposal_dataset_size=1) # new |
||||
|
||||
|
||||
## rknn.hybrid_quantization_step2 |
||||
- Toolkit1: |
||||
|
||||
hybrid_quantization_step2(model_input, |
||||
data_input, |
||||
model_quantization_cfg, |
||||
dataset, # abandoned |
||||
pre_compile=False) # abandoned |
||||
- Toolkit2: |
||||
|
||||
hybrid_quantization_step2(model_input, |
||||
data_input, |
||||
model_quantization_cfg) |
||||
|
||||
|
||||
## rknn.accuracy_analysis |
||||
- Toolkit1: |
||||
|
||||
accuracy_analysis(inputs, |
||||
output_dir='./snapshot', |
||||
calc_qnt_error=True, # abandoned |
||||
target=None, |
||||
device_id=None, |
||||
dump_file_type='tensor') # abandoned |
||||
- Toolkit2: |
||||
|
||||
accuracy_analysis(inputs, |
||||
output_dir='./snapshot', |
||||
target=None, |
||||
device_id=None) |
||||
|
||||
|
||||
## rknn.load_rknn |
||||
- Toolkit1: |
||||
|
||||
load_rknn(path, |
||||
load_model_in_npu=False) # abandoned |
||||
- Toolkit2: |
||||
|
||||
load_rknn(path) |
||||
|
||||
|
||||
## rknn.export_rknn |
||||
- Toolkit1: |
||||
|
||||
export_rknn(export_path) |
||||
- Toolkit2: |
||||
|
||||
export_rknn(export_path, |
||||
**kwargs) # new |
||||
|
||||
## rknn.load_firmware |
||||
- Toolkit1: |
||||
|
||||
load_firmware(fw_dir=None) |
||||
- Toolkit2: |
||||
|
||||
Not supported yet! |
||||
|
||||
|
||||
## rknn.init_runtime |
||||
- Toolkit1: |
||||
|
||||
init_runtime(target=None, |
||||
target_sub_class=None, |
||||
device_id=None, |
||||
perf_debug=False, |
||||
eval_mem=False, |
||||
async_mode=False, |
||||
rknn2precompile=False) # abandoned |
||||
- Toolkit2: |
||||
|
||||
init_runtime(target=None, |
||||
target_sub_class=None, |
||||
device_id=None, |
||||
perf_debug=False, |
||||
eval_mem=False, |
||||
async_mode=False) |
||||
|
||||
- In addition to the above abandoned items, there are other differences: |
||||
|
||||
target: |
||||
toolkit1: None(simulator), RK3399Pro, RK1808 |
||||
toolkit2: None(simulator), RK3566, RK3568, RK3588 |
||||
|
||||
|
||||
|
||||
## rknn.inference |
||||
- Toolkit1: |
||||
|
||||
inference(inputs, |
||||
data_type=None, # abandoned |
||||
data_format=None, |
||||
inputs_pass_through=None, |
||||
get_frame_id=False) |
||||
- Toolkit2: |
||||
|
||||
inference(inputs, |
||||
data_format=None, |
||||
inputs_pass_through=None, |
||||
get_frame_id=False) |
||||
|
||||
|
||||
## rknn.eval_perf |
||||
- Toolkit1: |
||||
|
||||
eval_perf(inputs=None, |
||||
data_type=None, # abandoned |
||||
data_format=None, |
||||
is_print=True, |
||||
loop_cnt=1) # abandoned |
||||
- Toolkit2: |
||||
|
||||
eval_perf(inputs=None, |
||||
data_format=None, |
||||
is_print=True) |
||||
|
||||
|
||||
## rknn.export_rknn_precompile_model |
||||
- Toolkit1: |
||||
|
||||
export_rknn_precompile_model(export_path=None) |
||||
- Toolkit2: |
||||
|
||||
Abandoned |
||||
|
||||
|
||||
## rknn.export_rknn_sync_model |
||||
- Toolkit1: |
||||
|
||||
export_rknn_sync_model(input_model=None, sync_uids=None, output_model=None) |
||||
- Toolkit2: |
||||
|
||||
Abandoned |
||||
|
||||
|
||||
## rknn.register_op |
||||
- Toolkit1: |
||||
|
||||
register_op(op_path) |
||||
- Toolkit2: |
||||
|
||||
Not supported yet |
||||
|
||||
|
||||
## rknn.fetch_rknn_model_config |
||||
- Toolkit1: |
||||
|
||||
fetch_rknn_model_config(model_path) |
||||
- Toolkit2: |
||||
|
||||
Not supported yet |
||||
|
||||
|
||||
## rknn.list_support_target_platform |
||||
- Toolkit1: |
||||
|
||||
list_support_target_platform(rknn_model=None) |
||||
- Toolkit2: |
||||
|
||||
Not supported yet |
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@ -1,4 +1,18 @@
@@ -1,4 +1,18 @@
|
||||
2021-8-20 |
||||
2021-1-12 |
||||
版本：v1.2.0 |
||||
更新内容: |
||||
1. 新功能: rk3588平台的支持; rknn模型加密支持; tensorflow/tflite/pytorch量化感知模型支持; 增加了一些新的 op 支持: InstanceNormalization, Swish, Conv1D等（详见 op support list）；增加了参数量计算以及算力分析 |
||||
2. examples 更新：增加了从 pytorch 转 onnx 的转换 demo：resnet18_export_onnx ；增加了pytorch量化感知模型的加载demo：resnet18_qat demo；增加了模型加密功能 |
||||
3. 接口更改：移除了 config，load_caffe，load_tensorflow等接口的一些不必要的参数设置，更新了 eval_perf 接口，详细改动见Uer_Guide文档 |
||||
4. 修复一些已知的bug: |
||||
1) 修复了一些模型无法转换rknn的问题 |
||||
2) 修复了一些图优化bug |
||||
3) 修复了一些模型推理结果错误的问题 |
||||
4) 修复了 pytorch、tflite 某些 op 转换失败的问题 |
||||
5. 优化: 精度分析耗时优化; 模型转换和量化耗时优化 |
||||
|
||||
|
||||
2021-8-12 |
||||
版本：v1.1.0 |
||||
更新内容: |
||||
1. 新功能: LSTM，GRU的支持；增加了accuracy_analysis对比项目；增加了一些op支持：caffe hardswish；onnx gather,reduceMax等op；更新了更全面的图优化规则。 |
||||
@ -1 +1,20 @@
@@ -1 +1,20 @@
|
||||
goldfish_224x224.jpg |
||||
./imgs/ILSVRC2012_val_00000665.JPEG |
||||
./imgs/ILSVRC2012_val_00001123.JPEG |
||||
./imgs/ILSVRC2012_val_00001129.JPEG |
||||
./imgs/ILSVRC2012_val_00001284.JPEG |
||||
./imgs/ILSVRC2012_val_00003026.JPEG |
||||
./imgs/ILSVRC2012_val_00005276.JPEG |
||||
./imgs/ILSVRC2012_val_00006178.JPEG |
||||
./imgs/ILSVRC2012_val_00007829.JPEG |
||||
./imgs/ILSVRC2012_val_00009338.JPEG |
||||
./imgs/ILSVRC2012_val_00010420.JPEG |
||||
./imgs/ILSVRC2012_val_00011878.JPEG |
||||
./imgs/ILSVRC2012_val_00011240.JPEG |
||||
./imgs/ILSVRC2012_val_00012037.JPEG |
||||
./imgs/ILSVRC2012_val_00012351.JPEG |
||||
./imgs/ILSVRC2012_val_00012470.JPEG |
||||
./imgs/ILSVRC2012_val_00020169.JPEG |
||||
./imgs/ILSVRC2012_val_00020173.JPEG |
||||
./imgs/ILSVRC2012_val_00021224.JPEG |
||||
./imgs/ILSVRC2012_val_00024249.JPEG |
||||
./imgs/ILSVRC2012_val_00025601.JPEG |
||||
|
Before Width: | Height: | Size: 18 KiB After Width: | Height: | Size: 18 KiB |
|
Before Width: | Height: | Size: 55 KiB |
|
After Width: | Height: | Size: 35 KiB |
|
After Width: | Height: | Size: 63 KiB |
|
After Width: | Height: | Size: 58 KiB |
|
After Width: | Height: | Size: 38 KiB |
|
After Width: | Height: | Size: 44 KiB |
|
After Width: | Height: | Size: 39 KiB |
|
After Width: | Height: | Size: 34 KiB |
|
After Width: | Height: | Size: 29 KiB |
|
After Width: | Height: | Size: 54 KiB |
|
After Width: | Height: | Size: 36 KiB |
|
After Width: | Height: | Size: 56 KiB |
|
After Width: | Height: | Size: 45 KiB |
|
After Width: | Height: | Size: 39 KiB |
|
After Width: | Height: | Size: 46 KiB |
|
After Width: | Height: | Size: 36 KiB |
|
After Width: | Height: | Size: 49 KiB |
|
After Width: | Height: | Size: 55 KiB |
|
After Width: | Height: | Size: 55 KiB |
|
After Width: | Height: | Size: 30 KiB |
|
After Width: | Height: | Size: 37 KiB |
@ -0,0 +1 @@
@@ -0,0 +1 @@
|
||||
dog_224x224.jpg |
||||
|
After Width: | Height: | Size: 18 KiB |
@ -0,0 +1,106 @@
@@ -0,0 +1,106 @@
|
||||
import numpy as np |
||||
import cv2 |
||||
from rknn.api import RKNN |
||||
|
||||
|
||||
def show_outputs(outputs): |
||||
output = outputs[0].reshape(-1) |
||||
output_sorted = sorted(output, reverse=True) |
||||
top5_str = 'mobilenet_v2\n-----TOP 5-----\n' |
||||
for i in range(5): |
||||
value = output_sorted[i] |
||||
index = np.where(output == value) |
||||
for j in range(len(index)): |
||||
if (i + j) >= 5: |
||||
break |
||||
if value > 0: |
||||
topi = '{}: {}\n'.format(index[j], value) |
||||
else: |
||||
topi = '-1: 0.0\n' |
||||
top5_str += topi |
||||
print(top5_str) |
||||
|
||||
|
||||
if __name__ == '__main__': |
||||
|
||||
# Create RKNN object |
||||
rknn = RKNN(verbose=True) |
||||
|
||||
# Pre-process config |
||||
print('--> Config model') |
||||
rknn.config(mean_values=[103.94, 116.78, 123.68], std_values=[58.82, 58.82, 58.82], |
||||
quant_img_RGB2BGR=True, target_platform='rk3566') |
||||
print('done') |
||||
|
||||
# Load model |
||||
print('--> Loading model') |
||||
ret = rknn.load_caffe(model='./../../caffe/mobilenet_v2/mobilenet_v2.prototxt', |
||||
blobs='./../../caffe/mobilenet_v2/mobilenet_v2.caffemodel') |
||||
if ret != 0: |
||||
print('Load model failed!') |
||||
exit(ret) |
||||
print('done') |
||||
|
||||
# Build model |
||||
print('--> Building model') |
||||
ret = rknn.build(do_quantization=True, dataset='./dataset.txt') |
||||
if ret != 0: |
||||
print('Build model failed!') |
||||
exit(ret) |
||||
print('done') |
||||
|
||||
# Export rknn model |
||||
print('--> Export rknn model') |
||||
ret = rknn.export_rknn('./mobilenet_v2.rknn') |
||||
if ret != 0: |
||||
print('Export rknn model failed!') |
||||
exit(ret) |
||||
print('done') |
||||
|
||||
# Export encrypted RKNN model |
||||
print('--> Export encrypted rknn model') |
||||
ret = rknn.export_encrypted_rknn_model('./mobilenet_v2.rknn', None, 3) |
||||
|
||||
# load rknn model |
||||
print('--> Load rknn model') |
||||
ret = rknn.load_rknn('./mobilenet_v2.rknn') |
||||
if ret != 0: |
||||
print('Load rknn model failed!') |
||||
exit(ret) |
||||
print('done') |
||||
|
||||
# Set inputs |
||||
img = cv2.imread('./dog_224x224.jpg') |
||||
|
||||
# Init runtime environment |
||||
print('--> Init runtime environment') |
||||
ret = rknn.init_runtime(target='rk3566', perf_debug=True, eval_mem=True) |
||||
if ret != 0: |
||||
print('Init runtime environment failed!') |
||||
exit(ret) |
||||
print('done') |
||||
|
||||
# eval perf |
||||
print('--> Eval perf') |
||||
rknn.eval_perf(inputs=[img]) |
||||
|
||||
# eval perf |
||||
print('--> Eval memory') |
||||
rknn.eval_memory() |
||||
|
||||
# Inference |
||||
print('--> Running model') |
||||
outputs = rknn.inference(inputs=[img]) |
||||
np.save('./functions_board_test_0.npy', outputs[0]) |
||||
show_outputs(outputs) |
||||
print('done') |
||||
|
||||
# Accuracy analysis |
||||
print('--> Accuracy analysis') |
||||
ret = rknn.accuracy_analysis(inputs=['./dog_224x224.jpg'], output_dir='./snapshot', target='rk3566') |
||||
if ret != 0: |
||||
print('Accuracy analysis failed!') |
||||
exit(ret) |
||||
print('done') |
||||
|
||||
rknn.release() |
||||
|
After Width: | Height: | Size: 177 KiB |
@ -1 +1 @@
@@ -1 +1 @@
|
||||
dog_bike_car_640x640.jpg |
||||
bus.jpg |
||||
|
||||
|
Before Width: | Height: | Size: 292 KiB |
@ -1,301 +1,308 @@
@@ -1,301 +1,308 @@
|
||||
import cv2 |
||||
from PIL import Image |
||||
import os |
||||
import urllib |
||||
import traceback |
||||
import time |
||||
import sys |
||||
import numpy as np |
||||
import torch |
||||
import onnxruntime |
||||
|
||||
import cv2 |
||||
from rknn.api import RKNN |
||||
|
||||
NUM_CLS = 80 |
||||
LISTSIZE = NUM_CLS+5 |
||||
SPAN = 3 |
||||
OBJ_THRESH = 0.2 |
||||
NMS_THRESH = 0.5 |
||||
CLASSES = ("person", "bicycle", "car","motorbike ","aeroplane ","bus ","train","truck ","boat","traffic light", |
||||
"fire hydrant","stop sign ","parking meter","bench","bird","cat","dog ","horse ","sheep","cow","elephant", |
||||
"bear","zebra ","giraffe","backpack","umbrella","handbag","tie","suitcase","frisbee","skis","snowboard","sports ball","kite", |
||||
"baseball bat","baseball glove","skateboard","surfboard","tennis racket","bottle","wine glass","cup","fork","knife ", |
||||
"spoon","bowl","banana","apple","sandwich","orange","broccoli","carrot","hot dog","pizza ","donut","cake","chair","sofa", |
||||
"pottedplant","bed","diningtable","toilet ","tvmonitor","laptop ","mouse ","remote ","keyboard ","cell phone","microwave ", |
||||
"oven ","toaster","sink","refrigerator ","book","clock","vase","scissors ","teddy bear ","hair drier", "toothbrush ") |
||||
|
||||
masks = [[0,1,2], [3,4,5], [6,7,8]] #yolov5s |
||||
anchors = [[10,13],[16,30],[33,23],[30,61],[62,45],[59,119],[116,90],[156,198],[373,326]] |
||||
|
||||
def letterbox_image(image, size): |
||||
iw, ih = image.size |
||||
w, h = size |
||||
scale = min(w / iw, h / ih) |
||||
nw = int(iw * scale) |
||||
nh = int(ih * scale) |
||||
|
||||
image = np.array(image) |
||||
image = cv2.resize(image, (nw, nh), interpolation=cv2.INTER_LINEAR) |
||||
image = Image.fromarray(image) |
||||
new_image = Image.new('RGB', size, (128, 128, 128)) |
||||
new_image.paste(image, ((w - nw) // 2, (h - nh) // 2)) |
||||
return new_image |
||||
|
||||
def w_bbox_iou(box1, box2, x1y1x2y2=True): |
||||
""" |
||||
计算IOU |
||||
ONNX_MODEL = 'yolov5s.onnx' |
||||
RKNN_MODEL = 'yolov5s.rknn' |
||||
IMG_PATH = './bus.jpg' |
||||
DATASET = './dataset.txt' |
||||
|
||||
QUANTIZE_ON = True |
||||
|
||||
BOX_THESH = 0.5 |
||||
NMS_THRESH = 0.6 |
||||
IMG_SIZE = 640 |
||||
|
||||
CLASSES = ("person", "bicycle", "car", "motorbike ", "aeroplane ", "bus ", "train", "truck ", "boat", "traffic light", |
||||
"fire hydrant", "stop sign ", "parking meter", "bench", "bird", "cat", "dog ", "horse ", "sheep", "cow", "elephant", |
||||
"bear", "zebra ", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball", "kite", |
||||
"baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork", "knife ", |
||||
"spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza ", "donut", "cake", "chair", "sofa", |
||||
"pottedplant", "bed", "diningtable", "toilet ", "tvmonitor", "laptop ", "mouse ", "remote ", "keyboard ", "cell phone", "microwave ", |
||||
"oven ", "toaster", "sink", "refrigerator ", "book", "clock", "vase", "scissors ", "teddy bear ", "hair drier", "toothbrush ") |
||||
|
||||
|
||||
def sigmoid(x): |
||||
return 1 / (1 + np.exp(-x)) |
||||
|
||||
|
||||
def xywh2xyxy(x): |
||||
# Convert [x, y, w, h] to [x1, y1, x2, y2] |
||||
y = np.copy(x) |
||||
y[:, 0] = x[:, 0] - x[:, 2] / 2 # top left x |
||||
y[:, 1] = x[:, 1] - x[:, 3] / 2 # top left y |
||||
y[:, 2] = x[:, 0] + x[:, 2] / 2 # bottom right x |
||||
y[:, 3] = x[:, 1] + x[:, 3] / 2 # bottom right y |
||||
return y |
||||
|
||||
|
||||
def process(input, mask, anchors): |
||||
|
||||
anchors = [anchors[i] for i in mask] |
||||
grid_h, grid_w = map(int, input.shape[0:2]) |
||||
|
||||
box_confidence = sigmoid(input[..., 4]) |
||||
box_confidence = np.expand_dims(box_confidence, axis=-1) |
||||
|
||||
box_class_probs = sigmoid(input[..., 5:]) |
||||
|
||||
box_xy = sigmoid(input[..., :2])*2 - 0.5 |
||||
|
||||
col = np.tile(np.arange(0, grid_w), grid_w).reshape(-1, grid_w) |
||||
row = np.tile(np.arange(0, grid_h).reshape(-1, 1), grid_h) |
||||
col = col.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2) |
||||
row = row.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2) |
||||
grid = np.concatenate((col, row), axis=-1) |
||||
box_xy += grid |
||||
box_xy *= int(IMG_SIZE/grid_h) |
||||
|
||||
box_wh = pow(sigmoid(input[..., 2:4])*2, 2) |
||||
box_wh = box_wh * anchors |
||||
|
||||
box = np.concatenate((box_xy, box_wh), axis=-1) |
||||
|
||||
return box, box_confidence, box_class_probs |
||||
|
||||
|
||||
def filter_boxes(boxes, box_confidences, box_class_probs): |
||||
"""Filter boxes with box threshold. It's a bit different with origin yolov5 post process! |
||||
|
||||
# Arguments |
||||
boxes: ndarray, boxes of objects. |
||||
box_confidences: ndarray, confidences of objects. |
||||
box_class_probs: ndarray, class_probs of objects. |
||||
|
||||
# Returns |
||||
boxes: ndarray, filtered boxes. |
||||
classes: ndarray, classes for boxes. |
||||
scores: ndarray, scores for boxes. |
||||
""" |
||||
if not x1y1x2y2: |
||||
b1_x1, b1_x2 = box1[:, 0] - box1[:, 2] / 2, box1[:, 0] + box1[:, 2] / 2 |
||||
b1_y1, b1_y2 = box1[:, 1] - box1[:, 3] / 2, box1[:, 1] + box1[:, 3] / 2 |
||||
b2_x1, b2_x2 = box2[:, 0] - box2[:, 2] / 2, box2[:, 0] + box2[:, 2] / 2 |
||||
b2_y1, b2_y2 = box2[:, 1] - box2[:, 3] / 2, box2[:, 1] + box2[:, 3] / 2 |
||||
else: |
||||
b1_x1, b1_y1, b1_x2, b1_y2 = box1[:, 0], box1[:, 1], box1[:, 2], box1[:, 3] |
||||
b2_x1, b2_y1, b2_x2, b2_y2 = box2[:, 0], box2[:, 1], box2[:, 2], box2[:, 3] |
||||
|
||||
inter_rect_x1 = torch.max(b1_x1, b2_x1) |
||||
inter_rect_y1 = torch.max(b1_y1, b2_y1) |
||||
inter_rect_x2 = torch.min(b1_x2, b2_x2) |
||||
inter_rect_y2 = torch.min(b1_y2, b2_y2) |
||||
|
||||
inter_area = torch.clamp(inter_rect_x2 - inter_rect_x1 + 1, min=0) * \ |
||||
torch.clamp(inter_rect_y2 - inter_rect_y1 + 1, min=0) |
||||
|
||||
b1_area = (b1_x2 - b1_x1 + 1) * (b1_y2 - b1_y1 + 1) |
||||
b2_area = (b2_x2 - b2_x1 + 1) * (b2_y2 - b2_y1 + 1) |
||||
|
||||
iou = inter_area / (b1_area + b2_area - inter_area + 1e-16) |
||||
|
||||
return iou |
||||
|
||||
|
||||
def w_non_max_suppression(prediction, num_classes, conf_thres=0.1, nms_thres=0.4): |
||||
# 求左上角和右下角 |
||||
# box_corner = prediction.new(prediction.shape) |
||||
box_corner = torch.FloatTensor(prediction.shape) |
||||
box_corner[:, :, 0] = prediction[:, :, 0] - prediction[:, :, 2] / 2 |
||||
box_corner[:, :, 1] = prediction[:, :, 1] - prediction[:, :, 3] / 2 |
||||
box_corner[:, :, 2] = prediction[:, :, 0] + prediction[:, :, 2] / 2 |
||||
box_corner[:, :, 3] = prediction[:, :, 1] + prediction[:, :, 3] / 2 |
||||
prediction[:, :, :4] = box_corner[:, :, :4] |
||||
|
||||
output = [None for _ in range(len(prediction))] |
||||
for image_i, image_pred in enumerate(prediction): |
||||
# 利用置信度进行第一轮筛选 |
||||
conf_mask = (image_pred[:, 4] >= conf_thres).squeeze() |
||||
image_pred = image_pred[conf_mask] |
||||
|
||||
if not image_pred.size(0): |
||||
continue |
||||
|
||||
# 获得种类及其置信度 |
||||
class_conf, class_pred = torch.max(image_pred[:, 5:5 + num_classes], 1, keepdim=True) |
||||
|
||||
# 获得的内容为(x1, y1, x2, y2, obj_conf, class_conf, class_pred) |
||||
detections = torch.cat((image_pred[:, :5], class_conf.float(), class_pred.float()), 1) |
||||
|
||||
# 获得种类 |
||||
unique_labels = detections[:, -1].cpu().unique() |
||||
|
||||
if prediction.is_cuda: |
||||
unique_labels = unique_labels.cuda() |
||||
|
||||
for c in unique_labels: |
||||
# 获得某一类初步筛选后全部的预测结果 |
||||
detections_class = detections[detections[:, -1] == c] |
||||
# 按照存在物体的置信度排序 |
||||
_, conf_sort_index = torch.sort(detections_class[:, 4], descending=True) |
||||
detections_class = detections_class[conf_sort_index] |
||||
# 进行非极大抑制 |
||||
max_detections = [] |
||||
while detections_class.size(0): |
||||
# 取出这一类置信度最高的，一步一步往下判断，判断重合程度是否大于nms_thres，如果是则去除掉 |
||||
max_detections.append(detections_class[0].unsqueeze(0)) |
||||
if len(detections_class) == 1: |
||||
break |
||||
ious = w_bbox_iou(max_detections[-1], detections_class[1:]) |
||||
detections_class = detections_class[1:][ious < nms_thres] |
||||
# 堆叠 |
||||
max_detections = torch.cat(max_detections).data |
||||
# Add max detections to outputs |
||||
output[image_i] = max_detections if output[image_i] is None else torch.cat( |
||||
(output[image_i], max_detections)) |
||||
|
||||
return output |
||||
|
||||
|
||||
def onnx_postprocess(outputs, img_size_w, img_size_h): |
||||
boxs = [] |
||||
a = torch.tensor(anchors).float().view(3, -1, 2) |
||||
anchor_grid = a.clone().view(3, 1, -1, 1, 1, 2) |
||||
for index, out in enumerate(outputs): |
||||
out = torch.from_numpy(out) |
||||
batch = out.shape[1] |
||||
feature_h = out.shape[2] |
||||
feature_w = out.shape[3] |
||||
|
||||
# Feature map corresponds to the original image zoom factor |
||||
stride_w = int(img_size_w / feature_w) |
||||
stride_h = int(img_size_h / feature_h) |
||||
|
||||
grid_x, grid_y = np.meshgrid(np.arange(feature_w), np.arange(feature_h)) |
||||
grid_x, grid_y = torch.from_numpy(np.array(grid_x)).float(), torch.from_numpy(np.array(grid_y)).float() |
||||
|
||||
# cx, cy, w, h |
||||
pred_boxes = torch.FloatTensor(out[..., :4].shape) |
||||
pred_boxes[..., 0] = (torch.sigmoid(out[..., 0]) * 2.0 - 0.5 + grid_x) * stride_w # cx |
||||
pred_boxes[..., 1] = (torch.sigmoid(out[..., 1]) * 2.0 - 0.5 + grid_y) * stride_h # cy |
||||
pred_boxes[..., 2:4] = (torch.sigmoid(out[..., 2:4]) * 2) ** 2 * anchor_grid[index] # wh |
||||
pred_boxes_np = pred_boxes.numpy() |
||||
|
||||
conf = torch.sigmoid(out[..., 4]) |
||||
pred_cls = torch.sigmoid(out[..., 5:]) |
||||
|
||||
output = torch.cat((pred_boxes.view(1, -1, 4), |
||||
conf.view(1, -1, 1), |
||||
pred_cls.view(1, -1, NUM_CLS)), |
||||
-1) |
||||
boxs.append(output) |
||||
|
||||
outputx = torch.cat(boxs, 1) |
||||
# NMS |
||||
batch_detections = w_non_max_suppression(outputx, NUM_CLS, conf_thres=OBJ_THRESH, nms_thres=NMS_THRESH) |
||||
|
||||
return batch_detections |
||||
|
||||
|
||||
def clip_coords(boxes, img_shape): |
||||
# Clip bounding xyxy bounding boxes to image shape (height, width) |
||||
boxes[:, 0].clamp_(0, img_shape[1]) # x1 |
||||
boxes[:, 1].clamp_(0, img_shape[0]) # y1 |
||||
boxes[:, 2].clamp_(0, img_shape[1]) # x2 |
||||
boxes[:, 3].clamp_(0, img_shape[0]) # y2 |
||||
|
||||
|
||||
def scale_coords(img1_shape, coords, img0_shape, ratio_pad=None): |
||||
# Rescale coords (xyxy) from img1_shape to img0_shape |
||||
if ratio_pad is None: # calculate from img0_shape |
||||
gain = min(img1_shape[0]/img0_shape[0], img1_shape[1]/img0_shape[1]) # gain = old / new |
||||
pad = (img1_shape[1] - img0_shape[1] * gain) / 2, (img1_shape[0] - img0_shape[0] * gain) / 2 # wh padding |
||||
else: |
||||
gain = ratio_pad[0][0] |
||||
pad = ratio_pad[1] |
||||
|
||||
coords[:, [0, 2]] -= pad[0] # x padding |
||||
coords[:, [1, 3]] -= pad[1] # y padding |
||||
coords[:, :4] /= gain |
||||
clip_coords(coords, img0_shape) |
||||
return coords |
||||
|
||||
|
||||
def display(detections=None, image_src=None, input_size=(640, 640), line_thickness=None, text_bg_alpha=0.0): |
||||
labels = detections[..., -1] |
||||
boxs = detections[..., :4] |
||||
confs = detections[..., 4] |
||||
|
||||
h, w, c = image_src.shape |
||||
|
||||
boxs[:, :] = scale_coords(input_size, boxs[:, :], (h, w)).round() |
||||
|
||||
tl = line_thickness or round(0.002 * (w + h) / 2) + 1 |
||||
for i, box in enumerate(boxs): |
||||
x1, y1, x2, y2 = box |
||||
|
||||
ratio = (y2-y1)/(x2-x1) |
||||
|
||||
x1, y1, x2, y2 = int(x1.numpy()), int(y1.numpy()), int(x2.numpy()), int(y2.numpy()) |
||||
np.random.seed(int(labels[i].numpy()) + 2020) |
||||
color = (np.random.randint(0, 255), 0, np.random.randint(0, 255)) |
||||
cv2.rectangle(image_src, (x1, y1), (x2, y2), color, max(int((w + h) / 600), 1), cv2.LINE_AA) |
||||
label = '{0:.3f}'.format(confs[i]) |
||||
t_size = cv2.getTextSize(label, 0, fontScale=tl / 3, thickness=1)[0] |
||||
c2 = x1 + t_size[0] + 3, y1 - t_size[1] - 5 |
||||
if text_bg_alpha == 0.0: |
||||
cv2.rectangle(image_src, (x1 - 1, y1), c2, color, cv2.FILLED, cv2.LINE_AA) |
||||
else: |
||||
# 透明文本背景 |
||||
alphaReserve = text_bg_alpha # 0：不透明 1：透明 |
||||
BChannel, GChannel, RChannel = color |
||||
xMin, yMin = int(x1 - 1), int(y1 - t_size[1] - 3) |
||||
xMax, yMax = int(x1 + t_size[0]), int(y1) |
||||
image_src[yMin:yMax, xMin:xMax, 0] = image_src[yMin:yMax, xMin:xMax, 0] * alphaReserve + BChannel * (1 - alphaReserve) |
||||
image_src[yMin:yMax, xMin:xMax, 1] = image_src[yMin:yMax, xMin:xMax, 1] * alphaReserve + GChannel * (1 - alphaReserve) |
||||
image_src[yMin:yMax, xMin:xMax, 2] = image_src[yMin:yMax, xMin:xMax, 2] * alphaReserve + RChannel * (1 - alphaReserve) |
||||
cv2.putText(image_src, label, (x1 + 3, y1 - 4), 0, tl / 3, [255, 255, 255], |
||||
thickness=1, lineType=cv2.LINE_AA) |
||||
box_classes = np.argmax(box_class_probs, axis=-1) |
||||
box_class_scores = np.max(box_class_probs, axis=-1) |
||||
pos = np.where(box_confidences[..., 0] >= BOX_THESH) |
||||
|
||||
boxes = boxes[pos] |
||||
classes = box_classes[pos] |
||||
scores = box_class_scores[pos] |
||||
|
||||
return boxes, classes, scores |
||||
|
||||
if __name__ == '__main__': |
||||
|
||||
exp = 'yolov5s' |
||||
Width = 640 |
||||
Height = 640 |
||||
MODEL_PATH = './yolov5s.onnx' |
||||
im_file = './dog_bike_car_640x640.jpg' |
||||
RKNN_MODEL_PATH = './{}.rknn'.format(exp + '-' + str(Width) + '-' + str(Height)) |
||||
DATASET = './dataset.txt' |
||||
def nms_boxes(boxes, scores): |
||||
"""Suppress non-maximal boxes. |
||||
|
||||
# Arguments |
||||
boxes: ndarray, boxes of objects. |
||||
scores: ndarray, scores of objects. |
||||
|
||||
# Returns |
||||
keep: ndarray, index of effective boxes. |
||||
""" |
||||
x = boxes[:, 0] |
||||
y = boxes[:, 1] |
||||
w = boxes[:, 2] - boxes[:, 0] |
||||
h = boxes[:, 3] - boxes[:, 1] |
||||
|
||||
areas = w * h |
||||
order = scores.argsort()[::-1] |
||||
|
||||
keep = [] |
||||
while order.size > 0: |
||||
i = order[0] |
||||
keep.append(i) |
||||
|
||||
xx1 = np.maximum(x[i], x[order[1:]]) |
||||
yy1 = np.maximum(y[i], y[order[1:]]) |
||||
xx2 = np.minimum(x[i] + w[i], x[order[1:]] + w[order[1:]]) |
||||
yy2 = np.minimum(y[i] + h[i], y[order[1:]] + h[order[1:]]) |
||||
|
||||
w1 = np.maximum(0.0, xx2 - xx1 + 0.00001) |
||||
h1 = np.maximum(0.0, yy2 - yy1 + 0.00001) |
||||
inter = w1 * h1 |
||||
|
||||
ovr = inter / (areas[i] + areas[order[1:]] - inter) |
||||
inds = np.where(ovr <= NMS_THRESH)[0] |
||||
order = order[inds + 1] |
||||
keep = np.array(keep) |
||||
return keep |
||||
|
||||
|
||||
def yolov5_post_process(input_data): |
||||
masks = [[0, 1, 2], [3, 4, 5], [6, 7, 8]] |
||||
anchors = [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45], |
||||
[59, 119], [116, 90], [156, 198], [373, 326]] |
||||
|
||||
boxes, classes, scores = [], [], [] |
||||
for input, mask in zip(input_data, masks): |
||||
b, c, s = process(input, mask, anchors) |
||||
b, c, s = filter_boxes(b, c, s) |
||||
boxes.append(b) |
||||
classes.append(c) |
||||
scores.append(s) |
||||
|
||||
boxes = np.concatenate(boxes) |
||||
boxes = xywh2xyxy(boxes) |
||||
classes = np.concatenate(classes) |
||||
scores = np.concatenate(scores) |
||||
|
||||
nboxes, nclasses, nscores = [], [], [] |
||||
for c in set(classes): |
||||
inds = np.where(classes == c) |
||||
b = boxes[inds] |
||||
c = classes[inds] |
||||
s = scores[inds] |
||||
|
||||
keep = nms_boxes(b, s) |
||||
|
||||
nboxes.append(b[keep]) |
||||
nclasses.append(c[keep]) |
||||
nscores.append(s[keep]) |
||||
|
||||
if not nclasses and not nscores: |
||||
return None, None, None |
||||
|
||||
boxes = np.concatenate(nboxes) |
||||
classes = np.concatenate(nclasses) |
||||
scores = np.concatenate(nscores) |
||||
|
||||
return boxes, classes, scores |
||||
|
||||
|
||||
def draw(image, boxes, scores, classes): |
||||
"""Draw the boxes on the image. |
||||
|
||||
# Argument: |
||||
image: original image. |
||||
boxes: ndarray, boxes of objects. |
||||
classes: ndarray, classes of objects. |
||||
scores: ndarray, scores of objects. |
||||
all_classes: all classes name. |
||||
""" |
||||
for box, score, cl in zip(boxes, scores, classes): |
||||
top, left, right, bottom = box |
||||
print('class: {}, score: {}'.format(CLASSES[cl], score)) |
||||
print('box coordinate left,top,right,down: [{}, {}, {}, {}]'.format(top, left, right, bottom)) |
||||
top = int(top) |
||||
left = int(left) |
||||
right = int(right) |
||||
bottom = int(bottom) |
||||
|
||||
cv2.rectangle(image, (top, left), (right, bottom), (255, 0, 0), 2) |
||||
cv2.putText(image, '{0} {1:.2f}'.format(CLASSES[cl], score), |
||||
(top, left - 6), |
||||
cv2.FONT_HERSHEY_SIMPLEX, |
||||
0.6, (0, 0, 255), 2) |
||||
|
||||
|
||||
def letterbox(im, new_shape=(640, 640), color=(0, 0, 0)): |
||||
# Resize and pad image while meeting stride-multiple constraints |
||||
shape = im.shape[:2] # current shape [height, width] |
||||
if isinstance(new_shape, int): |
||||
new_shape = (new_shape, new_shape) |
||||
|
||||
# Scale ratio (new / old) |
||||
r = min(new_shape[0] / shape[0], new_shape[1] / shape[1]) |
||||
|
||||
# Compute padding |
||||
ratio = r, r # width, height ratios |
||||
new_unpad = int(round(shape[1] * r)), int(round(shape[0] * r)) |
||||
dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1] # wh padding |
||||
|
||||
dw /= 2 # divide padding into 2 sides |
||||
dh /= 2 |
||||
|
||||
if shape[::-1] != new_unpad: # resize |
||||
im = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR) |
||||
top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1)) |
||||
left, right = int(round(dw - 0.1)), int(round(dw + 0.1)) |
||||
im = cv2.copyMakeBorder(im, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color) # add border |
||||
return im, ratio, (dw, dh) |
||||
|
||||
|
||||
if __name__ == '__main__': |
||||
|
||||
# Create RKNN object |
||||
rknn = RKNN(verbose=True) |
||||
|
||||
# pre-process config |
||||
print('--> Config model') |
||||
rknn.config(mean_values=[[0, 0, 0]], std_values=[[255, 255, 255]], output_tensor_type='int8') |
||||
print('done') |
||||
|
||||
rknn.config(mean_values=[[0, 0, 0]], std_values=[[255, 255, 255]]) |
||||
# Load model |
||||
# Load ONNX model |
||||
print('--> Loading model') |
||||
ret = rknn.load_onnx(MODEL_PATH) |
||||
ret = rknn.load_onnx(model=ONNX_MODEL, outputs=['378', '439', '500']) |
||||
if ret != 0: |
||||
print('load model failed!') |
||||
print('Load model failed!') |
||||
exit(ret) |
||||
print('done') |
||||
|
||||
# Build model |
||||
print('--> Building model') |
||||
ret = rknn.build(do_quantization=True, dataset=DATASET) |
||||
ret = rknn.build(do_quantization=QUANTIZE_ON, dataset=DATASET) |
||||
if ret != 0: |
||||
print('build model failed.') |
||||
print('Build model failed!') |
||||
exit(ret) |
||||
print('done') |
||||
|
||||
# Export rknn model |
||||
print('--> Export RKNN model') |
||||
ret = rknn.export_rknn(RKNN_MODEL_PATH) |
||||
# Export RKNN model |
||||
print('--> Export rknn model') |
||||
ret = rknn.export_rknn(RKNN_MODEL) |
||||
if ret != 0: |
||||
print('Export rknn model failed.') |
||||
print('Export rknn model failed!') |
||||
exit(ret) |
||||
print('done') |
||||
|
||||
# Set inputs |
||||
image_src = Image.open('./dog_bike_car_640x640.jpg') |
||||
img = letterbox_image(image_src, (Width, Height)) |
||||
img = np.array(img) |
||||
|
||||
# init runtime environment |
||||
# Init runtime environment |
||||
print('--> Init runtime environment') |
||||
ret = rknn.init_runtime() |
||||
# ret = rknn.init_runtime('rk3566') |
||||
if ret != 0: |
||||
print('Init runtime environment failed') |
||||
print('Init runtime environment failed!') |
||||
exit(ret) |
||||
print('done') |
||||
|
||||
# inference |
||||
print('--> inference') |
||||
start = time.time() |
||||
outputs = rknn.inference(inputs=[img]) |
||||
end = time.time() |
||||
print('inference time: ', end - start) |
||||
print('done') |
||||
# Set inputs |
||||
img = cv2.imread(IMG_PATH) |
||||
# img, ratio, (dw, dh) = letterbox(img, new_shape=(IMG_SIZE, IMG_SIZE)) |
||||
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) |
||||
img = cv2.resize(img, (IMG_SIZE, IMG_SIZE)) |
||||
|
||||
# Inference |
||||
print('--> Running model') |
||||
outputs = rknn.inference(inputs=[img]) |
||||
np.save('./onnx_yolov5_0.npy', outputs[0]) |
||||
np.save('./onnx_yolov5_1.npy', outputs[1]) |
||||
np.save('./onnx_yolov5_2.npy', outputs[2]) |
||||
print('done') |
||||
|
||||
# inference process |
||||
image_src = np.array(image_src) |
||||
detections = onnx_postprocess(outputs, Width, Height) |
||||
if detections[0] is not None: |
||||
display(detections[0], image_src) |
||||
# post process |
||||
input0_data = outputs[0] |
||||
input1_data = outputs[1] |
||||
input2_data = outputs[2] |
||||
|
||||
image_src = cv2.cvtColor(image_src,cv2.COLOR_BGR2RGB) |
||||
input0_data = input0_data.reshape([3, -1]+list(input0_data.shape[-2:])) |
||||
input1_data = input1_data.reshape([3, -1]+list(input1_data.shape[-2:])) |
||||
input2_data = input2_data.reshape([3, -1]+list(input2_data.shape[-2:])) |
||||
|
||||
cv2.imwrite("result.jpg", image_src) |
||||
input_data = list() |
||||
input_data.append(np.transpose(input0_data, (2, 3, 0, 1))) |
||||
input_data.append(np.transpose(input1_data, (2, 3, 0, 1))) |
||||
input_data.append(np.transpose(input2_data, (2, 3, 0, 1))) |
||||
|
||||
rknn.release() |
||||
boxes, classes, scores = yolov5_post_process(input_data) |
||||
|
||||
img_1 = cv2.cvtColor(img, cv2.COLOR_RGB2BGR) |
||||
if boxes is not None: |
||||
draw(img_1, boxes, scores, classes) |
||||
# show output |
||||
# cv2.imshow("post process result", img_1) |
||||
# cv2.waitKey(0) |
||||
# cv2.destroyAllWindows() |
||||
|
||||
rknn.release() |
||||
|
||||
Binary file not shown.
@ -0,0 +1,86 @@
@@ -0,0 +1,86 @@
|
||||
import torch |
||||
import torch.nn as nn |
||||
from torch.nn import functional as F |
||||
|
||||
|
||||
class RestNetBasicBlock(nn.Module): |
||||
def __init__(self, in_channels, out_channels, stride): |
||||
super(RestNetBasicBlock, self).__init__() |
||||
self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=stride, padding=1) |
||||
self.bn1 = nn.BatchNorm2d(out_channels) |
||||
self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=stride, padding=1) |
||||
self.bn2 = nn.BatchNorm2d(out_channels) |
||||
|
||||
def forward(self, x): |
||||
output = self.conv1(x) |
||||
output = F.relu(self.bn1(output)) |
||||
output = self.conv2(output) |
||||
output = self.bn2(output) |
||||
return F.relu(x + output) |
||||
|
||||
|
||||
class RestNetDownBlock(nn.Module): |
||||
def __init__(self, in_channels, out_channels, stride): |
||||
super(RestNetDownBlock, self).__init__() |
||||
self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=stride[0], padding=1) |
||||
self.bn1 = nn.BatchNorm2d(out_channels) |
||||
self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=stride[1], padding=1) |
||||
self.bn2 = nn.BatchNorm2d(out_channels) |
||||
self.extra = nn.Sequential( |
||||
nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=stride[0], padding=0), |
||||
nn.BatchNorm2d(out_channels) |
||||
) |
||||
|
||||
def forward(self, x): |
||||
extra_x = self.extra(x) |
||||
output = self.conv1(x) |
||||
out = F.relu(self.bn1(output)) |
||||
|
||||
out = self.conv2(out) |
||||
out = self.bn2(out) |
||||
return F.relu(extra_x + out) |
||||
|
||||
|
||||
class RestNet18(nn.Module): |
||||
def __init__(self): |
||||
super(RestNet18, self).__init__() |
||||
self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3) |
||||
self.bn1 = nn.BatchNorm2d(64) |
||||
self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) |
||||
|
||||
self.layer1 = nn.Sequential(RestNetBasicBlock(64, 64, 1), |
||||
RestNetBasicBlock(64, 64, 1)) |
||||
|
||||
self.layer2 = nn.Sequential(RestNetDownBlock(64, 128, [2, 1]), |
||||
RestNetBasicBlock(128, 128, 1)) |
||||
|
||||
self.layer3 = nn.Sequential(RestNetDownBlock(128, 256, [2, 1]), |
||||
RestNetBasicBlock(256, 256, 1)) |
||||
|
||||
self.layer4 = nn.Sequential(RestNetDownBlock(256, 512, [2, 1]), |
||||
RestNetBasicBlock(512, 512, 1)) |
||||
|
||||
self.avgpool = nn.AdaptiveAvgPool2d(output_size=(1, 1)) |
||||
|
||||
self.fc = nn.Linear(512, 10) |
||||
|
||||
def forward(self, x): |
||||
out = self.conv1(x) |
||||
out = self.layer1(out) |
||||
out = self.layer2(out) |
||||
out = self.layer3(out) |
||||
out = self.layer4(out) |
||||
out = self.avgpool(out) |
||||
out = out.reshape(x.shape[0], -1) |
||||
out = self.fc(out) |
||||
return out |
||||
|
||||
|
||||
if __name__ == '__main__': |
||||
# build model |
||||
model = RestNet18() |
||||
model.eval() |
||||
|
||||
# export onnx (rknn-toolkit2 only support opset_version=12) |
||||
x = torch.randn((1, 3, 224, 224)) |
||||
torch.onnx.export(model, x, './resnet18.onnx', opset_version=12, input_names=['input'], output_names=['output']) |
||||
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
@ -0,0 +1,103 @@
@@ -0,0 +1,103 @@
|
||||
import numpy as np |
||||
import cv2 |
||||
from rknn.api import RKNN |
||||
import torchvision.models as models |
||||
import torch |
||||
import os |
||||
|
||||
|
||||
def show_outputs(output): |
||||
output_sorted = sorted(output, reverse=True) |
||||
top5_str = '\n-----TOP 5-----\n' |
||||
for i in range(5): |
||||
value = output_sorted[i] |
||||
index = np.where(output == value) |
||||
for j in range(len(index)): |
||||
if (i + j) >= 5: |
||||
break |
||||
if value > 0: |
||||
topi = '{}: {}\n'.format(index[j], value) |
||||
else: |
||||
topi = '-1: 0.0\n' |
||||
top5_str += topi |
||||
print(top5_str) |
||||
|
||||
|
||||
def show_perfs(perfs): |
||||
perfs = 'perfs: {}\n'.format(perfs) |
||||
print(perfs) |
||||
|
||||
|
||||
def softmax(x): |
||||
return np.exp(x)/sum(np.exp(x)) |
||||
|
||||
def torch_version(): |
||||
import torch |
||||
torch_ver = torch.__version__.split('.') |
||||
torch_ver[2] = torch_ver[2].split('+')[0] |
||||
return [int(v) for v in torch_ver] |
||||
|
||||
if __name__ == '__main__': |
||||
|
||||
if torch_version() < [1, 9, 0]: |
||||
import torch |
||||
print("Your torch version is '{}', in order to better support the Quantization Aware Training (QAT) model,\n" |
||||
"Please update the torch version to '1.9.0' or higher!".format(torch.__version__)) |
||||
exit(0) |
||||
|
||||
model = './resnet18_i8.pt' |
||||
|
||||
input_size_list = [[1, 3, 224, 224]] |
||||
|
||||
# Create RKNN object |
||||
rknn = RKNN(verbose=True) |
||||
|
||||
# Pre-process config |
||||
print('--> Config model') |
||||
rknn.config(mean_values=[123.675, 116.28, 103.53], std_values=[58.395, 58.395, 58.395]) |
||||
print('done') |
||||
|
||||
# Load model |
||||
print('--> Loading model') |
||||
ret = rknn.load_pytorch(model=model, input_size_list=input_size_list) |
||||
if ret != 0: |
||||
print('Load model failed!') |
||||
exit(ret) |
||||
print('done') |
||||
|
||||
# Build model |
||||
print('--> Building model') |
||||
ret = rknn.build(do_quantization=False) |
||||
if ret != 0: |
||||
print('Build model failed!') |
||||
exit(ret) |
||||
print('done') |
||||
|
||||
# Export rknn model |
||||
print('--> Export rknn model') |
||||
ret = rknn.export_rknn('./resnet_18.rknn') |
||||
if ret != 0: |
||||
print('Export rknn model failed!') |
||||
exit(ret) |
||||
print('done') |
||||
|
||||
# Set inputs |
||||
img = cv2.imread('./space_shuttle_224.jpg') |
||||
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) |
||||
|
||||
# Init runtime environment |
||||
print('--> Init runtime environment') |
||||
ret = rknn.init_runtime() |
||||
if ret != 0: |
||||
print('Init runtime environment failed!') |
||||
exit(ret) |
||||
print('done') |
||||
|
||||
# Inference |
||||
print('--> Running model') |
||||
outputs = rknn.inference(inputs=[img]) |
||||
np.save('./pytorch_resnet18_qat_0.npy', outputs[0]) |
||||
show_outputs(softmax(np.array(outputs[0][0]))) |
||||
print('done') |
||||
|
||||
rknn.release() |
||||
|
Before Width: | Height: | Size: 85 KiB After Width: | Height: | Size: 85 KiB |
|
After Width: | Height: | Size: 18 KiB |
Loading…
Reference in new issue