license_plate_recognition

License Plate Recognition with od-bridge

Table of Contents

• About
• Requirements (server-side only)
• Installation
• Usage
  • Server
  • Client

About

This is a gRPC server which accepts image and can make license plate recognition (using YOLOv3 or YOLOv4 neural network).

Neural networks were trained on dataset of russian license plates. But you can train it on another dataset - read about process here https://github.com/AlexeyAB/darknet#how-to-train-to-detect-your-custom-objects

Server tries to find license plate at first. Then it does OCR (if it's possible) and provides output which can be represented later as follows:

Sample plate #1	Sample plate #2

Images has been taken near my house

Darknet architecture for finding license plates - Yolo V3

Darknet architecture for doing OCR stuff - Yolo V4

gRPC server accepts this data struct according proto3 specification

// Essential information to process
message LPRRequest{
    // Bytes representation of image (PNG)
    bytes image = 1;
    // Optional information about image. Could be usefull if client-side already knows where license plate should be located (due some object detections technique)
    BBox bbox = 2;
}

// Reference information about detection rectangle
message BBox{
    int32 x_left = 1;
    int32 y_top = 2;
    int32 height = 3;
    int32 width = 4;
}

The gRPC server response is:

// Response from server
message LPRResponse{
    // Set of found license plates with corresponding information
    repeated LPRInfo license_plates = 1;
    // Number of seconds has taken to proccess license plate detections and OCR
    float elapsed = 2;
    // Optional message from server
    string message = 3;
    // Optional warning message from server. If it is not empty you probably should investiage such behavior
    string warning = 4;
    // Optional error message from server. If it is not empty you should investiage the error
    string error = 5;
}

// Information about single license plate
message LPRInfo {
    // License plate location
    BBox bbox = 1;
    // License plate OCR bounding bboxes. Bounding bboxes are sorted by horizontal line
    // Warning: those coordinates are relative to license plate bounding box, not the parent image!
    repeated BBox ocr_bboxes = 2;
    // License plate text
    string text = 3;
}

Full gRPC documentation is here: HTML or Markdown

Requirements (server-side only)

The gRPC client does not require any native dependencies.

od-bridge (Rust inference backend)

The server uses od-bridge for ONNX model inference via CGO. See the od-bridge installation guide for build and install instructions.

# 1. Clone and build
git clone https://github.com/LdDl/od-bridge.git
cd od-bridge
cargo build --release                  # CPU only
# cargo build --release --features cuda  # with CUDA

# 2. Install shared library and header
sudo mkdir -p /usr/local/include/od-bridge
sudo cp od_bridge.h /usr/local/include/od-bridge/
sudo cp target/release/libod_bridge.so /usr/local/lib/

# 3. If built with --features cuda, also install ORT provider libraries:
# sudo cp target/release/libonnxruntime_providers_cuda.so /usr/local/lib/
# sudo cp target/release/libonnxruntime_providers_shared.so /usr/local/lib/

sudo ldconfig

Verify the installation:

ldconfig -p | grep od_bridge

See the od-bridge README for details.

Neural network weights

Models must be in ONNX format. If the pre-trained weights suit your needs, you can download them from the latest release:

mkdir -p data && cd data
curl -sLO https://github.com/LdDl/license_plate_recognition/releases/download/latest/license_plates.onnx
curl -sLO https://github.com/LdDl/license_plate_recognition/releases/download/latest/license_plates.names
curl -sLO https://github.com/LdDl/license_plate_recognition/releases/download/latest/ocr_plates.onnx
curl -sLO https://github.com/LdDl/license_plate_recognition/releases/download/latest/ocr_plates.names

If you want to convert darknet .cfg + .weights to ONNX yourself, download the source weights and use darknet2onnx:

mkdir -p data && cd data
curl -sLO https://github.com/LdDl/license_plate_recognition/releases/download/latest/license_plates_inference.cfg
curl -sLO https://github.com/LdDl/license_plate_recognition/releases/download/latest/license_plates_100000.weights
curl -sLO https://github.com/LdDl/license_plate_recognition/releases/download/latest/ocr_plates_inference.cfg
curl -sLO https://github.com/LdDl/license_plate_recognition/releases/download/latest/ocr_plates_140000.weights

darknet2onnx --cfg license_plates_inference.cfg --weights license_plates_100000.weights --output license_plates.onnx --format yolov8
darknet2onnx --cfg ocr_plates_inference.cfg --weights ocr_plates_140000.weights --output ocr_plates.onnx --format yolov8

Installation

Just pull source code:

go get github.com/LdDl/license_plate_recognition

Usage

Start server

• Navigate to folder with server application source code

  cd cmd/server
  

• Build source code of server application to executable

  go build -o recognition_server main.go
  

• Run server application

  ./recognition_server --cfg conf.toml
  

  Note: Please see conf.toml description for correct usage

• On server's side the directory './detected' will appear if you provide save_detected = true in TOML configuration. Detected license plates with OCR annotations will be stored there.

Test Client-Server

Notice: server should be started

• Navigate to folder with server application source code

  cd cmd/client
  

• Build source code of client application to executable

  go build -o client_app main.go
  

• Run client application

  ./client_app --host=localhost --port=50051 --file=sample.jpg -x 0 -y 0 --width=4032 --height=3024
  

• Check, if server can handle error (like negative height parameter):

  ./client_app --host=localhost --port=50051 --file=sample.jpg -x 0 -y 0 --width=42 --height=-24
  

• On client's side there will be output something like this:

  Elapsed seconds: 0.17216872
  Detections num: 2
  Detection #0:
          Text: A100CX777
          Plate bbox: x_left:2027 y_top:2027 height:304 width:646
          OCR bboxes: [x_left:109 y_top:109 height:71 width:71 x_left:186 y_top:186 height:77 width:59 x_left:252 y_top:252 height:79 width:66 x_left:323 y_top:323 height:80 width:73 x_left:404 y_top:404 height:77 width:77 x_left:485 y_top:485 height:70 width:83 x_left:584 y_top:584 height:72 width:71 x_left:657 y_top:657 height:76 width:68 x_left:731 y_top:731 height:76 width:66]
  Detection #1:
          Text: M288HO199
          Plate bbox: x_left:262 y_top:262 height:186 width:391
          OCR bboxes: [x_left:56 y_top:56 height:28 width:25 x_left:86 y_top:86 height:36 width:24 x_left:112 y_top:112 height:36 width:25 x_left:139 y_top:139 height:39 width:27 x_left:168 y_top:168 height:28 width:27 x_left:196 y_top:196 height:28 width:31 x_left:232 y_top:232 height:30 width:21 x_left:253 y_top:253 height:31 width:21 x_left:276 y_top:276 height:30 width:23]