Fast Inference for Immich ML OCR with PaddleX

This guide provides step-by-step instructions to set up PaddleX for fast OCR inference on GPU for Immich ML. By following these steps, you can significantly improve the performance of OCR tasks in your Immich setup. ~1s inference time per image with 10 concurrent requests on RTX 3080ti 12GB GPU with PaddleX vs 80s+ with Onnxruntime GPU execution provider. All code snippets and configuration files mentioned in this guide can be found in this GitHub repository. Context Note: Test with Immich Version: 2.3.1 Immich ML’s inference engine uses Onnxruntime as the default backend for model inference. However, for OCR tasks which were introduced in Immich 2.3.x, users have reported very slow GPU performance sometimes even slower than CPU inference (see Reddit discussion, GitHub Issue #23462) even when using a powerful GPU or the mobilde version of the OCR model.