DEV Community

Xiao Ling
Xiao Ling

Posted on • Edited on • Originally published at dynamsoft.com

How to Create a Flutter plugin of Passport MRZ Recognition for Windows, Linux, Android, iOS and Web

#flutter #mrz #ocr #android

Since Dynamsoft Label Recognizer supports Windows, Linux, Android, iOS and Web, we can create a 5 in 1 Flutter plugin for it to cover desktop, mobile and web platform development. In this article, you will see how to write an MRZ Flutter plugin using different platform-specific code, such as Java, Swift, Dart, C++ and JavaScript.

Download flutter_ocr_sdk from Pub.dev

https://pub.dev/packages/flutter_ocr_sdk

Initialize Flutter MRZ Detection Plugin for Multiple Platforms

We scaffold a Flutter plugin project for multiple platforms by running the following command:

flutter create --org com.dynamsoft --template=plugin --platforms=android,ios,windows,linux,web -a java flutter_ocr_sdk
Enter fullscreen mode Exit fullscreen mode

MRZ Model

To implement the package, the first step is to download the Dynamsoft Label Recognizer SDK and add it to the project. From https://www.dynamsoft.com/label-recognition/downloads, you can get shared libraries and model files.

The MRZ model consists of four files: MRZ.caffemodel, MRZ.json, MRZ.prototxt, and MRZ.txt. We put them in the lib/model folder of the Flutter project and add the asset path to pubspec.yaml:

assets:
    - lib/model/
Enter fullscreen mode Exit fullscreen mode

Linking MRZ OCR Libraries

The ways of linking third-party libraries in Flutter plugin project are different for multiple platforms.

Windows and Linux

Both Windows and Linux use CMake to build the project. We copy header files and shared libraries to the target platform folder and then configure the CMakeLists.txt file.

Windows


cmake_minimum_required(VERSION 3.14)

set(PROJECT_NAME "flutter_ocr_sdk")
project(${PROJECT_NAME} LANGUAGES CXX)

set(PLUGIN_NAME "flutter_ocr_sdk_plugin")

link_directories("${PROJECT_SOURCE_DIR}/lib/") 

list(APPEND PLUGIN_SOURCES
  "flutter_ocr_sdk_plugin.cpp"
  "flutter_ocr_sdk_plugin.h"
)

add_library(${PLUGIN_NAME} SHARED
  "include/flutter_ocr_sdk/flutter_ocr_sdk_plugin_c_api.h"
  "flutter_ocr_sdk_plugin_c_api.cpp"
  ${PLUGIN_SOURCES}
)

apply_standard_settings(${PLUGIN_NAME})

set_target_properties(${PLUGIN_NAME} PROPERTIES
  CXX_VISIBILITY_PRESET hidden)
target_compile_definitions(${PLUGIN_NAME} PRIVATE FLUTTER_PLUGIN_IMPL)

target_include_directories(${PLUGIN_NAME} INTERFACE
  "${CMAKE_CURRENT_SOURCE_DIR}/include")
target_link_libraries(${PLUGIN_NAME} PRIVATE flutter flutter_wrapper_plugin "DynamsoftLabelRecognizerx64")

set(flutter_ocr_sdk_bundled_libraries
  "${PROJECT_SOURCE_DIR}/bin/"
  PARENT_SCOPE
)
Enter fullscreen mode Exit fullscreen mode

Linux

cmake_minimum_required(VERSION 3.10)

set(PROJECT_NAME "flutter_ocr_sdk")
project(${PROJECT_NAME} LANGUAGES CXX)

set(PLUGIN_NAME "flutter_ocr_sdk_plugin")

link_directories("${PROJECT_SOURCE_DIR}/lib/") 

add_library(${PLUGIN_NAME} SHARED
  "flutter_ocr_sdk_plugin.cc"
)

apply_standard_settings(${PLUGIN_NAME})

set_target_properties(${PLUGIN_NAME} PROPERTIES
  CXX_VISIBILITY_PRESET hidden)
target_compile_definitions(${PLUGIN_NAME} PRIVATE FLUTTER_PLUGIN_IMPL)

target_include_directories(${PLUGIN_NAME} INTERFACE
  "${CMAKE_CURRENT_SOURCE_DIR}/include")
target_link_libraries(${PLUGIN_NAME} PRIVATE flutter "DynamsoftLabelRecognizer")
target_link_libraries(${PLUGIN_NAME} PRIVATE PkgConfig::GTK)

set(flutter_ocr_sdk_bundled_libraries
  ""
  PARENT_SCOPE
)
Enter fullscreen mode Exit fullscreen mode

Android and iOS

Gradle and CocoaPods are used to build Android and iOS projects. The minimum supported version of Android is 21 and the minimum supported version of iOS is 9.

Configure build.gradle for Android

rootProject.allprojects {
    repositories {
        maven {
            url "https://download2.dynamsoft.com/maven/aar"
        }
        google()
        mavenCentral()
    }
}

apply plugin: 'com.android.library'

android {
    compileSdkVersion 31

    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }

    defaultConfig {
        minSdkVersion 21
    }
}

dependencies {
    implementation "com.dynamsoft:dynamsoftlabelrecognizer:2.2.20"
}
Enter fullscreen mode Exit fullscreen mode

Configure flutter_ocr_sdk.podspec for iOS

Pod::Spec.new do |s|
  s.name             = 'flutter_ocr_sdk'
  s.version          = '1.0.0'
  s.summary          = 'A wrapper for Dynamsoft OCR SDK, detecting MRZ in passports, travel documents, and ID cards.'
  s.description      = <<-DESC
A wrapper for Dynamsoft OCR SDK, detecting MRZ in passports, travel documents, and ID cards.
                       DESC
  s.homepage         = 'https://github.com/yushulx/flutter_ocr_sdk'
  s.license          = { :file => '../LICENSE' }
  s.author           = { 'yushulx' => 'lingxiao1002@gmail.com' }
  s.source           = { :path => '.' }
  s.source_files = 'Classes/**/*'
  s.dependency 'Flutter'
  s.platform = :ios, '9.0'
  s.dependency 'DynamsoftLabelRecognizer', '2.2.20'
  # Flutter.framework does not contain a i386 slice.
  s.pod_target_xcconfig = { 'DEFINES_MODULE' => 'YES', 'EXCLUDED_ARCHS[sdk=iphonesimulator*]' => 'i386' }
  s.swift_version = '5.0'
end
Enter fullscreen mode Exit fullscreen mode

Web

When using the plugin for web, developers need to include the JavaScript library of Dynamsoft Label Recognizer in their index.html file:

<script src="https://cdn.jsdelivr.net/npm/dynamsoft-label-recognizer@2.2.11/dist/dlr.js"></script>
Enter fullscreen mode Exit fullscreen mode

In the next section, we will write the shared Dart code for all platforms.

Dart Code for MRZ Recognition

The flutter_ocr_sdk_platform_interface.dart defines the methods to be implemented by each platform.

abstract class FlutterOcrSdkPlatform extends PlatformInterface {
  FlutterOcrSdkPlatform() : super(token: _token);

  static final Object _token = Object();

  static FlutterOcrSdkPlatform _instance = MethodChannelFlutterOcrSdk();

  static FlutterOcrSdkPlatform get instance => _instance;

  static set instance(FlutterOcrSdkPlatform instance) {
    PlatformInterface.verifyToken(instance, _token);
    _instance = instance;
  }

  Future<int?> init(String path, String key) {
    throw UnimplementedError('init() has not been implemented.');
  }

  Future<List<List<MrzLine>>?> recognizeByBuffer(
      Uint8List bytes, int width, int height, int stride, int format) {
    throw UnimplementedError('recognizeByBuffer() has not been implemented.');
  }

  Future<List<List<MrzLine>>?> recognizeByFile(String filename) {
    throw UnimplementedError('recognizeByFile() has not been implemented.');
  }

  Future<int?> loadModel() async {
    throw UnimplementedError('loadModel() has not been implemented.');
  }
}
Enter fullscreen mode Exit fullscreen mode
  • init(): initialize the OCR SDK with the SDK path (Web Only) and the license key of Dynamsoft Label Recognizer.
  • loadModel(): load the MRZ recognition model.
  • recognizeByBuffer(): recognize the MRZ from the image buffer.
  • recognizeByFile(): recognize the MRZ from the image file.

The flutter_ocr_sdk_web.dart file implements the methods for web, and the flutter_ocr_sdk_method_channel.dart file implements the methods for Android, iOS, Windows and Linux.

MRZ Detection API for Web

  1. In the web_dlr_manager.dart file, declare the JavaScript properties and functions to be called:

    @JS('Dynamsoft')
library dynamsoft;

import 'dart:convert';
import 'dart:typed_data';

import 'package:js/js.dart';

import 'mrz_line.dart';
import 'utils.dart';

/// DocumentNormalizer class.
@JS('DLR.LabelRecognizer')
class LabelRecognizer {
  external static set license(String license);
  external static set engineResourcePath(String resourcePath);
  external static PromiseJsImpl<LabelRecognizer> createInstance();
  external PromiseJsImpl<void> updateRuntimeSettingsFromString(String settings);
  external PromiseJsImpl<List<dynamic>> recognize(dynamic source);
  external PromiseJsImpl<List<dynamic>> recognizeBuffer(
      Uint8List buffer, int width, int height, int stride, int format);
}

  2. Create a DLRManager class to interoperate with the JavaScript library:

    class DLRManager {
  LabelRecognizer? _recognizer;

  Future<int> init(String path, String key) async {
    LabelRecognizer.engineResourcePath = path;
    LabelRecognizer.license = key;

    _recognizer = await handleThenable(LabelRecognizer.createInstance());

    return 0;
  }

  Future<List<List<MrzLine>>?> recognizeByFile(String file) async {
    if (_recognizer != null) {
      List<dynamic> results =
          await handleThenable(_recognizer!.recognize(file));
      return _resultWrapper(results);
    }

    return [];
  }

  Future<List<List<MrzLine>>?> recognizeByBuffer(
      Uint8List bytes, int width, int height, int stride, int format) async {
    if (_recognizer != null) {
      List<dynamic> results = await handleThenable(
          _recognizer!.recognizeBuffer(bytes, width, height, stride, format));
      return _resultWrapper(results);
    }

    return [];
  }

  Future<int?> loadModel() async {
    if (_recognizer != null) {
      await handleThenable(_recognizer!.updateRuntimeSettingsFromString("MRZ"));
    }
    return 0;
  }

  List<List<MrzLine>> _resultWrapper(List<dynamic> results) {
    List<List<MrzLine>> output = [];

    for (dynamic result in results) {
      Map value = json.decode(stringify(result));

      List<dynamic> area = value['lineResults'];
      List<MrzLine> lines = [];
      if (area.length == 2 || area.length == 3) {
        for (int i = 0; i < area.length; i++) {
          MrzLine line = MrzLine();
          line.text = area[i]['text'];
          line.x1 = area[i]['location']['points'][0]['x'];
          line.y1 = area[i]['location']['points'][0]['y'];
          line.x2 = area[i]['location']['points'][1]['x'];
          line.y2 = area[i]['location']['points'][1]['y'];
          line.x3 = area[i]['location']['points'][2]['x'];
          line.y3 = area[i]['location']['points'][2]['y'];
          line.x4 = area[i]['location']['points'][3]['x'];
          line.y4 = area[i]['location']['points'][3]['y'];
          lines.add(line);
        }
      }
      output.add(lines);
    }

    return output;
  }
}

MRZ Detection API for Mobile and Desktop

Comparing to the web, the implementation of loading MRZ model for mobile and desktop is a little bit complicated, because the native APIs are different between mobile and desktop. Thus, the platform check is required in the flutter_ocr_sdk_method_channel.dart file.

Future<int?> loadModel() async {
  final directory = await getApplicationDocumentsDirectory();

  String modelPath = 'packages/flutter_ocr_sdk/lib/model/';

  bool isDesktop = false;
  if (Platform.isWindows || Platform.isLinux) {
    isDesktop = true;
  }

  int? ret = 0;
  var fileNames = ["MRZ"];
  for (var i = 0; i < fileNames.length; i++) {
    var fileName = fileNames[i];

    var prototxtName = '$fileName.prototxt';
    var prototxtBufferPath = join(modelPath, prototxtName);
    ByteData prototxtBuffer = await loadAssetBytes(prototxtBufferPath);

    var txtBufferName = '$fileName.txt';
    var txtBufferPath = join(modelPath, txtBufferName);
    ByteData txtBuffer = await loadAssetBytes(txtBufferPath);

    var characterModelName = '$fileName.caffemodel';
    var characterModelBufferPath = join(modelPath, characterModelName);
    ByteData characterModelBuffer =
        await loadAssetBytes(characterModelBufferPath);

    if (isDesktop) {
      List<int> bytes = prototxtBuffer.buffer.asUint8List();
      await File(join(directory.path, prototxtName)).writeAsBytes(bytes);

      bytes = txtBuffer.buffer.asUint8List();
      await File(join(directory.path, txtBufferName)).writeAsBytes(bytes);

      bytes = characterModelBuffer.buffer.asUint8List();
      await File(join(directory.path, characterModelName))
          .writeAsBytes(bytes);
    } else {
      loadModelFiles(
          fileName,
          prototxtBuffer.buffer.asUint8List(),
          txtBuffer.buffer.asUint8List(),
          characterModelBuffer.buffer.asUint8List());
    }
  }

  var templateName = 'MRZ.json';
  var templatePath = join(modelPath, templateName);
  String template = await loadAssetString(templatePath);
  if (isDesktop) {
    var templateMap = json.decode(template);
    templateMap['CharacterModelArray'][0]['DirectoryPath'] = directory.path;

    ByteData templateBuffer = await loadAssetBytes(templatePath);
    List<int> bytes = templateBuffer.buffer.asUint8List();
    await File(join(directory.path, templateName)).writeAsBytes(bytes);
    await methodChannel.invokeMethod('loadModel',
        {'path': directory.path, 'template': json.encode(templateMap)});
  } else {
    ret = await loadTemplate(template);
  }

  return ret;
}
Enter fullscreen mode Exit fullscreen mode

On mobile, the model files are loaded directly from the assets folder. Whereas on desktop, the model files are copied to the application documents directory and then loaded from the directory.

The other three methods are similar to the web implementation.

Future<int?> init(String path, String key) async {
  return await methodChannel
      .invokeMethod<int>('init', {'path': path, 'key': key});
}

Future<List<List<MrzLine>>?> recognizeByBuffer(
    Uint8List bytes, int width, int height, int stride, int format) async {
  List<dynamic>? results =
      await methodChannel.invokeMethod('recognizeByBuffer', {
    'bytes': bytes,
    'width': width,
    'height': height,
    'stride': stride,
    'format': format,
  });

  if (results == null || results.isEmpty) return [];

  return _resultWrapper(results);
}

Future<List<List<MrzLine>>?> recognizeByFile(String filename) async {
  List<dynamic>? results =
      await methodChannel.invokeMethod('recognizeByFile', {
    'filename': filename,
  });

  if (results == null || results.isEmpty) return [];

  return _resultWrapper(results);
}

List<List<MrzLine>> _resultWrapper(List<dynamic> data) {
  List<List<MrzLine>> results = [];

  for (List<dynamic> area in data) {
    List<MrzLine> lines = [];
    if (area.length == 2 || area.length == 3) {
      for (int i = 0; i < area.length; i++) {
        MrzLine line = MrzLine();
        Map<dynamic, dynamic> map = area[i];
        line.confidence = map['confidence'];
        line.text = map['text'];
        line.x1 = map['x1'];
        line.y1 = map['y1'];
        line.x2 = map['x2'];
        line.y2 = map['y2'];
        line.x3 = map['x3'];
        line.y3 = map['y3'];
        line.x4 = map['x4'];
        line.y4 = map['y4'];
        lines.add(line);
      }
    }

    results.add(lines);
  }

  return results;
}
Enter fullscreen mode Exit fullscreen mode

MRZ Parser

MRZ parser is used to parse the MRZ text and get the information of the passport. The MRZ parser is implemented in the mrz_parser.dart file.

class MRZ {
  /// Parse two lines of MRZ string.
  static MrzResult parseTwoLines(String line1, String line2) {
    MrzResult mrzInfo = MrzResult();
    String type = line1.substring(0, 1);
    RegExp exp = RegExp(r'[I|P|V]');
    RegExpMatch? match = exp.firstMatch(type);
    if (match == null) {
      return mrzInfo;
    }

    if (type == 'P') {
      mrzInfo.type = 'PASSPORT (TD-3)';
    } else if (type == 'V') {
      if (line1.length == 44) {
        mrzInfo.type = 'VISA (MRV-A)';
      } else if (line1.length == 36) {
        mrzInfo.type = 'VISA (MRV-B)';
      }
    } else if (type == 'I') {
      mrzInfo.type = 'ID CARD (TD-2)';
    }

    // Get issuing State information
    String nation = line1.substring(2, 5);
    exp = RegExp(r'[0-9]');
    match = exp.firstMatch(nation);
    if (match != null) return mrzInfo;
    if (nation[nation.length - 1] == '<') {
      nation = nation.substring(0, 2);
    }
    mrzInfo.nationality = nation;
    // Get surname information
    line1 = line1.substring(5);
    int pos = line1.indexOf('<<');
    String surName = line1.substring(0, pos);
    exp = RegExp(r'[0-9]');
    match = exp.firstMatch(surName);
    if (match != null) return mrzInfo;
    surName = surName.replaceAll('<', ' ');
    mrzInfo.surname = surName;
    // Get givenname information
    String givenName = line1.substring(surName.length + 2);
    exp = RegExp(r'[0-9]');
    match = exp.firstMatch(givenName);
    if (match != null) return mrzInfo;
    givenName = givenName.replaceAll('<', ' ');
    givenName = givenName.trim();
    mrzInfo.givenName = givenName;
    // Get passport number information
    String passportNumber = '';
    passportNumber = line2.substring(0, 9);
    passportNumber = passportNumber.replaceAll('<', ' ');
    mrzInfo.passportNumber = passportNumber;
    // Get Nationality information
    String issueCountry = line2.substring(10, 13);
    exp = RegExp(r'[0-9]');
    match = exp.firstMatch(issueCountry);
    if (match != null) return mrzInfo;
    if (issueCountry[issueCountry.length - 1] == '<') {
      issueCountry = issueCountry.substring(0, 2);
    }
    mrzInfo.issuingCountry = issueCountry;
    // Get date of birth information
    String birth = line2.substring(13, 19);
    DateTime now = DateTime.now();
    int currentYear = now.year;
    if (int.parse(birth.substring(0, 2)) > (currentYear % 100)) {
      birth = '19$birth';
    } else {
      birth = '20$birth';
    }
    birth =
        '${birth.substring(0, 4)}/${birth.substring(4, 6)}/${birth.substring(6, 8)}';
    mrzInfo.birthDate = birth;

    // Get gender information
    String gender = line2[20];
    exp = RegExp(r'[M|F|x|<]');
    match = exp.firstMatch(gender);
    if (match == null) return mrzInfo;
    mrzInfo.gender = gender;
    // Get date of expiry information
    String expiry = line2.substring(21, 27);
    exp = RegExp(r'[A-Za-z]');
    match = exp.firstMatch(expiry);
    if (match != null) return mrzInfo;
    if (int.parse(expiry.substring(0, 2)) >= 60) {
      expiry = '19$expiry';
    } else {
      expiry = '20$expiry';
    }
    expiry =
        '${expiry.substring(0, 4)}/${expiry.substring(4, 6)}/${expiry.substring(6)}';
    mrzInfo.expiration = expiry;

    return mrzInfo;
  }

  /// Parse three lines of MRZ string.
  static MrzResult parseThreeLines(String line1, String line2, String line3) {
    MrzResult mrzInfo = MrzResult();
    String type = line1.substring(0, 1);
    RegExp exp = RegExp(r'[I|P|V]');
    RegExpMatch? match = exp.firstMatch(type);
    if (match == null) {
      return mrzInfo;
    }

    mrzInfo.type = 'ID CARD (TD-1)';
    // Get nationality information
    String nation = line2.substring(15, 18);
    exp = RegExp(r'[0-9]');
    match = exp.firstMatch(nation);
    if (match != null) return mrzInfo;
    nation = nation.replaceAll('<', '');
    mrzInfo.nationality = nation;
    // Get surname information
    int pos = line3.indexOf('<<');
    String surName = line3.substring(0, pos);
    exp = RegExp(r'[0-9]');
    match = exp.firstMatch(surName);
    if (match != null) return mrzInfo;
    surName = surName.replaceAll('<', ' ');
    surName.trim();
    mrzInfo.surname = surName;
    // Get givenname information
    String givenName = line3.substring(surName.length + 2);
    exp = RegExp(r'[0-9]');
    match = exp.firstMatch(givenName);
    if (match != null) return mrzInfo;
    givenName = givenName.replaceAll('<', ' ');
    givenName = givenName.trim();
    mrzInfo.givenName = givenName;
    // Get passport number information
    String passportNumber = '';
    passportNumber = line1.substring(5, 14);
    passportNumber = passportNumber.replaceAll('<', ' ');
    mrzInfo.passportNumber = passportNumber;
    // Get issuing country or organization information
    String issueCountry = line1.substring(2, 5);
    exp = RegExp(r'[0-9]');
    match = exp.firstMatch(issueCountry);
    if (match != null) return mrzInfo;
    issueCountry = issueCountry.replaceAll('<', '');
    mrzInfo.issuingCountry = issueCountry;
    // Get date of birth information
    String birth = line2.substring(0, 6);
    exp = RegExp(r'[A-Za-z]');
    match = exp.firstMatch(birth);
    if (match != null) return mrzInfo;

    DateTime now = DateTime.now();
    int currentYear = now.year;
    if (int.parse(birth.substring(0, 2)) > (currentYear % 100)) {
      birth = '19$birth';
    } else {
      birth = '20$birth';
    }
    birth =
        '${birth.substring(0, 4)}/${birth.substring(4, 6)}/${birth.substring(6, 8)}';
    mrzInfo.birthDate = birth;

    // Get gender information
    String gender = line2[7];
    exp = RegExp(r'[M|F|x|<]');
    match = exp.firstMatch(gender);
    if (match == null) return mrzInfo;
    gender = gender.replaceAll('<', 'X');
    mrzInfo.gender = gender;
    // Get date of expiry information
    String expiry = '20$line2.substring(8, 14)';
    exp = RegExp(r'[A-Za-z]');
    match = exp.firstMatch(expiry);
    if (match != null) return mrzInfo;
    expiry =
        '${expiry.substring(0, 4)}/${expiry.substring(4, 6)}/${expiry.substring(6)}';
    mrzInfo.expiration = expiry;

    return mrzInfo;
  }
}
Enter fullscreen mode Exit fullscreen mode

Platform-specific Code of MRZ Recognition for Android, iOS, Windows and Linux

In the above section, we have implemented the MRZ recognition logic in Dart. For web, the Dart code can be directly used. Now, we will write Java, Swift and C++ code for Android, iOS, Windows, and Linux.

Get Function Names and Arguments via Method Channel

Android

public void onMethodCall(@NonNull MethodCall call, @NonNull Result result) {
  switch (call.method) {
    case "init": {
      final String license = call.argument("key");
      break;
    }
    case "recognizeByFile": {
      final String filename = call.argument("filename");
      final Result r = result;
    }
    break;
    case "recognizeByBuffer": {
      final byte[] bytes = call.argument("bytes");
      final int width = call.argument("width");
      final int height = call.argument("height");
      final int stride = call.argument("stride");
      final int format = call.argument("format");
      final Result r = result;
    }
    break;
    case "loadModelFiles": {
      final String name = call.argument("name");
      final byte[] prototxtBuffer = call.argument("prototxtBuffer");
      final byte[] txtBuffer = call.argument("txtBuffer");
      final byte[] characterModelBuffer = call.argument("characterModelBuffer");
    }
    break;
    case "loadTemplate": {
      final String template = call.argument("template");
    }
    break;
    default:
      result.notImplemented();
  }
}
Enter fullscreen mode Exit fullscreen mode

iOS

public func handle(_ call: FlutterMethodCall, result: @escaping FlutterResult) {
  let arguments: NSDictionary = call.arguments as! NSDictionary
  switch call.method {
      case "init":
          completionHandlers.append(result)
          let license: String = arguments.value(forKey: "key") as! String
      case "loadModelFiles":
          let name: String = arguments.value(forKey: "name") as! String
          let prototxtBuffer: FlutterStandardTypedData = arguments.value(forKey: "prototxtBuffer") as! FlutterStandardTypedData
          let txtBuffer: FlutterStandardTypedData = arguments.value(forKey: "txtBuffer") as! FlutterStandardTypedData
          let characterModelBuffer: FlutterStandardTypedData = arguments.value(forKey: "characterModelBuffer") as! FlutterStandardTypedData
      case "loadTemplate":
          if self.recognizer == nil {
              result(.none)
              return
          }
          let params: String = arguments.value(forKey: "template") as! String
      case "recognizeByFile":
          if recognizer == nil {
              result(.none)
              return
          }

          DispatchQueue.global().async {
              let filename: String = arguments.value(forKey: "filename") as! String
          }
      case "recognizeByBuffer":
          if self.recognizer == nil {
              result(.none)
              return
          }

          DispatchQueue.global().async {
              let buffer: FlutterStandardTypedData = arguments.value(forKey: "bytes") as! FlutterStandardTypedData
              let width: Int = arguments.value(forKey: "width") as! Int
              let height: Int = arguments.value(forKey: "height") as! Int
              let stride: Int = arguments.value(forKey: "stride") as! Int
              let format: Int = arguments.value(forKey: "format") as! Int
              let enumImagePixelFormat = EnumImagePixelFormat(rawValue: format)
          }
      default:
          result(.none)
      }
}
Enter fullscreen mode Exit fullscreen mode

Windows

void FlutterOcrSdkPlugin::HandleMethodCall(
      const flutter::MethodCall<flutter::EncodableValue> &method_call,
      std::unique_ptr<flutter::MethodResult<flutter::EncodableValue>> result)
{

  const auto *arguments = std::get_if<EncodableMap>(method_call.arguments());

  if (method_call.method_name().compare("init") == 0)
  {
    std::string license;
    int ret = 0;

    if (arguments)
    {
      auto license_it = arguments->find(EncodableValue("key"));
      if (license_it != arguments->end())
      {
        license = std::get<std::string>(license_it->second);
      }
    }
  }
  else if (method_call.method_name().compare("loadModel") == 0)
  {
    std::string path, params;
    int ret = 0;

    if (arguments)
    {
      auto path_it = arguments->find(EncodableValue("path"));
      if (path_it != arguments->end())
      {
        path = std::get<std::string>(path_it->second);
      }

      auto params_it = arguments->find(EncodableValue("template"));
      if (params_it != arguments->end())
      {
        params = std::get<std::string>(params_it->second);
      }
    }
  }
  else if (method_call.method_name().compare("recognizeByFile") == 0)
  {
    std::string filename;
    EncodableList results;

    if (arguments)
    {
      auto filename_it = arguments->find(EncodableValue("filename"));
      if (filename_it != arguments->end())
      {
        filename = std::get<std::string>(filename_it->second);
      }
    }
  }
  else if (method_call.method_name().compare("recognizeByBuffer") == 0)
  {
    EncodableList results;

    std::vector<unsigned char> bytes;
    int width = 0, height = 0, stride = 0, format = 0;

    if (arguments)
    {
      auto bytes_it = arguments->find(EncodableValue("bytes"));
      if (bytes_it != arguments->end())
      {
        bytes = std::get<vector<unsigned char>>(bytes_it->second);
      }

      auto width_it = arguments->find(EncodableValue("width"));
      if (width_it != arguments->end())
      {
        width = std::get<int>(width_it->second);
      }

      auto height_it = arguments->find(EncodableValue("height"));
      if (height_it != arguments->end())
      {
        height = std::get<int>(height_it->second);
      }

      auto stride_it = arguments->find(EncodableValue("stride"));
      if (stride_it != arguments->end())
      {
        stride = std::get<int>(stride_it->second);
      }

      auto format_it = arguments->find(EncodableValue("format"));
      if (format_it != arguments->end())
      {
        format = std::get<int>(format_it->second);
      }
    }
  }
  else
  {
    result->NotImplemented();
  }
}
Enter fullscreen mode Exit fullscreen mode

Linux

static void flutter_ocr_sdk_plugin_handle_method_call(
    FlutterOcrSdkPlugin* self,
    FlMethodCall* method_call) {
  g_autoptr(FlMethodResponse) response = nullptr;

  const gchar* method = fl_method_call_get_name(method_call);
  FlValue* args = fl_method_call_get_args(method_call);

  if (strcmp(method, "init") == 0)
  {
    if (fl_value_get_type(args) != FL_VALUE_TYPE_MAP)
    {
      return;
    }

    FlValue *value = fl_value_lookup_string(args, "key");
    if (value == nullptr)
    {
      return;
    }
    const char *license = fl_value_get_string(value);
  }
  else if (strcmp(method, "loadModel") == 0)
  {
    if (fl_value_get_type(args) != FL_VALUE_TYPE_MAP)
    {
      return;
    }

    FlValue *value = fl_value_lookup_string(args, "path");
    if (value == nullptr)
    {
      return;
    }
    const char *path = fl_value_get_string(value);

    value = fl_value_lookup_string(args, "template");
    if (value == nullptr)
    {
      return;
    }
    const char * params = fl_value_get_string(value);

    int ret = self->manager->LoadModel(path, params);
  }
  else if (strcmp(method, "recognizeByFile") == 0)
  {
    if (fl_value_get_type(args) != FL_VALUE_TYPE_MAP)
    {
      return;
    }

    FlValue *value = fl_value_lookup_string(args, "filename");
    if (value == nullptr)
    {
      return;
    }
    const char *filename = fl_value_get_string(value);

  }
  else if (strcmp(method, "recognizeByBuffer") == 0)
  {
    if (fl_value_get_type(args) != FL_VALUE_TYPE_MAP) {
      return;
    }

    FlValue* value = fl_value_lookup_string(args, "bytes");
    if (value == nullptr) {
      return;
    }
    unsigned char* bytes = (unsigned char*)fl_value_get_uint8_list(value);

    value = fl_value_lookup_string(args, "width");
    if (value == nullptr) {
      return;
    }
    int width = fl_value_get_int(value);

    value = fl_value_lookup_string(args, "height");
    if (value == nullptr) {
      return;
    }
    int height = fl_value_get_int(value);

    value = fl_value_lookup_string(args, "stride");
    if (value == nullptr) {
      return;
    }
    int stride = fl_value_get_int(value);

    value = fl_value_lookup_string(args, "format");
    if (value == nullptr) {
      return;
    }
    int format = fl_value_get_int(value);

  }
  else {
    response = FL_METHOD_RESPONSE(fl_method_not_implemented_response_new());
  }

  fl_method_call_respond(method_call, response, nullptr);
}
Enter fullscreen mode Exit fullscreen mode

The Init() Method

Android

LicenseManager.initLicense(
  license, context,
      new LicenseVerificationListener() {
          @Override
          public void licenseVerificationCallback(boolean isSuccessful, CoreException e) {
              if (isSuccessful)
              {
                  result.success(0);
              }
              else {
                  result.success(-1);
              }
          }
      });
Enter fullscreen mode Exit fullscreen mode

iOS

DynamsoftLicenseManager.initLicense(license, verificationDelegate: self)

public func licenseVerificationCallback(_ isSuccess: Bool, error: Error?) {
  if isSuccess {
      completionHandlers.first?(0)
  } else{
      completionHandlers.first?(-1)
  }
}
Enter fullscreen mode Exit fullscreen mode

Windows

char errorMsgBuffer[512];
int ret = DLR_InitLicense(license, errorMsgBuffer, 512);
recognizer = DLR_CreateInstance();
Enter fullscreen mode Exit fullscreen mode

Linux

char errorMsgBuffer[512];
int ret = DLR_InitLicense(license, errorMsgBuffer, 512);
recognizer = DLR_CreateInstance();
Enter fullscreen mode Exit fullscreen mode

The LoadModel() Method

Android

try {
    mLabelRecognizer.appendCharacterModelBuffer(name, prototxtBuffer, txtBuffer, characterModelBuffer);
    mLabelRecognizer.initRuntimeSettings(content);
} catch (Exception e) {
    e.printStackTrace();
}
Enter fullscreen mode Exit fullscreen mode

iOS

DynamsoftLabelRecognizer.appendCharacterModel(name, prototxtBuffer: prototxtBuffer.data, txtBuffer: txtBuffer.data, characterModelBuffer: characterModelBuffer.data)

try? self.recognizer!.initRuntimeSettings(params)
Enter fullscreen mode Exit fullscreen mode

Windows

char errorMessage[256];
int ret = DLR_AppendSettingsFromString(recognizer, params, errorMessage, 256);
Enter fullscreen mode Exit fullscreen mode

Linux

char errorMessage[256];
int ret = DLR_AppendSettingsFromString(recognizer, params, errorMessage, 256);
Enter fullscreen mode Exit fullscreen mode

The RecognizeByFile() Method

Android

DLRResult[] results = null;
try {
    results = mLabelRecognizer.recognizeFile(fileName);
} catch (Exception e) {
    Log.e(TAG, e.toString());
}
Enter fullscreen mode Exit fullscreen mode

iOS

let res = try? self.recognizer!.recognizeFile(filename)
Enter fullscreen mode Exit fullscreen mode

Windows

int ret = DLR_RecognizeByFile(recognizer, filename, "locr");
Enter fullscreen mode Exit fullscreen mode

Linux

int ret = DLR_RecognizeByFile(recognizer, filename, "locr");
Enter fullscreen mode Exit fullscreen mode

The RecognizeByBuffer() Method

Android

DLRResult[] results = null;
ImageData data = new ImageData();
data.bytes = bytes;
data.width = width;
data.height = height;
data.stride = stride;
data.format = format;
try {
    results = mLabelRecognizer.recognizeBuffer(data);
} catch (Exception e) {
    Log.e(TAG, e.toString());
}
Enter fullscreen mode Exit fullscreen mode

iOS

let imageData = iImageData.init()
imageData.bytes = buffer.data
imageData.width = width
imageData.height = height
imageData.stride = stride
imageData.format = enumImagePixelFormat!
let res = try? self.recognizer!.recognizeBuffer(imageData)
Enter fullscreen mode Exit fullscreen mode

Windows

ImageData data;
data.bytes = buffer;
data.width = width;
data.height = height;
data.stride = stride;
data.format = pixelFormat;
data.bytesLength = length;

int ret = DLR_RecognizeByBuffer(recognizer, &data, "locr");
Enter fullscreen mode Exit fullscreen mode

Linux

ImageData data;
data.bytes = buffer;
data.width = width;
data.height = height;
data.stride = stride;
data.format = pixelFormat;
data.bytesLength = length;

int ret = DLR_RecognizeByBuffer(recognizer, &data, "locr");
Enter fullscreen mode Exit fullscreen mode

Building a Flutter App to Recognize Passport MRZ

  1. Add image_picker, file_selector and flutter_ocr_sdk to pubspec.yaml. Both image_picker and file_selector are used to select an image file from the local file system, but they are implemented for different platforms.

    dependencies:
  ...

  flutter_ocr_sdk:
  image_picker:
  file_selector:

  2. Create a stateful widget:

    import 'dart:typed_data';

import 'package:file_selector/file_selector.dart';
import 'package:flutter/material.dart';
import 'dart:async';

import 'dart:io';

import 'package:flutter_ocr_sdk/flutter_ocr_sdk.dart';
import 'package:flutter_ocr_sdk/flutter_ocr_sdk_platform_interface.dart';
import 'package:flutter_ocr_sdk/mrz_line.dart';
import 'package:flutter_ocr_sdk/mrz_parser.dart';
import 'package:image_picker/image_picker.dart';

import 'package:flutter/foundation.dart' show kIsWeb;

import 'dart:ui' as ui;

Future<void> main() async {
  runApp(
    MaterialApp(
      title: 'MRZ OCR',
      home: Scaffold(
        appBar: AppBar(
          title: const Text("MRZ OCR"),
        ),
        body: MRZApp(),
      ),
    ),
  );
}

class MRZApp extends StatefulWidget {
  @override
  MobileState createState() => MobileState();
}

  3. Initialize the MRZ detection SDK:

    class MobileState extends State<MRZApp> {
  late FlutterOcrSdk _mrzDetector;
  final picker = ImagePicker();

  @override
  void initState() {
    super.initState();

    initSDK();
  }

  Future<void> initSDK() async {
    _mrzDetector = FlutterOcrSdk();
    int? ret = await _mrzDetector.init(
        "https://cdn.jsdelivr.net/npm/dynamsoft-label-recognizer@2.2.11/dist/",
        "DLS2eyJoYW5kc2hha2VDb2RlIjoiMjAwMDAxLTE2NDk4Mjk3OTI2MzUiLCJvcmdhbml6YXRpb25JRCI6IjIwMDAwMSIsInNlc3Npb25QYXNzd29yZCI6IndTcGR6Vm05WDJrcEQ5YUoifQ==");
    await _mrzDetector.loadModel();
  }
}

  4. Build the UI:

    Widget build(BuildContext context) {
  double width = MediaQuery.of(context).size.width;
  double height = MediaQuery.of(context).size.height;
  double left = 5;
  double mrzHeight = 50;
  double mrzWidth = width - left * 2;
  return Scaffold(
    body: Stack(children: [
      Center(
        child: Column(
            mainAxisAlignment: MainAxisAlignment.spaceEvenly,
            children: [
              MaterialButton(
                textColor: Colors.white,
                color: Colors.blue,
                onPressed: () async {
                  showDialog(
                      context: context,
                      builder: (BuildContext context) {
                        return const Center(
                          child: CircularProgressIndicator(),
                        );
                      });
                  pictureScan('gallery');
                },
                child: const Text('Pick gallery image'),
              ),
              MaterialButton(
                textColor: Colors.white,
                color: Colors.blue,
                onPressed: () async {
                  showDialog(
                      context: context,
                      builder: (BuildContext context) {
                        return const Center(
                          child: CircularProgressIndicator(),
                        );
                      });
                  pictureScan('camera');
                },
                child: const Text('Pick camera image'),
              ),
            ]),
      )
    ]),
  );
}

  5. Press the button to load an image file from the local file system and then recognize MRZ from the image:

    void pictureScan(String source) async {
  XFile? photo;
  if (kIsWeb || Platform.isAndroid || Platform.isIOS) {
    if (source == 'camera') {
      photo = await picker.pickImage(source: ImageSource.camera);
    } else {
      photo = await picker.pickImage(source: ImageSource.gallery);
    }
  } else if (Platform.isWindows || Platform.isLinux) {
    const XTypeGroup typeGroup = XTypeGroup(
      label: 'images',
      extensions: <String>['jpg', 'png', 'bmp', 'tiff', 'pdf', 'gif'],
    );
    photo = await openFile(acceptedTypeGroups: <XTypeGroup>[typeGroup]);
  }

  if (photo == null) {
    if (!mounted) return;
    Navigator.pop(context);
    return;
  }

  String information = 'No results';

  List<List<MrzLine>>? results =
      await _mrzDetector.recognizeByFile(photo.path);
  if (results != null && results.isNotEmpty) {
    for (List<MrzLine> area in results) {
      if (area.length == 2) {
        information =
            MRZ.parseTwoLines(area[0].text, area[1].text).toString();
      } else if (area.length == 3) {
        information = MRZ
            .parseThreeLines(area[0].text, area[1].text, area[2].text)
            .toString();
      }
    }
  }

  if (!mounted) return;
  Navigator.pop(context);
  Navigator.push(
    context,
    MaterialPageRoute(
      builder: (context) => DisplayPictureScreen(
          imagePath: photo!.path, mrzInformation: information),
    ),
  );
}

  6. Display the MRZ information on a stateless widget:

    Image getImage(String imagePath) {
  if (kIsWeb) {
    return Image.network(imagePath);
  } else {
    return Image.file(
      File(imagePath),
      fit: BoxFit.contain,
      height: double.infinity,
      width: double.infinity,
      alignment: Alignment.center,
    );
  }
}
class DisplayPictureScreen extends StatelessWidget {
  final String imagePath;
  final String mrzInformation;

  const DisplayPictureScreen(
      {Key? key, required this.imagePath, required this.mrzInformation})
      : super(key: key);

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text('MRZ OCR')),
      body: Stack(
        alignment: const Alignment(0.0, 0.0),
        children: [
          getImage(imagePath),
          Container(
            decoration: const BoxDecoration(
              color: Colors.black45,
            ),
            child: Text(
              mrzInformation,
              style: const TextStyle(
                fontSize: 14,
                color: Colors.white,
              ),
            ),
          ),
        ],
      ),
    );
  }
}

  7. Run the app on web, Windows, Linux, Android and iOS platforms.

    Web

    Flutter MRZ recognition in web

    Windows and Linux

    Flutter MRZ OCR in Windows

    Android and iOS

    Flutter Passport MRZ recognition

Source Code

https://github.com/yushulx/flutter_ocr_sdk

Top comments (0)

Read next

aadarshk7 profile image

Flutter BottomNavigationBar

Aadarsh Kunwar -

kiolk profile image

Developer diary #17. How does it work?

Kiolk -

lucasginard profile image

Como hacer bypass al SSL Pinning en Android con Frida y Objection en Windows 10🔓

Lucas Ginard -

autonomousapps profile image

Announcing Dependency Analysis Gradle Plugin 2.0.0!

Tony Robalik -