Integrating Document and MRZ Detection SDK into .NET MAUI for Windows

The Capture.Vision.Maui NuGet package aims to equip developers with a .NET MAUI library, featuring a camera view and various image processing functions. This library streamlines the development of .NET MAUI applications across multiple platforms, including Windows, Android, and iOS. Previously, we introduced barcode scanning capabilities within the .NET MAUI library. In this article, we will focus on integrating document and Machine-Readable Zone (MRZ) detection into .NET MAUI for Windows applications.

Demo: Detecting Barcodes, Documents, and MRZ in .NET MAUI Windows Applications

NuGet Package

https://www.nuget.org/packages/Capture.Vision.Maui

Integrating Document Detection SDK

1. Add the DocumentScannerSDK NuGet package to your .NET MAUI project's .csproj file.
   

   
   <ItemGroup Condition="$([MSBuild]::IsOSPlatform('windows'))">
       <PackageReference Include="DocumentScannerSDK " Version="1.1.0" />
   </ItemGroup>
   
   

   Note: When installing NuGet packages via the NuGet command line or the NuGet package explorer, the target platform is not specified. Since this package is only available for Windows, you need to manually add the platform condition.

2. Create a new class file named DocumentResult.cs. This class, DocumentResult, will contain the results of the document detection, including the confidence level, coordinates (points), and the rectified image data.
   

   using static Dynamsoft.DocumentScanner;
   
   namespace Capture.Vision.Maui
   {
       public class DocumentResult
       {
           public int Confidence { get; set; }
   
           public int[] Points { get; set; }
   
           public int Width;
   
           public int Height;
   
           public int Stride;
   
           public ImagePixelFormat Format;
   
           public byte[] Data { get; set; }
       }
   }
   
   

3. In the CameraView.cs file, add new BindableProperty variables to enable switching the document detection mode on and off.
   

   public static readonly BindableProperty EnableDocumentDetectProperty = BindableProperty.Create(nameof(EnableDocumentDetectProperty), typeof(bool), typeof(CameraView), false);
   public static readonly BindableProperty EnableDocumentRectifyProperty = BindableProperty.Create(nameof(EnableDocumentRectifyProperty), typeof(bool), typeof(CameraView), false);
   
   public bool EnableDocumentDetect
   {
       get { return (bool)GetValue(EnableDocumentDetectProperty); }
       set { SetValue(EnableDocumentDetectProperty, value); }
   }
   
   public bool EnableDocumentRectify
   {
       get { return (bool)GetValue(EnableDocumentRectifyProperty); }
       set { SetValue(EnableDocumentRectifyProperty, value); }
   }
   

4. Open the file Platforms/Windows/NativeCameraView.cs. Incorporate the document detection code into the ProcessFrames() function.
   

   private void ProcessFrames() {
       ...
   
       if (cameraView.EnableDocumentDetect)
       {
           DocumentScanner.Result[] results = documentScanner.DetectBuffer(buffer, bitmap.PixelWidth, bitmap.PixelHeight, bitmap.PixelWidth, DocumentScanner.ImagePixelFormat.IPF_GRAYSCALED);
           DocumentResult documentResults = new DocumentResult();
           if (results != null && results.Length > 0)
           {
               documentResults = new DocumentResult
               {
                   Confidence = results[0].Confidence,
                   Points = results[0].Points
               };
   
               if (cameraView.EnableDocumentRectify)
               {
                   NormalizedImage image = documentScanner.NormalizeBuffer(buffer, bitmap.PixelWidth, bitmap.PixelHeight, bitmap.PixelWidth, DocumentScanner.ImagePixelFormat.IPF_GRAYSCALED, documentResults.Points);
                   documentResults.Width = image.Width;
                   documentResults.Height = image.Height;
                   documentResults.Stride = image.Stride;
                   documentResults.Format = image.Format;
                   documentResults.Data = image.Data;
               }
           }
   
           cameraView.NotifyResultReady(documentResults, bitmap.PixelWidth, bitmap.PixelHeight);
   
       }
   
       ...
   }
   
   

Integrating MRZ Detection SDK

1. Add the MrzScannerSDK NuGet package to your .NET MAUI project's .csproj file.
   

   <ItemGroup Condition="$([MSBuild]::IsOSPlatform('windows'))">
       <PackageReference Include="MrzScannerSDK" Version="1.3.3" />
   </ItemGroup>
   

2. Create a new class file named MrzResult.cs. This class, MrzResult, will store the machine-readable zone (MRZ) detection results, which include the detected text line and parsed MRZ data. This data encompasses various elements such as type, nationality, surname, given name, passport number, issuing country, date of birth, gender, and expiration date.
   

   namespace Capture.Vision.Maui
   {
       public class Line
       {
           public int Confidence { get; set; }
   
           public string Text { get; set; }
   
           public int[] Points { get; set; }
       }
   
       public class MrzResult
       {
           public string Type { get; set; } = "N/A";
           public string Nationality { get; set; } = "N/A";
           public string Surname { get; set; } = "N/A";
           public string GivenName { get; set; } = "N/A";
           public string PassportNumber { get; set; } = "N/A";
           public string IssuingCountry { get; set; } = "N/A";
           public string BirthDate { get; set; } = "N/A";
           public string Gender { get; set; } = "N/A";
           public string Expiration { get; set; } = "N/A";
           public string Lines { get; set; } = "N/A";
   
           public Line[] RawData { get; set; }
       }
   }
   
   

3. In the CameraView.cs file, add a BindableProperty variable for toggling the MRZ detection mode.
   

   public static readonly BindableProperty EnableMrzProperty = BindableProperty.Create(nameof(EnableMrzProperty), typeof(bool), typeof(CameraView), false);
   
   public bool EnableMrz
   {
       get { return (bool)GetValue(EnableMrzProperty); }
       set { SetValue(EnableMrzProperty, value); }
   }
   

4. Open the file Platforms/Windows/NativeCameraView.cs and integrate the MRZ detection code into the ProcessFrames() function.
   

   private void ProcessFrames() {
       ...
   
       if (cameraView.EnableMrz)
       {
           MrzScanner.Result[] results = mrzScanner.DetectBuffer(buffer, bitmap.PixelWidth, bitmap.PixelHeight, bitmap.PixelWidth, MrzScanner.ImagePixelFormat.IPF_GRAYSCALED);
           MrzResult mrzResults = new MrzResult();
   
           if (results != null && results.Length > 0)
           {
               Line[] rawData = new Line[results.Length];
               string[] lines = new string[results.Length];
   
               for (int i = 0; i < results.Length; i++)
               {
                   rawData[i] = new Line()
                   {
                       Confidence = results[i].Confidence,
                       Text = results[i].Text,
                       Points = results[i].Points,
                   };
                   lines[i] = results[i].Text;
               }
   
               try
               {
                   Dynamsoft.MrzResult info = MrzParser.Parse(lines);
                   mrzResults = new MrzResult()
                   {
                       RawData = rawData,
                       Type = info.Type,
                       Nationality = info.Nationality,
                       Surname = info.Surname,
                       GivenName = info.GivenName,
                       PassportNumber = info.PassportNumber,
                       IssuingCountry = info.IssuingCountry,
                       BirthDate = info.BirthDate,
                       Gender = info.Gender,
                       Expiration = info.Expiration,
                       Lines = info.Lines
                   };
               }
               catch (Exception ex) { 
   
               }
           }
   
           cameraView.NotifyResultReady(mrzResults, bitmap.PixelWidth, bitmap.PixelHeight);
       }
       ...
   }
   
   

Building a .NET MAUI Windows App with Document and MRZ Detection

In the upcoming section, we're going to construct a .NET MAUI Windows application that is equipped with both document and MRZ (Machine-Readable Zone) detection features.

1. Create a new .NET MAUI project in Visual Studio and install the Capture.Vision.Maui and SkiaSharp.Views.Maui.Controls NuGet packages. The SkiaSharp.Views.Maui.Controls package is used for drawing the detection results.
   

   <ItemGroup>
       <PackageReference Include="SkiaSharp.Views.Maui.Controls" Version="2.88.3" />
     <PackageReference Include="Capture.Vision.Maui" Version="1.2.1" />
   </ItemGroup>
   

2. In MauiProgram.cs, add the necessary code to register the Capture.Vision.Maui and SkiaSharp.Views.Maui.Controls packages.
   

   
   using Capture.Vision.Maui;
   using SkiaSharp.Views.Maui.Controls.Hosting;
   ...
   var builder = MauiApp.CreateBuilder();
   builder.UseSkiaSharp().UseNativeCameraView()
   

3. Request a trial license for the Document Scanner SDK and MRZ Scanner SDK. Then, in the MainPage.xaml.cs file, set the license keys for Windows using directives.
   

   using Dynamsoft;
   
   namespace Capture.Vision.Maui.Example
   {
       public partial class MainPage : ContentPage
       {
           public MainPage()
           {
               InitializeComponent();
               InitService();
           }
   
           private async void InitService()
           {
               await Task.Run(() =>
               {
                   BarcodeQRCodeReader.InitLicense("LICENSEK-KEY    ");
   #if WINDOWS
       DocumentScanner.InitLicense("LICENSEK-KEY"); 
       MrzScanner.InitLicense("LICENSEK-KEY"); 
   #elif ANDROID
   #elif IOS
   #endif
                   return Task.CompletedTask;
               });
           }
       }
   }
   

4. Create a new content page and incorporate the CameraView control. Ensure that the EnableBarcode, EnableDocumentDetect, and EnableMrz properties are set to True. Use SKCanvasView to render the detection results.
   

   <?xml version="1.0" encoding="utf-8" ?>
   <ContentPage xmlns="http://schemas.microsoft.com/dotnet/2021/maui"
                xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
                xmlns:skia="clr-namespace:SkiaSharp.Views.Maui.Controls;assembly=SkiaSharp.Views.Maui.Controls"
                xmlns:cv="clr-namespace:Capture.Vision.Maui;assembly=Capture.Vision.Maui"
                x:Class="Capture.Vision.Maui.Example.CameraPage"
                Title="CameraPage">
       <ScrollView>
           <Grid>
               <cv:CameraView x:Name="cameraView" HorizontalOptions="FillAndExpand" "EnableBarcode="True" EnableDocumentDetect="True" EnableMrz="True"
               VerticalOptions="FillAndExpand" ResultReady="cameraView_ResultReady" FrameReady="cameraView_FrameReady"/>
               <skia:SKCanvasView x:Name="canvasView" 
              Margin="0"
              HorizontalOptions="FillAndExpand" VerticalOptions="FillAndExpand"
              PaintSurface="OnCanvasViewPaintSurface" />
           </Grid>
       </ScrollView>
   </ContentPage>
   

5. In the CameraPage.xaml.cs file, add code to handle the FrameReady and ResultReady events.
   

   BarcodeQrData[] barcodeData = null;
   DocumentData documentData = null;
   MrzData mrzData = null;
   
   private void cameraView_FrameReady(object sender, FrameReadyEventArgs e)
   {
       MainThread.BeginInvokeOnMainThread(() =>
       {
           canvasView.InvalidateSurface();
       });
   }
   
   private void cameraView_ResultReady(object sender, ResultReadyEventArgs e)
   {
       lock (_lockObject)
       {
   
           if (e.Result is BarcodeResult[])
           {
               barcodeData = null;
               BarcodeResult[] barcodeResults = (BarcodeResult[])e.Result;
           }
           else if (e.Result is DocumentResult)
           {
               documentData = null;
               DocumentResult documentResult = (DocumentResult)e.Result;
   
               if (documentResult.Points != null)
               {
                   documentData = new DocumentData()
                   {
                       Reference = documentResult
                   };
                   int[] coordinates = documentData.Reference.Points;
                   if (coordinates != null && coordinates.Length == 8)
                   {
                       documentData.Points = new SKPoint[4];
   
                       for (int i = 0; i < 4; ++i)
                       {
                           SKPoint p = new SKPoint();
                           p.X = coordinates[i * 2];
                           p.Y = coordinates[i * 2 + 1];
                           documentData.Points[i] = p;
   
                           if (orientation == DisplayOrientation.Portrait)
                           {
                               documentData.Points[i] = rotateCW90(documentData.Points[i], imageHeight);
                           }
   
                           documentData.Points[i].X = (float)(documentData.Points[i].X / scale);
                           documentData.Points[i].Y = (float)(documentData.Points[i].Y / scale);
                       }
                   }
               }
   
           }
           else if (e.Result is MrzResult)
           {
               mrzData = null;
               MrzResult mrzResult = (MrzResult)e.Result;
   
               if (mrzResult.RawData != null)
               {
                   mrzData = new MrzData()
                   {
                       Reference = mrzResult
                   };
   
                   Line[] rawData = mrzData.Reference.RawData;
                   mrzData.Points = new SKPoint[rawData.Length][];
                   for (int index = 0; index < rawData.Length; index++)
                   {
                       Line line = rawData[index];
                       int[] coordinates = line.Points;
                       mrzData.Points[index] = new SKPoint[4];
                       for (int i = 0; i < 4; ++i)
                       {
                           SKPoint p = new SKPoint();
                           p.X = coordinates[i * 2];
                           p.Y = coordinates[i * 2 + 1];
                           mrzData.Points[index][i] = p;
   
                           if (orientation == DisplayOrientation.Portrait)
                           {
                               mrzData.Points[index][i] = rotateCW90(mrzData.Points[index][i], imageHeight);
                           }
   
                           mrzData.Points[index][i].X = (float)(mrzData.Points[index][i].X / scale);
                           mrzData.Points[index][i].Y = (float)(mrzData.Points[index][i].Y / scale);
                       }
                   }
               }
           }
       }
   }
   

   Invoke canvasView.InvalidateSurface() to trigger the OnCanvasViewPaintSurface function, which will draw the detection results.
   

   void OnCanvasViewPaintSurface(object sender, SKPaintSurfaceEventArgs args)
   {    
       SKImageInfo info = args.Info;
       SKSurface surface = args.Surface;
       SKCanvas canvas = surface.Canvas;
   
       canvas.Clear();
   
       // Draw detection results
       lock (_lockObject)
       {
           if (documentData != null && cameraView.EnableDocumentDetect)
           {
               SKPaint skPaint = new SKPaint
               {
                   Style = SKPaintStyle.Stroke,
                   Color = SKColors.Red,
                   StrokeWidth = 10,
               };
   
               SKPaint textPaint = new SKPaint
               {
                   Style = SKPaintStyle.Stroke,
                   Color = SKColors.Red,
                   TextSize = (float)(16 * density),
                   StrokeWidth = 1,
               };
   
               canvas.DrawText("Detected Document", documentData.Points[0], textPaint);
               canvas.DrawLine(documentData.Points[0], documentData.Points[1], skPaint);
               canvas.DrawLine(documentData.Points[1], documentData.Points[2], skPaint);
               canvas.DrawLine(documentData.Points[2], documentData.Points[3], skPaint);
               canvas.DrawLine(documentData.Points[3], documentData.Points[0], skPaint);
           }
   
           if (mrzData != null && cameraView.EnableMrz)
           {
               SKPaint skPaint = new SKPaint
               {
                   Style = SKPaintStyle.Stroke,
                   Color = SKColors.Yellow,
                   StrokeWidth = 10,
               };
   
               SKPaint textPaint = new SKPaint
               {
                   Style = SKPaintStyle.Stroke,
                   Color = SKColors.Yellow,
                   TextSize = (float)(16 * density),
                   StrokeWidth = 1,
               };
   
               foreach (SKPoint[] line in mrzData.Points) {
                   canvas.DrawLine(line[0], line[1], skPaint);
                   canvas.DrawLine(line[1], line[2], skPaint);
                   canvas.DrawLine(line[2], line[3], skPaint);
                   canvas.DrawLine(line[3], line[0], skPaint);
               }
   
               SKPoint start = mrzData.Points[0][0];
               int delta = 20;
               start.X += 200;
               start.Y -= 200;
               canvas.DrawText(mrzData.Reference.Type, start, textPaint);
               start.Y += delta;
               canvas.DrawText(mrzData.Reference.Nationality, start, textPaint);
               start.Y += delta;
               canvas.DrawText(mrzData.Reference.Surname, start, textPaint);
               start.Y += delta;
               canvas.DrawText(mrzData.Reference.GivenName, start, textPaint);
               start.Y += delta;
               canvas.DrawText(mrzData.Reference.PassportNumber, start, textPaint);
               start.Y += delta;
               canvas.DrawText(mrzData.Reference.IssuingCountry, start, textPaint);
               start.Y += delta;
               canvas.DrawText(mrzData.Reference.BirthDate, start, textPaint);
               start.Y += delta;
               canvas.DrawText(mrzData.Reference.Gender, start, textPaint);
               start.Y += delta;
               canvas.DrawText(mrzData.Reference.Expiration, start, textPaint);
           }
       }
   }
   

6. Finally, run the .NET MAUI Windows application.

   

Source Code

https://github.com/yushulx/Capture-Vision-Maui