NIST’s Latest FRTE Report Shows State of the Art

The National Institute of Standards and Technology has released its latest 1:N report from its ongoing Face Recognition Technology Evaluation (FRTE), offering a glimpse at the current state of the art. The comprehensive evaluation includes data on some of the best-performing algorithms from various vendors, showcasing their accuracy in face recognition technology. Notably, vendors such as NEC, SenseTime, Cognitec, Idemia, Innovatrics, and Neurotechnology stood out with exceptional performances.

NEC’s algorithms were among the top performers, achieving a False Negative Identification Rate (FNIR) of 0.15 percent at a False Positive Identification Rate (FPIR) of 0.001. The result indicates that NEC’s systems correctly identified individuals 99.85 percent of the time, making it one of the most reliable in the evaluation. Similarly, SenseTime demonstrated strong accuracy with a FNIR of 0.10 percent at the same FPIR, reflecting its high precision in identifying individuals accurately. SenseTime also excelled at recognizing faces with significant pose variations, maintaining a 95 percent success rate.

Cognitec Systems GmbH also performed well, achieving a rank-one identification rate of 98.7 percent in the visa-like immigration application photos dataset. This metric indicates that the correct match was often the first suggestion, highlighting the efficiency of Cognitec’s algorithms. Cognitec also maintained over 97 percent accuracy in identifying individuals even with images taken several years apart, showcasing its long-term reliability.

Idemia’s algorithms achieved a FNIR of 0.12 percent at an FPIR of 0.001, indicating high accuracy in correct identifications. The company’s systems efficiently processed large datasets with an average accuracy of 99.8 percent across multiple datasets, including mugshots and visa-like photos. Innovatrics, another top performer, demonstrated strong performance with a FNIR of 0.11 percent at an FPIR of 0.001. Innovatrics maintained an accuracy rate of 98.5 percent across various datasets, including lower-quality images, showing its versatility.

Neurotechnology’s algorithms achieved a rank-one identification rate of 98.9 percent in the mugshot dataset, ensuring the top match was correct in nearly all cases. And Neurotechnology’s system maintained a 98 percent accuracy rate even as the enrolled population size increased to over 10 million individuals, highlighting its scalability and reliability.

Significant Accuracy Gains

The NIST report also discusses the significant accuracy gains achieved over recent years, attributing much of this progress to the adoption of deep convolutional neural networks (CNNs). Over the past decade, the industry has seen massive gains in accuracy, far exceeding improvements made in the period from 2010 to 2013. These gains stem from CNNs’ ability to handle poorly illuminated and low-quality images and to recognize faces with pose variations.

The report notes that while face recognition development continues to advance, the most accurate algorithm reported in the current evaluation is substantially more accurate than those reported in previous years. This continuous improvement demonstrates that the industry is rapidly evolving, with CNNs playing an important role in pushing the boundaries of what is possible in facial recognition.

New Developers

The latest NIST report also includes results for new developers to the evaluation program. Newcomers such as Sansap Technology, ALTTEKGlobal, Aratek Biometrics Co Ltd, BioID Technologies SA, EI Networks Private Ltd, Kogniza Technology, STCON LLC, VinBigData, FPT Smart Cloud, QazSmartVision.AI, and Yuan High-Tech Development have contributed to the ongoing evaluation.

These new developers brought innovative algorithms to the table, with some achieving commendable performance. For example, FPT Smart Cloud and QazSmartVision.AI demonstrated promising results, although specific accuracy rates were not as high as the established vendors like NEC and SenseTime. Nevertheless, the inclusion of these new developers in the NIST evaluation highlights the expanding landscape of face recognition technology and the continuous influx of new talent and innovation.

Other Notable Updates

NIST Interagency Report 8271 includes several significant updates that would catch the attention of stakeholders. The report introduces extended dataset evaluations, including profile view mugshots and lower-quality webcam photos, and new benchmarks for visa-border and visa-kiosk images, providing insights into real-world use cases in immigration and border control scenarios. NIST has also enhanced the algorithm-specific report cards to include figures showing how low threshold values can reduce candidate list lengths for human review, allowing for better-informed decisions about algorithm deployment in various applications.

A detailed analysis in the report examines the impact of ageing and population size on algorithm performance. It quantifies the increase in false negative identification rates (FNIR) due to ageing and demonstrates how FNIR and false positive identification rates (FPIR) change with increasing population sizes. This information is important for understanding the scalability and long-term reliability of face recognition systems.

The evaluation also addresses the challenges of identifying twins and lookalikes, which can cause higher false positive rates. Specific analysis shows how algorithms handle the presence of twins and lookalikes in the dataset, offering insights for applications where these factors are critical.

The report emphasizes the importance of human review in investigational applications and notes that real-world databases may contain images with various anomalies, such as rotated images or images with multiple faces, which can affect overall error rates.

Performance figures for prototype algorithms from a substantial majority of the face recognition industry, including commercial developers and a few academic institutions, are included in the report. This wide participation ensures a comprehensive benchmark for comparing different algorithms. And NIST emphasizes that its report is continuously updated as new algorithms are submitted and new datasets are evaluated, ensuring it’s getting up-to-date information on the latest advancements in face recognition technology.

Source: NIST

–

July 11, 2024 – by Cass Kennedy and Alex Perala