NISTIR 8311 Ongoing Face Recognition Vendor Test (FRVT) Part 6A: Face recognition accuracy with masks using pre-COVID-19 algorithms Mei Ngan Patrick Grother Kayee Hanaoka This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 NISTIR 8311 Ongoing Face Recognition Vendor Test (FRVT) Part 6A: Face recognition accuracy with masks using pre-COVID-19 algorithms Mei Ngan Patrick Grother Kayee Hanaoka Information Access Division Information Technology Laboratory This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 July 2020 U.S. Department of Commerce Wilbur L. Ross, Jr., Secretary National Institute of Standards and Technology Walter Copan, NIST Director and Undersecretary of Commerce for Standards and Technology Certain commercial entities, equipment, or materials may be identified in this document in order to describe an experimental procedure or concept adequately. Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended to imply that the entities, materials, or equipment are necessarily the best available for the purpose. National Institute of Standards and Technology Interagency or Internal Report 8311 Natl. Inst. Stand. Technol. Interag. Intern. Rep. 8311, 58 pages (July 2020) This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS i Executive Summary This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 OVERVIEW This is the first of a series of reports on the performance of face recognition algorithms on faces occluded by protective face masks [2] commonly worn to reduce inhalation of viruses or other contaminants. This study is being run under the Ongoing Face Recognition Vendor Test (FRVT) executed by the National Institute of Standards and Technology (NIST). This report documents accuracy of algorithms to recognize persons wearing face masks. The results in this report apply to algorithms provided to NIST before the COVID-19 pandemic, which were developed without expectation that NIST would execute them on masked face images. NIST had informed the FRVT developer community of our intent to run existing algorithms on masked images prior to the outset of this study and invited submission of mask-enabled algorithms for the next phase of this work. This report is intended to support end-users to understand how a pre-pandemic algorithm might be affected by the arrival of a substantial number of subjects wearing face masks. The next report will document accuracy values for more recent algorithms, some developed with capabilities for recognition of masked faces. The algorithms tested were one-to-one algorithms submitted to the FRVT 1:1 Verification track. Future iterations of this document will also report accuracy of one-to-many algorithms. MOTIVATION Traditionally, face recognition systems (in cooperative settings) are presented with mostly nonoccluded faces, which include primary facial features such as the eyes, nose, and mouth. However, there are a number of circumstances in which faces are occluded by masks such as in pandemics, medical settings, excessive pollution, or laboratories. Inspired by the COVID-19 pandemic response, the widespread requirement that people wear protective face masks in public places has driven a need to understand how cooperative face recognition technology deals with occluded faces, often with just the periocular area and above visible. An increasing number of research publications have surfaced on the topic of face recognition on people wearing masks along with face-masked research datasets [7]. Several commercial providers have recently developed ”face mask capable” face recognition systems which were not tested in this report. Results for such face mask capable or post-COVID algorithms will be published in the next report of this face mask evaluation series. This report documents results for pre-COVID algorithms developed primarily for non-covered faces, comparing an unmasked portrait quality enrollment image to a synthetically-masked webcam probe image. To date, we are not aware of any large-scale, independent, and publicly reported evaluation on the effects of face mask occlusion on face recognition. WHAT WE DID The NIST Information Technology Laboratory (ITL) quantified the accuracy of pre-COVID face recognition algorithms on faces occluded by masks applied digitally to a large set of photos that has been used in an FRVT verification benchmark since 2018. These algorithms were submitted to FRVT 1:1 prior to the COVID-19 pandemic and were developed without expectation that NIST would execute them on masked face images. Using the original unmasked images to form a baseline for accuracy, we measured the impact of occlusion by digitally applying a mask to the face and varying mask shape, mask color, and nose coverage. We used these algorithms with two large datasets of photographs collected in U.S. governmental applications that are currently in operation: unmasked application photographs from a global population of applicants for immigration benefits and digitally-masked border crossing photographs of travelers entering the United States. Both datasets were collected for authorized travel or immigration processes. The application photos (used as reference images) have good compliance with image capture standards. The digitally-masked border crossing photos (used as probe images) are not in good compliance with image capture standards given constraints on capture duration and environment. The application photos were left unmasked, and synthetic masks were applied to the border crossing photos. This mimics an operational scenario where a person wearing a mask attempts to authenticate against a prior visa or passport photo. Together these datasets allowed us to process a total of 6.2 million images of 1 million people through 89 algorithms. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT WHAT WE DID ( CONTINUED ) This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 WHAT WE FOUND - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS ii Our use of software to apply masks to face images has the following advantages: it allows very rapid characterization of the effect of masks on face recognition; it allows controlled exploration of factors such as mask size, shape, and color; it affords repeatability, which is key to the fair comparison of algorithms; it scales to very large datasets - in our study, some 6.2 million photographs - which allows fine-grained characterization of false positive rates in addition to false negative rates. Inversely, our use of digital masks presents a number of limitations: our digital masks are tailored to faces whereas realistically, mass-produced real masks may fit differently on different people; our use of uniformlycolored masks does not capture the impact of mask texture or pattern on face recognition; we were not able to pursue an exhaustive simulation of the endless variations in color, design, shape, texture, bands, and ways masks can be worn; our study does not capture any camera-mask interactions which may cause over or underexposure of the periocular region or face detection issues. Please see the Limitations section of this executive summary for a more detailed discussion on the limitations of this study. The following results apply to algorithms provided to NIST before the COVID-19 pandemic, which were developed without expectation that NIST would execute them on masked face images. The study has certain limitations, and these are discussed in the next section. . False rejection performance: All algorithms ive increased false non-match rates when the probes are masked. Using border crossing images, without masks, the most accurate algorithms will fail to authenticate about 0.3% of persons while falsely accepting no more than 1 in 100000 impostors (i.e. FNMR= 0.003 at FMR= 0.00001). With the highest coverage mask we tested and the most accurate algorithms, this failure rate rises to about 5% (FNMR = 0.05). This is noteworthy given that around 70% of the face area is occluded by the mask. However, many algorithms are much less tolerant: some algorithms that are quite competitive with unmasked faces (FNMR < 0.01) fail to authenticate between 20% and 50% of images (FNMR → 0.5). See Table 3 and Figure 3 In cooperative access control applications, false negatives can traditionally be remedied by users making second attempts. This is effective when users correct pose, expression, or illumination aspects of their presentation. With masked faces, however, a second attempt may not be effective if the failure is a systematic property of the algorithm. . False acceptance performance: As most systems are configured with a fixed threshold, it is necessary to report both false negative and false positive rates for each group at that threshold. In most cases false match rates are reduced by masks. The effect is generally modest with reductions in FMR usually being smaller than a factor of two. This property is valuable in that masks do not impart adverse false match security consequences for verification. See Figure 27 . Coverage of the masks: Unsurprisingly masks that occlude more of the face give larger false nonmatch rates. We surveyed over the extent to which the mask covers the nose, from not at all (“low”) to typical (“medium”) to near the eyes (“high”). We baselined those with unmasked faces with the result that FNMR increases by factors of around 10, 25, and 36 respectively for the median algorithm. However, as noted, algorithms vary considerably in the their tolerance of coverage. Readers should consult tabulated values for specific algorithms. See Table 3 and Figure 3 We included the “low” option not because it is a common position for a mask but as an option for authentication applications where it would be tenable to ask the user to pull the mask down to just below the nose for the duration of the authentication attempt. . Shape of the masks: The shape of the masks matters. Full-face-width masks generally cover more of the face than rounder N95 type masks. Results show that wide-width masks generally give false negative rates about a factor of two higher than do rounder type masks. See Figure 14 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT WHAT WE FOUND ( CONTINUED ) - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS iii . Color of the masks: We considered light-blue and black masks. Most algorithms have higher error rates in black masks than light-blue masks. The reason for observed accuracy differences between mask color is unknown but is a point for consideration by impacted developers. Mask color also affects the rate at which some algorithms fail to produce a template from an image. See Table 5 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 . Failure to detect and template: The false negative rates in this report include the effects of both face detection and localization errors, and low-similarity matching errors. We separately include tables detailing how often an algorithm does not make a template from an input image. This can occur because the algorithm doesn’t detect a face, or electively chooses not to extract features from it. While many algorithms give low failure-to-template rates, some give high values ranging up to 100%. Inversely, the successful creation of a template does not guarantee proper facial localization (e.g. algorithms may incorrectly detect something that’s not a face). Such localization failures will not be captured as a failure to detect and template event but will impact accuracy rates nonetheless. See Table 5 LIMITATIONS As a simulation, this study likely doesn’t fully capture the effects of masks on face recognition. Particularly the following points should be weighed by readers in the near term. Some of these will be addressed in subsequent work at NIST. . Evaluate “mask-enabled” algorithms: The algorithms used so far were submitted to the FRVT by corporate research and development laboratories and a few universities in 2019 and early 2020. Several of the algorithms were submitted to NIST as recently as March 2020, but because the algorithms were developed without expectation that NIST would run them on faces occluded by masks, we consider all algorithms evaluated here as “pre-pandemic”. . Apply masks to both photos: We masked only the probe image. We did not mask the reference photo. This situation represents authentication against an unmasked photo drawn from a prepandemic credential (e.g. passport) or database. While in some applications masks could appear on both enrollment and recognition images, we anticipate “mask-enabled” algorithms will need to extract and compare features from all combinations of masked and unmasked photos. . Train algorithms: As with all NIST evaluations, we regard the software as a black box whose parameters (models) remain fixed for the entirety of its use without learning from the test data. We do not train or fine-tune algorithms. . Evaluate one-to-many algorithms: We have only run one-to-one verification algorithms with masks. This elicits data on the effect of masks on the underlying feature extraction and discrimination of algorithms and can therefore be be expected to give first-order indications of the effect on one-to-many identification algorithms. . Consider the effect of eye occlusion: We did not address the effect of eye-glasses or eye-protection. While our dataset includes examples of people wearing glasses, we didn’t collect such data nor simulate it with digital addition. . Test with images of real masks: Given time and resource constraints, we didn’t collect photos of subjects wearing masks. The possible downsides of this are several. First, our digital masks are tailored to faces; while a few don’t fit realistically, mass-produced real masks may not fit all actual persons correctly either. Second, because many cameras run with exposure-control, it is possible that a dark mask will cause less light to be reflecting and the camera to increase gain on the sensor causing overexposure of the periocular region. Likewise a white mask could lead to underexposure problems. Third, it is possible that some cameras that include a face detector, may fail to focus or acquire a masked face correctly. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT LIMITATIONS ( CONTINUED ) - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS iv . Use textured masks: All masks synthesized by NIST in this study have a uniform color. The consequences of this are that we do not capture the the increasing diversity of masks worn recently, including those with corporate logos, text, patterns, or those advertised to thwart face recognition. The possibility exists for patterned masks to induce higher facial localization errors, which is not captured in our current study. We received a suggestion that such information may serve as a soft biometric, in that a subject that always wears the same textured mask will be more identifiable. We don’t intend to encourage algorithm development along this line, because as mass-produced high-efficacy masks become more common, mask diversity may actually drop. This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 . Study demographic effects on masked images: This report estimates overall performance of existing algorithms on recognition of faces occluded by masks. We deferred tabulating accuracy for different demographic groups until more capable mask-enabled algorithms have been submitted to FRVT. . Evaluate algorithms on non-cooperative, unconstrained imagery: This report documents results for matching masked webcam images to unmasked portrait-style photos. While the properties of the two sets of images differ, subjects are operating in cooperative mode and are for the most part, looking at the camera. . Consider effects of human examination: This report does not consider the various ways humans are involved in face recognition systems. For example, analysts can correct face detection or localization errors induced by masks, prior to automated recognition. Likewise, humans are often tasked with adjudication of images following a rejection or other exception from an automated system. Analysis of human capability and role is pertinent to those operations, but is beyond the scope of this study. IMPLICATIONS AND FUTURE WORK Know Your Algorithm: Operational implementations usually employ a single face recognition algorithm. Given algorithm-specific variation, it is incumbent upon the system owner to know their algorithm. While publicly available test data from NIST and elsewhere can inform owners, it will usually be informative to specifically measure accuracy of the operational algorithm on the operational image data collected with actual masks. NIST plans on releasing a series of reports, iteratively assessing different aspects and use cases of face masking on recognition performance. In the near term, we anticipate the next report in this series to evaluate the performance of “mask-enabled” algorithms submitted to FRVT. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS v ACKNOWLEDGMENTS This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 This work was conducted in collaboration with the Department of Homeland Security’s Science & Technology Directorate (S&T), Office of Biometric Identity Management (OBIM), and Customs and Border Protection (CBP). Additionally, the authors are grateful to staff in the NIST Biometrics Resesarch Laboratory for infrastructure supporting rapid evaluation of algorithms. DISCLAIMER Specific hardware and software products identified in this report were used in order to perform the evaluations described in this document. In no case does identification of any commercial product, trade name, or vendor, imply recommendation or endorsement by the National Institute of Standards and Technology, nor does it imply that the products and equipment identified are necessarily the best available for the purpose. INSTITUTIONAL REVIEW BOARD The National Institute of Standards and Technology’s Research Protections Office reviewed the protocol for this project and determined it is not human subjects research as defined in Department of Commerce Regulations, 15 CFR 27, also known as the Common Rule for the Protection of Human Subjects (45 CFR 46, Subpart A). 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS vi Contents This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 E XECUTIVE S UMMARY I A CKNOWLEDGMENTS V D ISCLAIMER V I NSTIUTIONAL R EVIEW B OARD V 1 FACE M ASK E FFECTS 1.1 S TATUS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 I NTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1 1 2 I MAGE D ATASETS 2.1 A PPLICATION I MAGES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 W EBCAM I MAGES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 S YNTHETICALLY M ASKED I MAGES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2 2 2 3 M ETRICS 3.1 M ATCHING ACCURACY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 FAILURE TO E NROLL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 4 4 4 A LGORITHMS 4 5 R ESULTS 7 A PPENDIX A D LIB M ASKING M ETHODOLOGY 48 List of Tables 1 2 3 4 5 6 7 A LGORITHM SUMMARY . . A LGORITHM SUMMARY . . FALSE N ON -M ATCH R ATE FALSE N ON -M ATCH R ATE FAILURE TO E NROL R ATES FAILURE TO E NROL R ATES FAILURE TO E NROL R ATES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 . 6 . 8 . 9 . 23 . 24 . 25 E NROLLMENT IMAGE EXAMPLES . S YNTHETIC FACE MASK EXAMPLES DET UNMASKED VERSUS MASKED DET UNMASKED VERSUS MASKED DET UNMASKED VERSUS MASKED DET UNMASKED VERSUS MASKED DET UNMASKED VERSUS MASKED DET UNMASKED VERSUS MASKED DET UNMASKED VERSUS MASKED DET UNMASKED VERSUS MASKED DET UNMASKED VERSUS MASKED DET UNMASKED VERSUS MASKED FNMR GAIN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . List of Figures 1 2 3 4 5 6 7 8 9 10 11 12 13 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” 2 3 10 11 12 13 14 15 16 17 18 19 20 FRVT This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS FNMR GAIN . . . . . . . . . . FNMR GAIN . . . . . . . . . . R OLE OF FTE . . . . . . . . . FNMR C ALIBRATION C URVES FNMR C ALIBRATION C URVES FNMR C ALIBRATION C URVES FNMR C ALIBRATION C URVES FNMR C ALIBRATION C URVES FNMR C ALIBRATION C URVES FNMR C ALIBRATION C URVES FNMR C ALIBRATION C URVES FNMR C ALIBRATION C URVES FNMR C ALIBRATION C URVES FMR C ALIBRATION C URVES . FMR C ALIBRATION C URVES . FMR C ALIBRATION C URVES . FMR C ALIBRATION C URVES . FMR C ALIBRATION C URVES . FMR C ALIBRATION C URVES . FMR C ALIBRATION C URVES . FMR C ALIBRATION C URVES . FMR C ALIBRATION C URVES . FMR C ALIBRATION C URVES . D LIB MASKING METHODOLOGY 2020/07/24 11:10:31 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . FNMR(T) FMR(T) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . “False non-match rate” “False match rate” 21 22 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 48 FRVT 1 1.1 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS 1 Face Mask Effects Status This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 NIST has conducted the first out of a series of tests aimed at quantifying face recognition accuracy for people wearing masks. Our initial approach has been to apply masks to faces digitally (i.e., using software to apply a synthetic mask). This allowed us to leverage large datasets that we already have. This initial report documents results for 1:1 verification algorithms. We tested algorithms that were already submitted to FRVT 1:1 prior to mid-March 2020. The results in this report apply to algorithms provided to NIST before the COVID-19 pandemic and for which developers had no expectation that NIST would execute them on masked face images. This report is intended to support end-users to understand how a pre-pandemic algorithm might be affected by the arrival of substantial number of subjects wearing face masks. The next report will document accuracy values for more recent algorithms, some developed with capabilities for recognition of masked faces. These initial results reflect the case when only the probe image is masked and the reference photo is left as-is. Future reports will consider the effect of masking both enrollment and verification images. This report quantifies the effect of masks on both false negative and false positives match rates. The FRVT evaluation is an ongoing test that remains open to new participation. Comments and suggestions should be directed to frvt@nist.gov. 1.2 Introduction The majority of face recognition systems have been deployed in applications where subjects make cooperative presentations to a camera, for example as part of an application for a benefit or ID credential, or as during access control. With very few exceptions such images would not include face masks or other occlusions. However, with the SARS-CoV-2 pandemic, we can anticipate a demand to authenticate persons wearing masks, for example in immigration settings, without the need to the subjects to remove those masks. This presents a problem for face recognition, because regions of the face occluded by masks - the mouth and nose - include information useful for both recognition and, potentially, the detection stage that precedes it. Previous work on face recognition of occluded faces has been directed at situations such as crime scenes where subjects were actively un-cooperative i.e. acting to evade face detection and recognition. Those applications are often characterized by image properties (low resolution, video compression, uncontrolled head orientation) that are known [4] to degrade recognition accuracy. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT 2 2.1 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS 2 Image Datasets Application Images This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 The images are collected in an attended interview setting using dedicated capture equipment and lighting. The images, at size 300x300 pixels. The images are all high-quality frontal portraits collected in immigration offices and with a white background. As such, potential quality related drivers of high false match rates (such as blur) can be expected to be absent. The images are encoded as ISO/IEC 10918 i.e. JPEG. Over a random sample of 1000 images, the images have compressed file sizes (mean: 42KB, median: 58KB, 25-th percentile: 15KB, and 75-th percentile: 66KB). The implied bitrates are mostly benign and superior to many e-Passports. When these images are provided as input into the algorithm, they are labeled with the type ”ISO”. This report used 1 019 232 application images. Figure 1: Examples of images with properties similar to the enrollment application photos used in this study. The subjects in the photos are all NIST employees. 2.2 Webcam Images These images are taken with a camera oriented by an attendant toward a cooperating subject. This is done under time constraints, so there are roll, pitch, and yaw angle variations. Also, background illumination is sometimes bright, so the face is under exposed. Sometimes, there is perspective distortion due to close range images. The images are generally in poor conformance with the ISO/IEC 19794-5 Full Frontal image type. The images have mean interocular distance of 38 pixels. The images are all live capture. When these images are provided as input into the algorithm, they are labeled with the type ”WILD”. Examples of such images are included in Figure 2 and Figure 4 in NIST Interagency Report 8271. Results for verification of these images (unmasked) appear in FRVT Part 1 - Verification both compared against images of the same type, and with those described in section 2.1. This report used 5 225 633 border webcam images. 2.3 Synthetically Masked Images In this test, synthetically-generated masks were overlaid on top of all probe images, which in this case, were webcam images described in Section 2.2. The Dlib [5] C++ toolkit version 19.19 was used to detect and establish key facial points on the face, and with the facial points, solid masks of different shape, height, and color were drawn on the face. The exact Dlib facial points and details used to generate the masks are documented in Appendix A. In the event that Dlib was unable to detect a face in the image, eye coordinates were used to generate a mask leveraging standardized token frontal geometry [1]. Examples of unmasked enrollment images and synthetically-masked probe images are presented in Figures 1 and 2, respectively. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS 3 1 ORIGINAL IMAGE 2 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 WIDE , HIGH COVERAGE 3 WIDE , MEDIUM COVERAGE 4 WIDE , LOW COVERAGE 5 ROUND , HIGH COVERAGE 6 AS ROW 3 IN BLACK Figure 2: Examples of synthetically-generated face masks used in this study. The original images are from the NIST MEDS-II Dataset [3]. They were collected in operational settings using the same camera and procedure as is used for the border images that form the mainstay of the experiments in this report. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT 3 3.1 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS 4 Metrics Matching accuracy Given a vector of N genuine scores, u, the false non-match rate (FNMR) is computed as the proportion below some threshold, T: N 1 X FNMR(T ) = 1 − H(ui − T ) (1) N i=1 where H(x) is the unit step function, and H(0) taken to be 1. This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 Similarly, given a vector of N impostor scores, v, the false match rate (FMR) is computed as the proportion above T: FMR(T ) = N 1 X H(vi − T ) N i=1 (2) The threshold, T, can take on any value. We typically generate a set of thresholds from quantiles of the observed impostor scores, v, as follows. Given some interesting false match rate range, [FMRL , FMRU ], we form a vector of K thresholds corresponding to FMR measurements evenly spaced on a logarithmic scale Tk = Qv (1 − FMRk ) (3) where Q is the quantile function, and FMRk comes from log10 FMRk = log10 FMRL + k [log10 FMRU − log10 FMRL ] K (4) Error tradeoff characteristics are plots of FNMR(T) vs. FMR(T). These are plotted with FMRU → 1 and FMRL as low as is sustained by the number of impostor comparisons, N. This is somewhat higher than the “rule of three” limit 3/N because samples are not independent, due to re-use of images. 3.2 Failure to Enroll Failure to enroll (FTE) is the proportion of failed template generation attempts. Failures can occur because the software throws an exception, or because the software electively refuses to process the input image. This would typically occur if a face is not detected. FTE is measured as the number of function calls that give EITHER a non-zero error code OR that give a ”small” template. This is defined as one whose size is less than 60 bytes. This second rule is needed because some algorithms incorrectly fail to return a non-zero error code when template generation fails yet do return a valid default data structure. The effects of FTE are included in the accuracy results of this report by regarding any template comparison involving a failed template to produce a low similarity score. Thus higher FTE results in higher FNMR and lower FMR. 4 Algorithms The FRVT activity is open to participation worldwide, and the test will evaluate submissions on an ongoing basis. There is no charge to participate. The process and format of algorithm submissions to NIST are described in the FRVT 1:1 Verification Application Programming Interface (API) [6] document. Participants provide their submissions in the form of libraries compiled on a specific Linux kernel, which are linked against NIST’s test harness to produce executables. NIST provides a validation package to participants to ensure that NIST’s execution of submitted libraries produces the expected output on NIST’s test machines. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS 5 This report documents the results of algorithms submitted to FRVT 1:1 for testing from April 2019 to March 2020, without specific claims to being able to recognize people wearing face masks. Table 2 lists the algorithms that were tested. Note that algorithms that expired prior to June 2020 were not included in this report. This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 Developer Algorithm Submission Date 3Divi ADVANCE.AI ASUSTek Computer Inc Ability Enterprise Co. Ltd - Andro Video Acer Incorporated Ai First AiUnion Technology Co Ltd AlphaSSTG Anke Investments Antheus Technologia Ltda Aware Awidit Systems Beijing Alleyes Technology Co Ltd BioID Technologies SA CSA IntelliCloud Technology CTBC Bank Co Ltd Camvi Technologies Canon Information Technology (Beijing) Co Ltd China University of Petroleum Chinese Univeristy of Hong Kong Chosun University Chunghwa Telecom Co. Ltd Cyberlink Corp DSK Dahua Technology Co Ltd Deepglint DiDi ChuXing Technology Co Expasoft LLC FaceSoft Ltd Fujitsu Research and Development Center Co Ltd Glory Ltd Gorilla Technology Guangzhou Pixel Solutions Co Ltd ITMO University Idemia Imagus Technology Pty Ltd Imperial College London Incode Technologies Inc Innovative Technology Ltd Innovatrics Institute of Information Technologies Intel Research Group Intellivision Kakao Enterprise Kedacom International Pte Kneron Inc Lomonosov Moscow State University Lookman Electroplast Industries Luxand Inc MVision Momentum Digital Co Ltd Moontime Smart Technology N-Tech Lab Netbridge Technology Incoporation Neurotechnology Nodeflux NotionTag Technologies Private Limited 3divi-004 advance-002 asusaics-000 androvideo-000 acer-000 aifirst-001 aiunionface-000 alphaface-002 anke-005 antheus-000 aware-005 awiros-001 alleyes-000 bioidtechswiss-000 intellicloudai-001 ctbcbank-000 camvitech-004 cib-000 upc-001 cuhkee-001 chosun-000 chtface-002 cyberlink-004 dsk-000 dahua-004 deepglint-002 didiglobalface-001 expasoft-000 facesoft-000 fujitsulab-000 glory-002 gorilla-005 pixelall-003 itmo-007 Idemia-005 imagus-001 imperial-002 incode-006 innovativetechnologyltd-002 innovatrics-006 iitvision-002 intelresearch-001 intellivision-002 kakao-003 kedacom-000 kenron-005 intsysmsu-002 lookman-004 luxand-000 mvision-001 sertis-000 mt-000 ntech-008 netbridgetech-001 neurotech-008 nodeflux-002 notiontag-000 2019-07-22 2019-12-19 2019-10-24 2020-02-03 2020-01-08 2019-11-21 2019-10-22 2020-02-20 2019-11-21 2019-12-05 2020-02-27 2019-09-23 2020-03-09 2019-11-15 2019-08-13 2019-06-28 2019-07-12 2019-12-11 2019-06-05 2020-03-18 2020-02-12 2019-12-07 2020-02-27 2019-06-28 2019-12-18 2019-11-15 2019-10-23 2020-01-06 2019-07-10 2020-02-04 2019-11-12 2020-03-11 2019-10-15 2020-01-06 2019-10-11 2019-10-22 2019-08-28 2020-02-20 2020-02-26 2019-08-13 2019-12-04 2020-01-14 2019-08-23 2020-02-26 2019-06-03 2020-02-21 2020-03-12 2019-06-03 2019-11-07 2019-11-12 2019-10-07 2019-06-03 2020-01-06 2020-01-08 2020-01-08 2019-08-13 2019-06-12 Table 1: List of algorithms included in this report. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS 6 Developer Algorithm Submission Date Panasonic R+D Center Singapore Paravision (EverAI) Rank One Computing Remark Holdings Rokid Corporation Ltd Samsung S1 Corp Scanovate Ltd Sensetime Group Ltd Shanghai Jiao Tong University Shanghai Ulucu Electronics Technology Co. Ltd Shanghai Universiy - Shanghai Film Academy Shenzhen AiMall Tech Ltd Shenzhen Intellifusion Technologies Co Ltd Star Hybrid Limited Synology Inc TUPU Technology Co Ltd Taiwan AI Labs Tech5 SA Tencent Deepsea Lab Tevian Trueface.ai Universidade de Coimbra Via Technologies Inc Videmo Intelligente Videoanalyse Videonetics Technology Pvt Ltd Vigilant Solutions VisionLabs Vocord Winsense Co Ltd X-Laboratory Xforward AI Technology Co LTD iQIYI Inc psl-004 paravision-004 rankone-008 remarkai-001 rokid-000 s1-001 scanovate-001 sensetime-003 sjtu-002 uluface-002 shu-002 aimall-002 intellifusion-002 starhybrid-001 synology-000 tuputech-000 ailabs-001 tech5-004 deepsea-001 tevian-005 trueface-000 visteam-000 via-001 videmo-000 videonetics-002 vigilant-007 visionlabs-008 vocord-008 winsense-001 x-laboratory-001 xforwardai-000 iqface-000 2020-03-03 2019-12-11 2019-11-12 2019-11-21 2019-08-01 2019-12-06 2019-11-12 2019-06-04 2020-02-12 2019-07-10 2019-12-10 2020-03-12 2020-03-18 2019-06-19 2019-10-23 2019-10-11 2019-12-18 2020-03-09 2019-06-03 2019-09-21 2019-10-08 2020-01-14 2020-01-08 2019-12-19 2019-11-21 2019-06-27 2020-01-06 2020-01-031 2019-10-16 2020-01-21 2020-02-06 2019-06-04 Table 2: List of algorithms included in this report. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT 5 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS 7 Results This section includes accuracy results for the 89 one-to-one verification algorithms listed in section 4. We do not include speed and computational resource requirements - they are given in Table 1 in the FRVT 1:1 report. The results, which span many pages, are comprised of: . FNMR: Table 3 tabulates false non-match rates by color, shape and nose coverage. It includes also FNMR without any mask. FNMR values are stated at a fixed threshold calibrated to give FMR = 0.00001 on unmasked images. . DET: Figure 3 shows detection error trade of characteristics spanning false match rates from 3 10−7 to 1. This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 . Mask vs. no mask: The scatter plot in Figure 13 shows variation across all algorithms of FNMR without masks against FNMR with a common type of mask. . Mask shape: The scatter plot in Figure 14 shows for all algorithms the increase in false negative results for wide masks vs. narrower round masks. . Nose coverage: The scatter plot in Figure 15 shows for all algorithms the increase in false negative rates for masks that substantially cover the nose and those pulled beneath the nose. . FTE: Table 5 gives empirical failure-to-template results by color, shape, and nose coverage. The table was produced using 10 000 images of each kind of mask. . FTE as contributor to FNMR: The FNMR results include failure-to-template rates (FTE). Figure 16 shows the proportion of template generation failures. . FNMR vs. threshold: Figure 17 shows explicit dependence of false non-match rate on threshold. . FMR vs. threshold: Likewise Figure 27 shows explicit dependence of false match rate on threshold. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” NOT MASKED MASKED COLOR SHAPE COVERAGE = LIGHTBLUE = WIDE = BLACK = WIDE MASKED COLOR SHAPE = ROUND SHAPE MED HI LO MED HI LO MED HI 3divi-004 acer-000 advance-002 aifirst-001 ailabs-001 0.013049 0.843285 0.032869 0.007929 0.024362 0.412355 0.999568 0.077828 - 0.676068 0.999985 0.235124 0.256727 0.679269 - - - - - - - 6 7 8 9 10 aimall-002 aiunionface-000 alleyes-000 alphaface-002 androvideo-000 0.013351 0.009434 0.00447 1.000088 0.033370 0.091734 1.000070 0.317751 0.391944 0.293535 0.103810 1.000088 0.649865 - - - - - - - 0.01818 - 0.054210 - 0.105010 - 0.026211 - 0.128713 - 0.199113 - 11 12 13 14 15 anke-005 antheus-000 asusaics-000 aware-005 awiros-001 0.006223 0.731984 0.009033 0.030868 0.123376 0.067121 0.999467 0.496257 0.682360 0.320739 0.999984 0.361642 0.887675 0.863574 - - - - - - - 16 17 18 19 20 bioidtechswiss-000 camvi-004 chosun-000 chtface-002 cib-000 0.005010 0.006324 1.000089 0.010841 0.024963 0.030810 0.069723 1.000080 0.142339 0.075726 0.115512 0.217922 1.000089 0.430348 0.167016 0.184011 - 0.022311 - 0.063212 - 0.120712 - 0.033113 - 0.116311 - 0.178611 - 21 22 23 24 25 ctbcbank-000 cuhkee-001 cyberlink-004 dahua-004 deepglint-002 0.013350 0.00416 0.006121 0.00384 0.00395 0.159444 0.01435 0.053818 0.032812 0.00771 0.744873 0.05725 0.211521 0.178418 0.02371 0.09635 0.202613 0.04551 0.01434 0.00781 0.03333 0.01411 0.07153 0.02921 0.01644 0.02267 0.00831 0.06524 0.118612 0.02541 0.11934 0.198312 0.05131 26 27 28 29 30 deepsea-001 didiglobalface-001 dsk-000 expasoft-000 facesoft-000 0.011043 0.005011 0.196177 0.051975 0.005716 0.121837 0.910863 0.318652 0.039713 0.309437 0.09869 0.992980 0.679670 0.142814 0.377817 0.15179 - 0.092218 0.025512 - 0.221719 0.05159 - 0.446918 0.09798 - 0.029112 - 0.10339 16 0.1573 0.15589 - 31 32 33 34 35 fujitsulab-000 glory-002 gorilla-005 idemia-005 iit-002 0.018059 0.010942 0.011746 0.011144 0.014155 0.146341 0.205146 - 0.505257 0.272933 0.503755 0.646964 0.307836 - - - - - - 0.696821 - 0.134919 - 0.438721 - - 0.278621 - 0.740224 - 0.811920 - 36 37 38 39 40 imagus-001 imperial-002 incode-006 innovativetechnologyltd-002 innovatrics-006 0.027665 0.005513 0.009536 0.025164 0.005919 0.348854 0.032011 0.270149 0.054320 0.651066 0.135013 0.372543 0.645463 0.221023 0.197212 0.311815 0.025813 0.036915 0.077513 0.110916 0.155613 0.198415 0.035914 0.055717 0.151015 0.190919 0.230215 0.276417 41 42 43 44 45 intellicloudai-001 intellifusion-002 intellivision-002 intelresearch-001 intsysmsu-002 0.009535 0.005615 0.046374 0.022061 0.008932 0.104436 0.053919 0.599958 0.225447 0.082731 0.439450 0.169017 0.902876 0.618461 0.313838 - - - - - 0.182218 - - - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS “False non-match rate” “False match rate” LO 1 2 3 4 5 FRVT FNMR(T) FMR(T) This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 2020/07/24 11:10:31 Algorithm Name Table 3: This table summarizes False Non-Match Rate (FNMR) on unmasked and masked probe images. FNMR is the proportion of mated comparisons below a threshold set to achieve FMR=1e-05 on unmasked probe images. False Match Rate (FMR) is the proportion of impostor comparisons at or above that threshold. The red superscripts give rank over all algorithms in that column. Missing entries generally mean the algorithm was not run on that particular mask variation due to time and resource constraints. Algorithms with FTE=1.00 were not run at all. 8 NOT MASKED MASKED COLOR SHAPE COVERAGE = WIDE = LIGHTBLUE SHAPE = MASKED COLOR = BLACK SHAPE = WIDE ROUND MED HI LO MED HI LO MED HI iqface-000 itmo-007 kakao-003 kedacom-000 kneron-005 0.012848 0.009838 0.017058 0.039171 0.029667 0.088533 0.084032 0.154143 0.344453 - 0.286734 0.268531 0.412346 0.618862 0.456752 - - - 0.684820 - 0.266321 - 0.597522 - - - - - 51 52 53 54 55 lookman-004 luxand-000 mt-000 mvision-001 netbridgetech-001 0.039872 0.216779 0.007528 0.013753 0.267381 0.973264 0.076827 0.894062 0.652067 0.998881 0.270032 0.398745 0.987879 - - - - - - 0.373616 - 0.048216 - 0.174617 - - 0.074918 - 0.308420 - 0.423918 - 56 57 58 59 60 neurotechnology-008 nodeflux-002 notiontag-000 ntechlab-008 paravision-004 0.010039 0.042473 0.681483 0.00331 0.008831 0.079429 0.417756 0.996666 0.01796 0.01242 0.345041 0.730772 0.999282 0.06427 0.02812 0.446018 0.11267 2 0.0476 0.081817 0.01373 2 0.0125 0.183418 0.04137 2 0.0181 0.312717 0.09537 2 0.0313 0.095319 0.02086 2 0.0135 0.489321 0.08428 2 0.0327 0.547219 0.13488 2 0.0581 61 62 63 64 65 pixelall-003 psl-004 rankone-008 remarkai-002 rokid-000 0.008630 0.005920 0.013452 0.007326 0.011745 0.074624 0.044914 0.241648 0.068522 0.144840 0.268029 0.186219 0.547058 0.235225 0.434649 - - 0.620119 - 0.184820 - 0.108215 0.380120 - 0.225616 0.737919 - 0.047316 0.231420 - 0.173917 0.668423 - 0.230916 0.962521 - 66 67 68 69 70 s1-001 scanovate-001 sensetime-003 sertis-000 shu-002 0.027766 0.240380 0.00459 0.006625 1.000087 0.677659 0.01857 0.075125 - 0.945977 0.597360 0.05444 0.268530 1.000087 - - - - - - - 0.09124 - 0.022110 - 0.03654 - 0.07394 - 0.02329 - 0.06545 - 0.12305 - 71 72 73 74 75 sjtu-002 starhybrid-001 synology-000 tech5-004 tevian-005 0.005212 0.010440 0.012347 0.00458 0.006122 0.047516 0.192345 0.02188 0.096135 0.191220 0.503354 0.445951 0.08398 0.504456 - - - - - - - 0.13898 - 0.01726 - 0.04648 - 0.09056 - 0.02288 - 0.08187 0.617822 0.12887 - 76 77 78 79 80 trueface-000 tuputech-000 uluface-002 upc-001 via-001 0.014356 0.201478 0.007327 0.016757 0.009737 0.151242 0.874361 0.079630 0.123438 0.416447 0.973178 0.245026 0.472353 0.340640 - - - - - - - 81 82 83 84 85 videmo-000 videonetics-002 vigilantsolutions-007 visionlabs-008 visteam-000 0.014054 0.603282 0.019460 0.00342 0.996086 0.994165 0.284950 0.01393 1.000069 0.550959 0.999683 0.683971 0.05796 1.000086 - - - - - - - 0.10146 - 0.01545 - 0.04126 - 0.10049 - 0.01875 - 0.06646 - 0.12846 - 86 87 88 89 vocord-008 winsense-001 x-laboratory-001 xforwardai-000 0.00383 0.005817 0.005818 0.005614 0.01404 0.047315 0.051717 0.02359 0.05003 0.162615 0.256928 0.106411 0.07623 0.224414 0.161510 0.01767 0.032514 0.01979 0.03935 0.094614 0.060611 0.08925 0.185314 0.115611 0.01353 0.040615 0.025510 0.04593 0.147114 0.109110 0.07713 0.223114 0.160810 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS “False non-match rate” “False match rate” LO 46 47 48 49 50 FRVT FNMR(T) FMR(T) This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 2020/07/24 11:10:31 Algorithm Name Table 4: This table summarizes False Non-Match Rate (FNMR) on unmasked and masked probe images. FNMR is the proportion of mated comparisons below a threshold set to achieve FMR=1e-05 on unmasked probe images. False Match Rate (FMR) is the proportion of impostor comparisons at or above that threshold. The red superscripts give rank over all algorithms in that column. Missing entries generally mean the algorithm was not run on that particular mask variation due to time and resource constraints. Algorithms with FTE=1.00 were not run at all. 9 FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS deepglint_002 paravision_004 10 sensetime_003 0.500 0.200 ● 0.100 0.050 ● ● 0.020 ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.005 ● ● 0.002 ● ● ● ● ● ● ● ● 0.001 Mask Color vocord_008 visionlabs_008 black cuhkee_001 lightblue 0.500 False non−match rate (FNMR) This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.010 ● ● not masked 0.200 ● ● 0.100 ● ● 0.050 ● ● ● ● ● 0.020 ● 0.010 ● ● Mask Shape ● ● ● ● ● ● wide ● ● ● ● ● ● not masked ● ● ● ● ● 0.005 round ● ● Mask Top Coverage 0.002 low 0.001 medium high ntechlab_008 tech5_004 alleyes_000 not masked 0.500 0.200 0.100 ● ● ● ● ● ● ● ● ● 0.050 ● ● 0.020 0.010 ● ● ● ● ● ● ● ● 0.005 0.002 ● ● ● ● ● ● ● 0.001 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e False match rate (FMR) Figure 3: DET curves showing error rates on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS didiglobalface_001 bioidtechswiss_000 11 xforwardai_000 0.500 0.200 ● 0.100 ● 0.050 ● ● ● ● ● ● ● ● ● 0.020 ● ● ● ● ● 0.005 ● ● ● ● 0.002 0.001 Mask Color imperial_002 psl_004 black dahua_004 lightblue 0.500 False non−match rate (FNMR) This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.010 0.200 not masked ● ● ● ● ● ● 0.100 0.050 ● Mask Shape wide ● 0.020 ● ● ● ● ● round not masked ● 0.010 0.005 ● Mask Top Coverage 0.002 low 0.001 medium high facesoft_000 winsense_001 ● cyberlink_004 not masked 0.500 ● 0.200 ● 0.100 ● 0.050 0.020 0.010 0.005 0.002 0.001 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e False match rate (FMR) ● Figure 4: DET curves showing error rates on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT sjtu_002 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS uluface_002 12 intellifusion_002 0.500 0.200 0.100 0.050 0.020 0.005 0.002 0.001 Mask Color camvi_004 remarkai_002 black innovatrics_006 lightblue 0.500 False non−match rate (FNMR) This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.010 not masked ● 0.200 ● Mask Shape 0.100 ● wide ● 0.050 ● ● 0.020 not masked ● 0.010 round ● ● 0.005 ● Mask Top Coverage 0.002 low 0.001 medium high intsysmsu_002 advance_002 pixelall_003 not masked 0.500 0.200 0.100 0.050 0.020 0.010 0.005 0.002 0.001 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e False match rate (FMR) Figure 5: DET curves showing error rates on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS aifirst_001 sertis_000 13 deepsea_001 ● 0.500 ● ● ● ● 0.200 0.100 ● 0.050 0.020 0.005 0.002 0.001 Mask Color x−laboratory_001 mt_000 iqface_000 black ● lightblue 0.500 False non−match rate (FNMR) This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.010 not masked ● 0.200 0.100 Mask Shape ● wide 0.050 ● ● round 0.020 not masked ● 0.010 ● 0.005 Mask Top Coverage ● 0.002 low 0.001 medium high aiunionface_000 glory_002 kakao_003 not masked 0.500 0.200 0.100 0.050 0.020 0.010 0.005 0.002 0.001 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e False match rate (FMR) Figure 6: DET curves showing error rates on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS anke_005 itmo_007 14 iit_002 0.500 0.200 0.100 0.050 0.020 0.005 0.002 0.001 Mask Color via_001 asusaics_000 black incode_006 lightblue 0.500 False non−match rate (FNMR) This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.010 not masked 0.200 Mask Shape 0.100 wide 0.050 ● round 0.020 not masked 0.010 0.005 Mask Top Coverage 0.002 low 0.001 medium high neurotechnology_008 aimall_002 rokid_000 not masked 0.500 ● ● 0.200 ● 0.100 0.050 0.020 0.010 0.005 ● ● ● 0.002 0.001 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e False match rate (FMR) Figure 7: DET curves showing error rates on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT chtface_002 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS mvision_001 15 cib_000 0.500 0.200 0.100 0.050 0.020 0.005 0.002 0.001 Mask Color synology_000 tevian_005 black kneron_005 lightblue 0.500 False non−match rate (FNMR) This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.010 not masked 0.200 Mask Shape 0.100 wide 0.050 ● round 0.020 not masked 0.010 0.005 Mask Top Coverage 0.002 low 0.001 medium high upc_001 fujitsulab_000 starhybrid_001 not masked 0.500 0.200 0.100 0.050 0.020 0.010 0.005 0.002 0.001 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e False match rate (FMR) Figure 8: DET curves showing error rates on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS intellicloudai_001 gorilla_005 16 lookman_004 0.500 0.200 0.100 0.050 0.020 0.005 0.002 0.001 Mask Color idemia_005 videmo_000 black kedacom_000 lightblue ● 0.500 False non−match rate (FNMR) This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.010 ● ● ● 0.200 ● not masked ● ● ● Mask Shape 0.100 ● wide 0.050 ● ● round 0.020 not masked 0.010 0.005 Mask Top Coverage ● 0.002 low ● 0.001 medium high rankone_008 ● ● 0.500 ● 0.200 ● ● 0.100 0.050 scanovate_001 intelresearch_001 not masked ● ● ● ● ● ● ● ● ● ● 0.020 0.010 0.005 0.002 0.001 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e False match rate (FMR) Figure 9: DET curves showing error rates on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT 3divi_004 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS ailabs_001 17 imagus_001 0.500 0.200 0.100 0.050 0.020 0.005 0.002 0.001 Mask Color vigilantsolutions_007 androvideo_000 black expasoft_000 lightblue 0.500 False non−match rate (FNMR) This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.010 not masked 0.200 Mask Shape 0.100 wide 0.050 ● round 0.020 not masked 0.010 0.005 Mask Top Coverage 0.002 low 0.001 medium high innovativetechnologyltd_002 nodeflux_002 ctbcbank_000 not masked 0.500 0.200 0.100 0.050 0.020 0.010 0.005 0.002 0.001 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e False match rate (FMR) Figure 10: DET curves showing error rates on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT aware_005 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS intellivision_002 18 awiros_001 0.500 0.200 0.100 0.050 0.020 0.005 0.002 0.001 Mask Color s1_001 trueface_000 black tuputech_000 lightblue 0.500 False non−match rate (FNMR) This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.010 not masked 0.200 Mask Shape 0.100 wide 0.050 ● round 0.020 not masked 0.010 0.005 Mask Top Coverage 0.002 low 0.001 medium high dsk_000 netbridgetech_001 luxand_000 not masked 0.500 0.200 0.100 0.050 0.020 0.010 0.005 0.002 0.001 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e False match rate (FMR) Figure 11: DET curves showing error rates on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT notiontag_000 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS videonetics_002 19 visteam_000 0.500 0.200 0.100 0.050 0.020 0.005 0.002 0.001 Mask Color antheus_000 acer_000 black alphaface_002 lightblue 0.500 False non−match rate (FNMR) This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.010 not masked 0.200 Mask Shape 0.100 wide 0.050 ● round 0.020 not masked 0.010 0.005 Mask Top Coverage 0.002 low 0.001 medium shu_002 chosun_000 7 6 6 5 5 4 4 3 3 2 2 1 1 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e high not masked 0.500 0.200 0.100 0.050 0.020 0.010 0.005 0.002 0.001 7 6 6 5 5 4 4 3 3 2 2 1 1 7 6 6 5 5 4 4 3 3 2 2 1 1 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 −0 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e 1e 3e False match rate (FMR) Figure 12: DET curves showing error rates on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” intelli1 idemia5 3divi4 ctbcban0 luxand0 intelre1 dsk0 s11 netbrid1 videone2 notiont0 0.80 chtface2 rankone8 gorilla5 videmo0 0.60 asusaic0 tevian5 ● False non−match rate with masks, FNMR(T0) sertis0 ● 0.30 ● sjtu2 intelli2 ● ● ● ● ● ● ● ● ● ● ● ● visteam0 alphafa2 chosun0 ● ● intelli2 ● ● ● ● ● ● ● innovat2 expasof0 ● ● kedacom0 ●● ● ● ● neurote8 ● upc1 ● imagus1 nodeflu2 ● ● aiunion0 iqface0 ● ● ●● rokid0 ● ● ● ● ● innovat6 ● ● mt0 acer0 kakao3 lookman4 ● remarka2 imperia2 dahua4 ● ● itmo7 bioidte0 0.10 ● ● ● 0.08 awiros1 winsens1 ● ● alleyes0 kneron5 mvision1 via1 cyberli4 androvi0 fujitsu0 didiglo1 ntechla8 cuhkee1 ● 0.06 ● 0.05 ● scanova1 uluface2 aimall2 cib0 ● ● xforwar0 tech54 visionl8 aifirst1 0.04 iit2 advance2 psl4 senseti3 vocord8 0.03 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS x−labor1 antheus0 FRVT intsysm2 synolog0 ● ● ● pixelal3 0.40 0.20 ● ● incode6 anke5 ● ● ● starhyb1 ● 0.50 ● aware5 ● ● ● tuputec0 ailabs1 vigilan7 ● FNMR(T) FMR(T) “False non-match rate” “False match rate” This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 2020/07/24 11:10:31 Impact of medium wide lightblue masks The lower line is y = x; the upper line is y = 27.9x shu2 glory2 ● ● deepgli2 paravis4 facesof0 camvi4 deepsea1 truefac0 0.02 0.003 0.005 0.010 0.020 0.030 0.050 False non−match rate without masks, FNMR(T0) with FMR(T0) = 0.000010 20 Figure 13: At a fixed threshold, a plot of FNMR with and without masks. The displacement of the red line relative to the black “parity” line shows a large increase in FNMR with masks. The value in the title is the median increase multiplier. 0.50 ● ● ● 0.40 kedacom0 idemia5 neurote8 ● 0.30 ● - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS False non−match rate with wide masks, FNMR(T0) ● 0.20 ● didiglo1 deepsea1 ● imperia2 ● 0.10 bioidte0 alleyes0 psl4 ● mt0 ● ● 0.08 winsens1 ● ● senseti3 cuhkee1 0.06 0.05 ● 0.04 visionl8 ● ● ● tech54 ● ntechla8 FRVT rankone8 innovat6 FNMR(T) FMR(T) “False non-match rate” “False match rate” This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 2020/07/24 11:10:31 Impact of wide vs. round shape for medium lightblue masks The lower line is y = x; the upper line is y = 1.7x xforwar0 0.03 ● paravis4 vocord8 ● 0.02 0.01 deepgli2 0.02 0.03 0.05 0.10 0.20 0.30 0.50 False non−match rate with round masks, FNMR(T0) with FMR(T0) = 0.000010 21 Figure 14: At a fixed threshold, a plot of FNMR with round versus wide masks. The displacement of the red line relative to the black “parity” lines shows a modest increase in FNMR with wide masks, the value in the title is the median increase multiplier. 0.60 ● neurote8 ● ● idemia5 0.50 kedacom0 ● 0.40 innovat6 mt0 rankone8 ● FRVT 0.30 ● tech54 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS False non−match rate with high coverage masks, FNMR(T0) ● imperia2 winsens1 deepsea1 ● 0.20 xforwar0 ● ● dahua4 ● ● ● ntechla8 ● bioidte0 0.10 ● visionl8 ● ● 0.08 ● senseti3 cuhkee1 vocord8 0.06 FNMR(T) FMR(T) 0.05 ● ● “False non-match rate” “False match rate” This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 2020/07/24 11:10:31 Impact of high vs. low nose coverage for wide lightblue masks The lower line is y = x; the upper line is y = 5.7x 0.04 paravis4 deepgli2 0.01 0.02 0.03 0.05 0.10 0.20 0.30 False non−match rate with low coverage masks, FNMR(T0) with FMR(T0) = 0.000010 22 Figure 15: At a fixed threshold, a plot of FNMR with round versus wide masks. The displacement of the red line relative to the black “parity” lines shows a considerable increase in FNMR with high vs. low nose coverage masks, the value in the title is the median increase multiplier. COVERAGE COLOR SHAPE = WIDE = WHITE COLOR SHAPE = SHAPE ROUND = LIGHTBLUE = WIDE COLOR SHAPE = ROUND SHAPE = WIDE = BLACK SHAPE = ROUND LO MED HI LO MED HI LO MED HI LO MED HI LO MED HI LO MED HI 3divi-004 acer-000 advance-002 aifirst-001 ailabs-001 0.514 0.048 0.019 0.000 0.071 0.659 0.105 0.046 0.000 0.208 0.627 0.139 0.096 0.000 0.248 0.431 0.071 0.027 0.000 0.116 0.693 0.103 0.040 0.000 0.186 0.762 0.195 0.092 0.000 0.340 0.420 0.035 0.020 0.000 0.061 0.599 0.080 0.045 0.000 0.194 0.603 0.114 0.096 0.000 0.233 0.378 0.052 0.026 0.000 0.102 0.663 0.078 0.037 0.000 0.177 0.769 0.137 0.085 0.000 0.314 0.653 0.107 0.034 0.000 0.116 0.920 0.197 0.104 0.000 0.310 0.939 0.270 0.200 0.000 0.465 0.438 0.089 0.033 0.000 0.129 0.799 0.161 0.061 0.000 0.242 0.931 0.387 0.158 0.000 0.416 6 7 8 9 10 aimall-002 aiunionface-000 alleyes-000 alphaface-002 androvideo-000 0.073 0.000 0.006 0.025 0.000 0.129 0.000 0.023 0.056 0.000 0.225 0.000 0.062 0.099 0.000 0.088 0.000 0.008 0.035 0.000 0.140 0.000 0.014 0.048 0.000 0.215 0.000 0.034 0.079 0.000 0.095 0.000 0.006 0.024 0.000 0.152 0.000 0.020 0.054 0.000 0.260 0.000 0.056 0.095 0.000 0.107 0.000 0.007 0.033 0.000 0.159 0.000 0.012 0.044 0.000 0.236 0.000 0.028 0.072 0.000 0.049 0.000 0.010 0.027 0.000 0.071 0.000 0.043 0.071 0.000 0.154 0.000 0.104 0.132 0.000 0.083 0.000 0.009 0.031 0.000 0.107 0.000 0.018 0.051 0.000 0.144 0.000 0.054 0.111 0.000 11 12 13 14 15 anke-005 antheus-000 asusaics-000 aware-005 awiros-001 0.009 0.000 0.000 0.053 0.195 0.028 0.000 0.000 0.151 0.370 0.066 0.000 0.000 0.218 0.450 0.013 0.000 0.000 0.042 0.180 0.020 0.000 0.000 0.093 0.309 0.048 0.000 0.000 0.250 0.460 0.011 0.000 0.000 0.039 0.162 0.030 0.000 0.000 0.129 0.298 0.069 0.000 0.000 0.211 0.379 0.012 0.000 0.000 0.046 0.161 0.018 0.000 0.000 0.089 0.258 0.041 0.000 0.000 0.244 0.355 0.009 0.000 0.000 0.091 0.198 0.056 0.000 0.000 0.236 0.415 0.091 0.000 0.000 0.449 0.642 0.015 0.000 0.000 0.058 0.216 0.032 0.000 0.000 0.133 0.350 0.086 0.000 0.000 0.371 0.584 16 17 18 19 20 bioidtechswiss-000 camvi-004 chosun-000 chtface-002 cib-000 0.005 0.000 0.000 0.033 0.000 0.022 0.000 0.000 0.100 0.000 0.061 0.000 0.000 0.154 0.000 0.008 0.000 0.000 0.036 0.000 0.018 0.000 0.000 0.071 0.000 0.039 0.000 0.000 0.159 0.000 0.006 0.000 0.000 0.026 0.000 0.028 0.000 0.000 0.081 0.000 0.070 0.000 0.000 0.126 0.000 0.010 0.000 0.000 0.031 0.000 0.021 0.000 0.000 0.056 0.000 0.046 0.000 0.000 0.107 0.000 0.006 0.000 0.000 0.042 0.000 0.021 0.000 0.000 0.144 0.000 0.058 0.000 0.000 0.270 0.000 0.011 0.000 0.000 0.058 0.000 0.019 0.000 0.000 0.104 0.000 0.043 0.000 0.000 0.254 0.000 21 22 23 24 25 ctbcbank-000 cuhkee-001 cyberlink-004 dahua-004 deepglint-002 0.179 0.009 0.014 0.033 0.002 0.794 0.029 0.042 0.150 0.009 0.803 0.069 0.096 0.087 0.028 0.212 0.017 0.020 0.055 0.003 0.667 0.026 0.030 0.135 0.005 0.924 0.059 0.071 0.196 0.014 0.171 0.009 0.013 0.027 0.002 0.786 0.031 0.039 0.126 0.012 0.865 0.074 0.091 0.094 0.031 0.205 0.014 0.018 0.047 0.004 0.620 0.025 0.029 0.121 0.006 0.915 0.057 0.063 0.190 0.017 0.189 0.013 0.018 0.011 0.003 0.806 0.048 0.064 0.057 0.010 0.895 0.140 0.136 0.183 0.024 0.180 0.015 0.022 0.019 0.003 0.477 0.031 0.039 0.048 0.006 0.925 0.093 0.097 0.213 0.018 26 27 28 29 30 deepsea-001 didiglobalface-001 dsk-000 expasoft-000 f8-001 0.000 0.025 0.000 0.000 1.000 0.000 0.056 0.000 0.000 1.000 0.000 0.099 0.000 0.000 1.000 0.000 0.035 0.000 0.000 1.000 0.000 0.048 0.000 0.000 1.000 0.000 0.079 0.000 0.000 1.000 0.000 0.024 0.000 0.000 1.000 0.000 0.054 0.000 0.000 1.000 0.000 0.095 0.000 0.000 1.000 0.000 0.033 0.000 0.000 1.000 0.000 0.044 0.000 0.000 1.000 0.000 0.072 0.000 0.000 1.000 0.000 0.027 0.000 0.000 1.000 0.000 0.071 0.000 0.000 1.000 0.000 0.132 0.000 0.000 1.000 0.000 0.031 0.000 0.000 1.000 0.000 0.051 0.000 0.000 1.000 0.000 0.111 0.000 0.000 1.000 31 32 33 34 35 facesoft-000 fujitsulab-000 glory-002 gorilla-005 hr-002 0.000 0.006 0.059 0.006 1.000 0.000 0.013 0.106 0.018 1.000 0.000 0.018 0.128 0.040 1.000 0.000 0.008 0.055 0.009 1.000 0.000 0.011 0.080 0.012 1.000 0.000 0.019 0.139 0.027 1.000 0.000 0.006 0.056 0.007 1.000 0.000 0.013 0.101 0.018 1.000 0.000 0.019 0.124 0.038 1.000 0.000 0.008 0.053 0.009 1.000 0.000 0.011 0.074 0.012 1.000 0.000 0.018 0.126 0.024 1.000 0.000 0.014 0.054 0.012 1.000 0.000 0.033 0.154 0.037 1.000 0.000 0.045 0.279 0.071 1.000 0.000 0.012 0.072 0.012 1.000 0.000 0.021 0.106 0.021 1.000 0.000 0.046 0.240 0.049 1.000 36 37 38 39 40 idemia-005 iit-002 imagus-001 imperial-002 incode-006 0.002 0.012 0.016 0.000 0.002 0.008 0.036 0.040 0.000 0.008 0.028 0.074 0.074 0.000 0.020 0.003 0.014 0.026 0.000 0.002 0.006 0.024 0.033 0.000 0.003 0.021 0.059 0.064 0.000 0.008 0.002 0.013 0.014 0.000 0.002 0.007 0.043 0.037 0.000 0.008 0.023 0.091 0.066 0.000 0.018 0.002 0.015 0.023 0.000 0.002 0.004 0.027 0.029 0.000 0.003 0.015 0.072 0.056 0.000 0.007 0.002 0.015 0.021 0.000 0.002 0.010 0.087 0.085 0.000 0.012 0.029 0.185 0.149 0.000 0.031 0.003 0.027 0.038 0.000 0.002 0.007 0.057 0.065 0.000 0.004 0.029 0.187 0.167 0.000 0.012 41 42 43 44 45 innovativetechnologyltd-002 innovatrics-006 intellicloudai-001 intellifusion-002 intellivision-002 0.082 0.002 0.000 0.000 0.073 0.176 0.017 0.000 0.001 0.213 0.232 0.051 0.000 0.004 0.267 0.098 0.006 0.000 0.000 0.173 0.142 0.012 0.000 0.001 0.239 0.285 0.035 0.000 0.010 0.380 0.074 0.003 0.000 0.000 0.068 0.172 0.018 0.000 0.000 0.210 0.233 0.054 0.000 0.001 0.261 0.091 0.005 0.000 0.000 0.143 0.131 0.012 0.000 0.000 0.204 0.265 0.035 0.000 0.002 0.340 0.149 0.005 0.000 0.000 0.137 0.362 0.037 0.000 0.001 0.396 0.516 0.087 0.000 0.004 0.469 0.129 0.010 0.000 0.001 0.179 0.208 0.022 0.000 0.002 0.339 0.535 0.076 0.000 0.013 0.703 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS 1 2 3 4 5 FRVT FNMR(T) FMR(T) “False non-match rate” “False match rate” This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 2020/07/24 11:10:31 Algorithm Name Table 5: This table summarizes Failure to Enroll (FTE) rates surveyed over 10 000 images of each mask variant. FTE is the proportion of failed template generation attempts. Failures can occur because the software throws an exception, or because the software electively refuses to process the input image as would occur if the algorithms does not detect a face or determines that the face has insufficient information. FTE is measured as the number of function calls that give EITHER a non-zero error code OR that give a “small” template containing fewer than 60 bytes. This second rule is needed because some algorithms incorrectly fail to return a non-zero error code when template generation fails but do produce a skeletal template. The effects of FTE are included in the accuracy results of this report by regarding any template comparison involving a failed template to produce a low similarity score. Thus higher FTE results in higher FNMR and lower FMR. 23 COVERAGE COLOR SHAPE = WIDE = WHITE SHAPE COLOR = ROUND SHAPE = LIGHTBLUE = WIDE COLOR SHAPE = ROUND SHAPE = WIDE = BLACK SHAPE = ROUND LO MED HI LO MED HI LO MED HI LO MED HI LO MED HI LO MED HI intelresearch-001 intsysmsu-002 iqface-000 isap-001 itmo-007 0.088 0.008 0.000 1.000 0.008 0.212 0.055 0.000 1.000 0.034 0.242 0.117 0.000 1.000 0.086 0.138 0.021 0.000 1.000 0.013 0.197 0.041 0.000 1.000 0.027 0.328 0.120 0.000 1.000 0.059 0.086 0.007 0.000 1.000 0.009 0.213 0.047 0.000 1.000 0.046 0.257 0.110 0.000 1.000 0.106 0.132 0.015 0.000 1.000 0.017 0.191 0.033 0.000 1.000 0.034 0.316 0.100 0.000 1.000 0.071 0.068 0.036 0.000 1.000 0.011 0.230 0.105 0.000 1.000 0.034 0.358 0.231 0.000 1.000 0.082 0.114 0.040 0.000 1.000 0.015 0.185 0.075 0.000 1.000 0.030 0.406 0.218 0.000 1.000 0.064 51 52 53 54 55 kakao-003 kedacom-000 kneron-005 lookman-004 luxand-000 0.000 0.000 0.063 0.000 0.000 0.000 0.000 0.184 0.000 0.000 0.000 0.000 0.206 0.000 0.000 0.000 0.000 0.106 0.000 0.000 0.000 0.000 0.163 0.000 0.000 0.000 0.000 0.307 0.000 0.000 0.000 0.000 0.058 0.000 0.000 0.000 0.000 0.166 0.000 0.000 0.000 0.000 0.212 0.000 0.000 0.000 0.000 0.094 0.000 0.000 0.000 0.000 0.146 0.000 0.000 0.000 0.000 0.276 0.000 0.000 0.000 0.000 0.101 0.000 0.000 0.000 0.000 0.440 0.000 0.000 0.000 0.000 0.505 0.000 0.000 0.000 0.000 0.154 0.000 0.000 0.000 0.000 0.325 0.000 0.000 0.000 0.000 0.574 0.000 0.000 56 57 58 59 60 mt-000 mvision-001 netbridgetech-001 neurotechnology-008 nodeflux-002 0.005 0.000 0.000 0.008 0.402 0.021 0.000 0.000 0.029 0.598 0.061 0.000 0.000 0.035 0.538 0.011 0.000 0.000 0.009 0.449 0.022 0.000 0.000 0.013 0.635 0.047 0.000 0.000 0.021 0.835 0.006 0.000 0.000 0.007 0.440 0.024 0.000 0.000 0.025 0.671 0.063 0.000 0.000 0.032 0.628 0.011 0.000 0.000 0.007 0.482 0.021 0.000 0.000 0.010 0.681 0.045 0.000 0.000 0.020 0.877 0.007 0.000 0.000 0.019 0.602 0.023 0.000 0.000 0.107 0.835 0.059 0.000 0.000 0.082 0.915 0.011 0.000 0.000 0.009 0.418 0.021 0.000 0.000 0.018 0.604 0.046 0.000 0.000 0.040 0.927 61 62 63 64 65 notiontag-000 ntechlab-008 paravision-004 pixelall-003 psl-004 0.000 0.064 0.002 0.000 0.004 0.000 0.126 0.011 0.000 0.017 0.000 0.196 0.027 0.000 0.042 0.000 0.079 0.004 0.000 0.009 0.000 0.108 0.004 0.000 0.018 0.000 0.020 0.011 0.000 0.038 0.000 0.053 0.002 0.000 0.004 0.000 0.011 0.010 0.000 0.015 0.000 0.183 0.024 0.000 0.037 0.000 0.003 0.003 0.000 0.007 0.000 0.095 0.004 0.000 0.014 0.000 0.018 0.009 0.000 0.029 0.000 0.003 0.003 0.000 0.011 0.000 0.016 0.016 0.000 0.028 0.000 0.042 0.043 0.000 0.058 0.000 0.004 0.004 0.000 0.018 0.000 0.009 0.006 0.000 0.034 0.000 0.026 0.019 0.000 0.078 66 67 68 69 70 rankone-008 remarkai-002 rokid-000 s1-001 scanovate-001 0.136 0.000 0.194 0.647 0.544 0.414 0.000 0.372 0.943 0.601 0.293 0.000 0.370 0.911 0.596 0.180 0.000 0.239 0.632 0.547 0.276 0.000 0.401 0.932 0.629 0.459 0.000 0.683 0.959 0.733 0.117 0.000 0.220 0.617 0.515 0.358 0.000 0.444 0.930 0.553 0.292 0.000 0.450 0.915 0.579 0.154 0.000 0.265 0.616 0.513 0.229 0.000 0.457 0.919 0.565 0.386 0.000 0.749 0.954 0.664 0.153 0.000 0.367 0.646 0.554 0.470 0.000 0.677 0.962 0.676 0.770 0.000 0.806 0.962 0.806 0.109 0.000 0.230 0.435 0.516 0.230 0.000 0.405 0.881 0.682 0.770 0.000 0.808 0.964 0.903 71 72 73 74 75 sensetime-003 sertis-000 shu-002 sjtu-002 starhybrid-001 0.009 0.002 0.011 0.011 0.192 0.029 0.012 0.031 0.031 0.468 0.069 0.034 0.080 0.080 0.461 0.017 0.003 0.028 0.028 0.161 0.026 0.006 0.045 0.045 0.371 0.059 0.016 0.115 0.115 0.527 0.009 0.002 0.009 0.009 0.149 0.031 0.012 0.026 0.026 0.406 0.074 0.032 0.083 0.083 0.483 0.014 0.003 0.023 0.023 0.137 0.025 0.005 0.037 0.037 0.321 0.057 0.013 0.103 0.103 0.487 0.013 0.005 0.016 0.016 0.133 0.048 0.020 0.056 0.056 0.372 0.140 0.052 0.167 0.167 0.565 0.015 0.005 0.022 0.022 0.149 0.031 0.010 0.040 0.040 0.303 0.093 0.026 0.139 0.139 0.644 76 77 78 79 80 synesis-006 synology-000 tech5-004 tevian-005 trueface-000 0.001 0.000 0.005 0.125 0.000 0.003 0.000 0.022 0.463 0.000 0.007 0.000 0.061 0.370 0.000 0.001 0.000 0.008 0.181 0.000 0.001 0.000 0.018 0.271 0.000 0.003 0.000 0.039 0.581 0.000 0.001 0.000 0.006 0.148 0.000 0.003 0.000 0.028 0.650 0.000 0.007 0.000 0.070 0.557 0.000 0.001 0.000 0.010 0.208 0.000 0.001 0.000 0.021 0.359 0.000 0.003 0.000 0.046 0.705 0.000 0.001 0.000 0.006 0.131 0.000 0.004 0.000 0.021 0.786 0.000 0.008 0.000 0.058 0.787 0.000 0.001 0.000 0.011 0.122 0.000 0.002 0.000 0.019 0.272 0.000 0.003 0.000 0.043 0.758 0.000 81 82 83 84 85 tuputech-000 uluface-002 upc-001 veridas-003 via-001 0.517 0.000 0.002 1.000 0.000 0.679 0.000 0.005 1.000 0.000 0.684 0.000 0.012 1.000 0.000 0.456 0.000 0.001 1.000 0.000 0.592 0.000 0.002 1.000 0.000 0.679 0.000 0.004 1.000 0.000 0.626 0.000 0.002 1.000 0.000 0.758 0.000 0.005 1.000 0.000 0.765 0.000 0.012 1.000 0.000 0.502 0.000 0.002 1.000 0.000 0.619 0.000 0.002 1.000 0.000 0.714 0.000 0.005 1.000 0.000 0.661 0.000 0.003 1.000 0.000 0.904 0.000 0.007 1.000 0.000 0.933 0.000 0.018 1.000 0.000 0.595 0.000 0.002 1.000 0.000 0.830 0.000 0.004 1.000 0.000 0.964 0.000 0.011 1.000 0.000 86 87 88 89 90 videmo-000 videonetics-002 vigilantsolutions-007 visionlabs-008 visteam-000 0.019 0.338 0.062 0.013 0.058 0.067 0.581 0.168 0.035 0.150 0.125 0.557 0.220 0.083 0.210 0.029 0.390 0.077 0.023 0.059 0.057 0.593 0.153 0.045 0.114 0.142 0.849 0.275 0.124 0.233 0.018 0.330 0.052 0.012 0.048 0.051 0.569 0.137 0.031 0.118 0.106 0.542 0.193 0.072 0.176 0.023 0.378 0.069 0.019 0.052 0.040 0.559 0.126 0.038 0.092 0.089 0.785 0.206 0.097 0.156 0.027 0.396 0.072 0.024 0.074 0.100 0.702 0.273 0.061 0.202 0.296 0.848 0.493 0.124 0.369 0.036 0.302 0.088 0.025 0.088 0.062 0.508 0.180 0.056 0.159 0.192 0.947 0.449 0.165 0.374 - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS 46 47 48 49 50 FRVT FNMR(T) FMR(T) “False non-match rate” “False match rate” This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 2020/07/24 11:10:31 Algorithm Name Table 6: This table summarizes Failure to Enroll (FTE) rates surveyed over 10 000 images of each mask variant. FTE is the proportion of failed template generation attempts. Failures can occur because the software throws an exception, or because the software electively refuses to process the input image as would occur if the algorithms does not detect a face or determines that the face has insufficient information. FTE is measured as the number of function calls that give EITHER a non-zero error code OR that give a “small” template containing fewer than 60 bytes. This second rule is needed because some algorithms incorrectly fail to return a non-zero error code when template generation fails but do produce a skeletal template. The effects of FTE are included in the accuracy results of this report by regarding any template comparison involving a failed template to produce a low similarity score. Thus higher FTE results in higher FNMR and lower FMR. 24 COVERAGE 91 92 93 vocord-008 winsense-001 xforwardai-000 SHAPE = COLOR WIDE = WHITE SHAPE = ROUND SHAPE = COLOR WIDE = LIGHTBLUE SHAPE = ROUND SHAPE = COLOR WIDE = BLACK SHAPE = ROUND LO MED HI LO MED HI LO MED HI LO MED HI LO MED HI LO MED HI 0.013 0.000 0.000 0.046 0.000 0.000 0.087 0.000 0.000 0.025 0.000 0.000 0.047 0.000 0.000 0.096 0.000 0.000 0.011 0.000 0.000 0.052 0.000 0.000 0.089 0.000 0.000 0.031 0.000 0.000 0.059 0.000 0.000 0.111 0.000 0.000 0.009 0.000 0.000 0.050 0.000 0.000 0.093 0.000 0.000 0.018 0.000 0.000 0.037 0.000 0.000 0.095 0.000 0.000 Table 7: This table summarizes Failure to Enroll (FTE) rates surveyed over 10 000 images of each mask variant. FTE is the proportion of failed template generation attempts. Failures can occur because the software throws an exception, or because the software electively refuses to process the input image as would occur if the algorithms does not detect a face or determines that the face has insufficient information. FTE is measured as the number of function calls that give EITHER a non-zero error code OR that give a “small” template containing fewer than 60 bytes. This second rule is needed because some algorithms incorrectly fail to return a non-zero error code when template generation fails but do produce a skeletal template. The effects of FTE are included in the accuracy results of this report by regarding any template comparison involving a failed template to produce a low similarity score. Thus higher FTE results in higher FNMR and lower FMR. FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS FNMR(T) FMR(T) “False non-match rate” “False match rate” This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 2020/07/24 11:10:31 Algorithm Name 25 FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS 26 Failure−to−template contribution toward total false rejection for medium wide lightblue masks This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 Algorithm Kind chosun_000 alphaface_002 shu_002 visteam_000 acer_000 antheus_000 videonetics_002 notiontag_000 luxand_000 dsk_000 netbridgetech_001 tuputech_000 s1_001 intellivision_002 aware_005 awiros_001 ctbcbank_000 nodeflux_002 vigilantsolutions_007 expasoft_000 ailabs_001 3divi_004 lookman_004 imagus_001 androvideo_000 idemia_005 innovativetechnologyltd_002 intelresearch_001 kedacom_000 scanovate_001 videmo_000 rankone_008 tevian_005 starhybrid_001 fujitsulab_000 gorilla_005 upc_001 kneron_005 synology_000 intellicloudai_001 rokid_000 chtface_002 trueface_000 kakao_003 mvision_001 aimall_002 incode_006 asusaics_000 neurotechnology_008 via_001 anke_005 intsysmsu_002 deepsea_001 iit_002 aiunionface_000 iqface_000 glory_002 mt_000 pixelall_003 itmo_007 sertis_000 aifirst_001 x−laboratory_001 uluface_002 advance_002 remarkai_002 innovatrics_006 camvi_004 cyberlink_004 sjtu_002 psl_004 dahua_004 intellifusion_002 cib_000 winsense_001 facesoft_000 imperial_002 bioidtechswiss_000 xforwardai_000 alleyes_000 didiglobalface_001 tech5_004 ntechlab_008 visionlabs_008 cuhkee_001 sensetime_003 vocord_008 paravision_004 deepglint_002 ● FNMR ● FTE ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.001 0.003 ● 0.010 0.030 0.100 0.300 Fraction of rejections due to FTE and total max(0.0003,FTE) and FNMR Figure 16: For each algorithm the rightmost dot shows FNMR @ FMR=0.00001 (as reported throughout this report). The left most dot shows the failure-to-template (FTE) rate over the masked verification set of 5.2M images. The gap between the two dots is attributable to low similarity score. Some FTE rates are zero - rates below 0.001 are shown as 0.001. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS FRVT deepglint_002 paravision_004 27 sensetime_003 0.300 ● 0.100 ● ● ● ● 0.010 ● ● ● 0.030 ● ● ● ● ● ● ● ● ● ● 0. 40 0. 35 8 2. 0. 30 6 2. 0 2. 4 8 1. 2. 3 1. 2 2 1. 2. 1 1. 1. 0 0.001 Mask Color black vocord_008 visionlabs_008 cuhkee_001 lightblue not masked False non−match rate (FNMR) 0.300 ● ● ● ● ● 0.100 ● 0.030 ● wide ● ● ● ● ● ● ● 0.010 Mask Shape ● ● ● ● ● ● ● round not masked 0.003 Mask Top Coverage 0.001 low ntechlab_008 tech5_004 0. 6 0. 5 0. 4 0. 3 75 0. 50 0. 25 0. 00 0. 8. 8 99 8. 6 99 8. 4 99 8. 2 99 8. 0 medium 99 high not masked alleyes_000 0.300 ● ● ● ● ● 0.100 ● ● ● ● 0.030 ● ● 0.010 ● ● ● ● ● ● 0.003 7 2. 4 2. 1 2. 5 0 9. 50 5 9. 50 0 8. 50 5 8. 50 7. 50 4 1. 2 1. 0 0.001 1. This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.003 Threshold Figure 17: FNMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS didiglobalface_001 bioidtechswiss_000 28 xforwardai_000 0.300 ● ● 0.100 ● ● ● ● ● ● ● 0.030 ● ● 0.010 ● ● 0 1 2 3 4 1. 1. 1. 1. 1. 0 7. 9 5 6. 0. 4 0. 0 3 0. 8. 2 0. 5 1 0. imperial_002 7. 0 0. 0.001 psl_004 Mask Color black dahua_004 lightblue not masked False non−match rate (FNMR) 0.300 ● ● ● ● Mask Shape 0.100 wide ● 0.030 ● ● ● ● ● 0.010 round not masked 0.003 Mask Top Coverage low 0.001 facesoft_000 00 70 00 60 00 50 00 30 winsense_001 40 00 70 0. 65 0. 60 0. 55 0. 50 0. 4 1. 3 1. 2 1. 1 1. 1. 0 medium high not masked cyberlink_004 ● 0.300 ● 0.100 ● ● 0.030 ● ● 0.010 0.003 1. 5 1. 4 1. 3 1. 2 1. 1 1. 0 0. 9 0. 4 0. 3 0. 2 0. 1 0. 0 1. 4 1. 3 1. 2 1. 1 0.001 1. 0 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.003 Threshold Figure 18: FNMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS sjtu_002 uluface_002 29 intellifusion_002 0.300 0.100 0.030 0.010 0.001 0. 4 0. 3 0. 2 0. 1 0. 0 0. 8 0. 6 0. 4 0. 2 1. 3 1. 2 1. 1 1. 0 0. 9 0. 8 Mask Color black camvi_004 remarkai_002 innovatrics_006 lightblue not masked False non−match rate (FNMR) ● 0.300 ● Mask Shape 0.100 ● ● 0.030 wide ● ● ● 0.010 round not masked 0.003 Mask Top Coverage 0.001 low intsysmsu_002 advance_002 40 30 20 10 0 7 0. 6 0. 5 0. 4 0. 3 0. 2 0. 1 0. 0. 0 medium pixelall_003 high not masked 0.300 0.100 0.030 0.010 0.003 0. 5 0. 4 0. 3 0. 2 0. 1 0. 5 0. 0 0. 4 0. 3 0. 2 0. 1 0. 0 1. −0 5 .1 1. 4 1. 3 1. 2 1. 1 1. 0 0.001 0. 9 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.003 Threshold Figure 19: FNMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS aifirst_001 sertis_000 30 deepsea_001 ● 0.300 ● ● ● ● 0.100 ● 0.030 0.010 6 7 5 6 7 9 0 1 2 3 4 0. 0. 0. 0. 0. 0. 1. 1. 1. 1. 1. 0. 5 0.001 x−laboratory_001 mt_000 Mask Color black iqface_000 lightblue not masked False non−match rate (FNMR) 0.300 ● 0.100 Mask Shape ● wide ● 0.030 ● ● 0.010 round not masked 0.003 Mask Top Coverage low 0.001 glory_002 5 1. 4 1. 3 1. 2 1. 1 1. 0 1. 5 1. 4 3 1. 1. 2 1 aiunionface_000 1. 1. 0 1. 9 0. 4 0. 3 0. 2 0. 1 0. 0 0. .1 medium −0 high not masked kakao_003 0.300 0.100 0.030 0.010 0.003 0. 5 0. 4 0. 3 0. 2 0. 1 0. 0 0 0. 6 5 0. 5 0 0. 5 0. 7 0. 6 0.001 0. 5 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.003 Threshold Figure 20: FNMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS anke_005 itmo_007 31 iit_002 0.300 0.100 0.030 0.010 0. 75 0. 70 0. 65 0. 60 0. 55 0. 50 70 60 50 0. 5 0. 4 0. 3 0. 2 0. 1 0. 0 0.001 Mask Color black via_001 asusaics_000 incode_006 lightblue not masked False non−match rate (FNMR) 0.300 Mask Shape 0.100 wide 0.030 ● round not masked 0.010 0.003 Mask Top Coverage 0.001 low neurotechnology_008 aimall_002 high 1. 4 1. 2 1. 0 1. 5 1. 4 1. 3 1. 2 1. 1 1. 0 3. 0 0. 9 2. 7 2. 4 2. 1 1. 8 medium not masked rokid_000 ● ● 0.300 ● 0.100 0.030 0.010 0.003 0. 7 0. 6 0. 5 0. 4 0. 7 0. 6 0. 5 0. 4 50 0. 3 12 00 10 75 0 50 0 25 0 0.001 0 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.003 Threshold Figure 21: FNMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS chtface_002 mvision_001 32 cib_000 0.300 0.100 0.030 0.010 synology_000 tevian_005 1. 5 1. 4 1. 3 1. 2 1. 1 1. 0 0. 8 0. 7 0. 6 0. 5 0. 4 0. 3 1. 5 1. 4 1. 3 1. 2 1. 1 1. 0 0. 9 0.001 Mask Color black kneron_005 lightblue not masked False non−match rate (FNMR) 0.300 Mask Shape 0.100 wide 0.030 ● round not masked 0.010 0.003 Mask Top Coverage low 0.001 upc_001 fujitsulab_000 0. 5 0. 4 0. 3 0. 2 0. 1 0. 0 5 0. 7 0 0. 5 5 0. 2 0 0. 0 0. 4 0. 2 0. 0 medium high not masked starhybrid_001 0.300 0.100 0.030 0.010 0.003 0. 4 0. 3 0. 2 0. 1 0. 0 0. 5 0. 4 0. 3 0. 2 0. 1 0. 0 0. 4 0. 2 0.001 0. 0 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.003 Threshold Figure 22: FNMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS intellicloudai_001 gorilla_005 33 lookman_004 0.300 0.100 0.030 0.010 idemia_005 videmo_000 Mask Color 0. 7 0. 6 0. 5 0. 4 0. 4 0. 2 0. 0 0. 7 0. 6 0. 5 0.001 black kedacom_000 lightblue ● 0.300 ● ● ● 0.100 not masked ● ● False non−match rate (FNMR) Mask Shape ● wide ● 0.030 ● 0.010 round not masked 0.003 Mask Top Coverage 0.001 low scanovate_001 ● 0.300 ● 7 0. 6 0. 5 0. 4 0. 3 0. 7 0. 6 5 rankone_008 0. ● ● ● 0. 00 45 00 40 00 35 00 30 00 25 20 00 medium high not masked intelresearch_001 ● ● ● ● 0.100 0.030 0.010 0.003 60 0 40 0 20 0 0 0. 7 0. 6 0. 5 0. 4 0. 3 0. 5 0. 0 −0 .5 0.001 −1 .0 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.003 Threshold Figure 23: FNMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS 3divi_004 ailabs_001 34 imagus_001 0.300 0.100 0.030 0.010 0.001 0. 4 0. 2 0. 0 2. 4 2. 2 2. 0 3. 0 1. 8 2. 5 2. 0 1. 5 Mask Color black vigilantsolutions_007 androvideo_000 expasoft_000 lightblue not masked False non−match rate (FNMR) 0.300 Mask Shape 0.100 wide 0.030 ● round not masked 0.010 0.003 Mask Top Coverage 0.001 low innovativetechnologyltd_002 nodeflux_002 0. 5 0. 4 0. 3 0. 2 0. 1 0. 0 0. 4 0. 2 0. 0 2. 8 2. 4 2. 0 medium ctbcbank_000 high not masked 0.300 0.100 0.030 0.010 0.003 5 3. 7 0 3. 5 5 3. 2 0 3. 0 0 0. 5 5 0. 4 0 0. 4 5 0. 3 0 0. 3 0. 4 0. 2 0.001 0. 0 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.003 Threshold Figure 24: FNMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS aware_005 intellivision_002 35 awiros_001 0.300 0.100 0.030 0.010 0. 80 0. 78 0. 76 0. 74 40 30 20 10 0 3 2 1 0 0.001 Mask Color black s1_001 trueface_000 tuputech_000 lightblue not masked False non−match rate (FNMR) 0.300 Mask Shape 0.100 wide 0.030 ● round not masked 0.010 0.003 Mask Top Coverage 0.001 low dsk_000 netbridgetech_001 high 0. 4 0. 3 0. 2 0. 1 0. 0 0. 6 0. 4 0. 2 0. 0 0. 6 0. 5 0. 4 medium not masked luxand_000 0.300 0.100 0.030 0.010 0.003 0 1. 0 5 0. 7 0 0. 5 5 0. 2 1. 0 0. 00 0. 8 0. 6 0. 4 1. 2 0. 2 1. 1 1. 0 0. 9 0. 8 0.001 0. 7 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.003 Threshold Figure 25: FNMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS notiontag_000 videonetics_002 36 visteam_000 0.300 0.100 0.030 0.010 00 Mask Color 1. 75 0. 50 0. 25 0. 0. 00 0. 9 0. 8 0. 7 12 8 4 0.001 black antheus_000 acer_000 alphaface_002 lightblue not masked False non−match rate (FNMR) 0.300 Mask Shape 0.100 wide 0.030 ● round not masked 0.010 0.003 Mask Top Coverage 0.001 low shu_002 2 0. 1 0. 0. 0 75 0. 50 0. 25 0. 00 0. 0 2. 8 1. 6 1. 1. 4 medium high not masked chosun_000 0.300 0.100 0.030 0.010 0.003 0 1. 0 5 0. 7 0 0. 5 5 0. 2 0 0. 0 0 1. 0 5 0. 9 0 0. 9 5 0.001 0. 8 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 0.003 Threshold Figure 26: FNMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS deepglint_002 paravision_004 37 sensetime_003 1e−01 3e−02 1e−02 3e−03 1e−03 ● ● ● ● ● ● 3e−04 1e−04 1e−05 3e−06 1e−06 ● ●● 0. 40 0. 35 8 2. 0. 30 6 2. 0 2. 4 8 1. 2. 3 1. 2 2 1. 2. 1 ● ● ● 1. 1. 0 ● ● ● Mask Color black vocord_008 visionlabs_008 cuhkee_001 lightblue 1e−01 not masked 3e−02 False match rate (FMR) 1e−02 3e−03 Mask Shape ● ● wide 1e−03 ● ● ● ● 3e−04 1e−04 round not masked 3e−05 1e−05 Mask Top Coverage 3e−06 low 1e−06 ● ● ● ntechlab_008 tech5_004 0. 6 ● ● 0. 5 0. 4 ● 0. 3 75 0. 50 0. 25 0. 00 0. 8. 8 99 8. 6 99 8. 4 99 8. 2 99 8. 0 ● ● 99 medium high not masked alleyes_000 1e−01 3e−02 1e−02 3e−03 1e−03 ● ● ● 3e−04 1e−04 3e−05 1e−05 3e−06 1e−06 7 2. 4 2. 1 ●● ● 2. 5 0 9. 50 5 9. 50 0 8. 50 8. 50 7. 5 ● 50 4 1. 2 1. 0 ●● 1. This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 3e−05 Threshold Figure 27: FMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS FRVT didiglobalface_001 bioidtechswiss_000 38 xforwardai_000 1e−01 3e−02 1e−02 3e−03 1e−03 ● ● ● ● 3e−04 1e−04 1e−05 3e−06 1e−06 ● ● ● 0 1 2 3 4 1. 1. 1. 1. 1. 0 7. 9 5 6. 0. 4 0. 0 3 0. 8. 2 0. 5 1 0. ●● ● 7. 0 0. ● ●● imperial_002 False match rate (FMR) psl_004 Mask Color black dahua_004 1e−01 lightblue 3e−02 not masked 1e−02 Mask Shape 3e−03 1e−03 ● ● wide ● ● ● 3e−04 1e−04 round not masked 3e−05 1e−05 Mask Top Coverage 3e−06 low 1e−06 medium facesoft_000 00 70 00 60 00 50 00 30 winsense_001 40 00 70 0. 65 0. 60 0. 55 0. 50 ● ● 0. 4 1. 3 1. 2 1. 1 1. 1. 0 ● ●● high not masked cyberlink_004 1e−01 3e−02 1e−02 3e−03 1e−03 ● 3e−04 1e−04 3e−05 1e−05 3e−06 1e−06 1. 5 1. 4 1. 3 1. 2 1. 1 1. 0 0. 9 0. 4 0. 3 0. 2 0. 1 0. 0 1. 4 1. 3 1. 2 1. 1 ●●● 1. 0 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 3e−05 Threshold Figure 28: FMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS sjtu_002 uluface_002 39 intellifusion_002 1e−01 3e−02 1e−02 3e−03 1e−03 3e−04 1e−04 1e−05 3e−06 1e−06 0. 4 0. 3 0. 2 0. 1 0. 0 0. 8 0. 6 0. 4 0. 2 1. 3 1. 2 1. 1 1. 0 0. 9 0. 8 Mask Color black camvi_004 remarkai_002 innovatrics_006 lightblue 1e−01 not masked 3e−02 False match rate (FMR) 1e−02 Mask Shape 3e−03 wide 1e−03 ● ● ● 3e−04 round not masked 1e−04 3e−05 1e−05 Mask Top Coverage 3e−06 low 1e−06 medium intsysmsu_002 advance_002 40 30 20 10 0 7 0. 6 0. 5 0. 4 0. 3 0. 2 0. 1 0. 0. 0 ● ● pixelall_003 high not masked 1e−01 3e−02 1e−02 3e−03 1e−03 3e−04 1e−04 3e−05 1e−05 3e−06 0. 5 0. 4 0. 3 0. 2 0. 1 0. 5 0. 0 0. 4 0. 3 0. 2 0. 1 0. 0 1. −0 5 .1 1. 4 1. 3 1. 2 1. 1 1. 0 1e−06 0. 9 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 3e−05 Threshold Figure 29: FMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS aifirst_001 sertis_000 40 deepsea_001 1e−01 3e−02 1e−02 3e−03 1e−03 ● ● ● 3e−04 1e−04 1e−05 3e−06 1e−06 6 7 5 6 7 9 0 1 2 3 4 0. 0. 0. 0. 0. 0. 1. 1. 1. 1. 1. 0. 5 ● ●● x−laboratory_001 mt_000 Mask Color black iqface_000 lightblue 1e−01 not masked 3e−02 False match rate (FMR) 1e−02 Mask Shape 3e−03 wide 1e−03 ● ● ● 3e−04 1e−04 round not masked 3e−05 1e−05 Mask Top Coverage 3e−06 low 1e−06 medium glory_002 5 1. 4 1. 3 1. 2 1. 1 1. 0 1. 5 1. 4 1. 3 1. 2 1 aiunionface_000 1. 1. 0 9 1. 0. 0. 4 0. 3 0. 2 0. 1 0. 0 .1 ●● −0 high not masked kakao_003 1e−01 3e−02 1e−02 3e−03 1e−03 3e−04 1e−04 3e−05 1e−05 3e−06 0. 5 0. 4 0. 3 0. 2 0. 1 0. 0 0 0. 6 5 0. 5 0 0. 5 0. 7 0. 6 1e−06 0. 5 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 3e−05 Threshold Figure 30: FMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS anke_005 itmo_007 41 iit_002 1e−01 3e−02 1e−02 3e−03 1e−03 3e−04 1e−04 1e−05 3e−06 0. 75 0. 70 0. 65 0. 60 0. 55 0. 50 70 60 50 0. 5 0. 4 0. 3 0. 2 0. 1 0. 0 1e−06 Mask Color black via_001 asusaics_000 incode_006 lightblue 1e−01 not masked 3e−02 1e−02 Mask Shape 3e−03 wide 1e−03 ● 3e−04 round not masked 1e−04 3e−05 neurotechnology_008 aimall_002 high 1. 4 1. 2 1. 0 1. 5 1. 4 1. 3 1. 2 1. 1 medium 1. 0 1e−06 3. 0 0. 9 low 2. 7 3e−06 2. 4 Mask Top Coverage 2. 1 1e−05 1. 8 False match rate (FMR) not masked rokid_000 1e−01 3e−02 1e−02 3e−03 1e−03 3e−04 1e−04 3e−05 1e−05 3e−06 1e−06 0. 7 0. 6 0. 5 0. 4 0. 7 0. 6 0. 5 0. 4 50 0. 3 12 00 10 75 0 50 0 25 0 ●● ●● 0 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 3e−05 Threshold Figure 31: FMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS chtface_002 mvision_001 42 cib_000 1e−01 3e−02 1e−02 3e−03 1e−03 3e−04 1e−04 1e−05 3e−06 synology_000 tevian_005 1. 5 1. 4 1. 3 1. 2 1. 1 1. 0 0. 8 0. 7 0. 6 0. 5 0. 4 0. 3 1. 5 1. 4 1. 3 1. 2 1. 1 1. 0 0. 9 1e−06 Mask Color black kneron_005 1e−01 lightblue 3e−02 not masked 1e−02 Mask Shape 3e−03 wide 1e−03 ● 3e−04 1e−04 round not masked 3e−05 fujitsulab_000 0. 5 0. 4 0. 3 0. 2 0. 1 0. 0 0. 7 0. 5 0. 2 0. 0 upc_001 5 medium 0 1e−06 5 low 0 3e−06 0. 4 Mask Top Coverage 0. 2 1e−05 0. 0 False match rate (FMR) high not masked starhybrid_001 1e−01 3e−02 1e−02 3e−03 1e−03 3e−04 1e−04 3e−05 1e−05 3e−06 0. 4 0. 3 0. 2 0. 1 0. 0 0. 5 0. 4 0. 3 0. 2 0. 1 0. 0 0. 4 0. 2 1e−06 0. 0 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 3e−05 Threshold Figure 32: FMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS FRVT intellicloudai_001 gorilla_005 43 lookman_004 1e−01 3e−02 1e−02 3e−03 1e−03 3e−04 1e−04 1e−05 3e−06 idemia_005 videmo_000 Mask Color 0. 7 0. 6 0. 5 0. 4 0. 4 0. 2 0. 0 0. 7 0. 6 0. 5 1e−06 False match rate (FMR) black kedacom_000 1e−01 lightblue 3e−02 not masked 1e−02 Mask Shape 3e−03 1e−03 wide ●● ● ● 3e−04 ● 1e−04 round not masked 3e−05 1e−05 Mask Top Coverage 3e−06 low 1e−06 medium rankone_008 scanovate_001 ● ● 7 0. 6 0. 5 0. 4 0. 3 0. 7 0. 6 0. 5 0. 00 45 00 00 40 35 00 00 30 25 20 00 ● ● high not masked intelresearch_001 1e−01 3e−02 1e−02 3e−03 1e−03 ● ● ● 3e−04 1e−04 3e−05 1e−05 3e−06 1e−06 60 0 40 0 20 0 0 0. 7 0. 6 0. 5 0. 4 0. 3 0. 5 0. 0 −0 .5 ● ● ● ● −1 .0 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 3e−05 Threshold Figure 33: FMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS 3divi_004 ailabs_001 44 imagus_001 1e−01 3e−02 1e−02 3e−03 1e−03 3e−04 1e−04 1e−05 3e−06 1e−06 0. 4 0. 2 0. 0 2. 4 2. 2 2. 0 3. 0 1. 8 2. 5 2. 0 1. 5 Mask Color black vigilantsolutions_007 androvideo_000 expasoft_000 lightblue 1e−01 not masked 3e−02 1e−02 Mask Shape 3e−03 wide 1e−03 ● 3e−04 round not masked 1e−04 3e−05 innovativetechnologyltd_002 nodeflux_002 0. 5 0. 4 0. 3 0. 2 0. 1 0. 0 medium 0. 4 1e−06 0. 2 low 0. 0 3e−06 2. 8 Mask Top Coverage 2. 4 1e−05 2. 0 False match rate (FMR) ctbcbank_000 high not masked 1e−01 3e−02 1e−02 3e−03 1e−03 3e−04 1e−04 3e−05 1e−05 3e−06 5 3. 7 0 3. 5 5 3. 2 0 3. 0 0 0. 5 5 0. 4 0 0. 4 5 0. 3 0 0. 3 0. 4 0. 2 1e−06 0. 0 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 3e−05 Threshold Figure 34: FMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS aware_005 intellivision_002 45 awiros_001 1e−01 3e−02 1e−02 3e−03 1e−03 3e−04 1e−04 1e−05 3e−06 0. 80 0. 78 0. 76 0. 74 40 30 20 10 0 3 2 1 0 1e−06 Mask Color black s1_001 trueface_000 tuputech_000 lightblue 1e−01 not masked 3e−02 1e−02 Mask Shape 3e−03 wide 1e−03 ● 3e−04 round not masked 1e−04 3e−05 dsk_000 netbridgetech_001 high 0. 4 0. 3 0. 2 0. 1 0. 0 0. 6 medium 0. 4 1e−06 0. 2 low 0. 0 3e−06 0. 6 Mask Top Coverage 0. 5 1e−05 0. 4 False match rate (FMR) not masked luxand_000 1e−01 3e−02 1e−02 3e−03 1e−03 3e−04 1e−04 3e−05 1e−05 3e−06 0 1. 0 5 0. 7 0 0. 5 5 0. 2 1. 0 0. 00 0. 8 0. 6 0. 4 1. 2 0. 2 1. 1 1. 0 0. 9 0. 8 1e−06 0. 7 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 3e−05 Threshold Figure 35: FMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS notiontag_000 videonetics_002 46 visteam_000 1e−01 3e−02 1e−02 3e−03 1e−03 3e−04 1e−04 1e−05 3e−06 00 Mask Color 1. 75 0. 50 0. 25 0. 0. 00 0. 9 0. 8 0. 7 12 8 4 1e−06 black antheus_000 acer_000 alphaface_002 lightblue 1e−01 not masked 3e−02 1e−02 Mask Shape 3e−03 wide 1e−03 ● 3e−04 round not masked 1e−04 3e−05 2 0. 1 0. 0. 0 75 0. 0. 0. 0. 2. 1. 1. 1. shu_002 50 medium 25 1e−06 00 low 0 3e−06 8 Mask Top Coverage 6 1e−05 4 False match rate (FMR) high not masked chosun_000 1e−01 3e−02 1e−02 3e−03 1e−03 3e−04 1e−04 3e−05 1e−05 3e−06 0 1. 0 5 0. 7 0 0. 5 5 0. 2 0 0. 0 0 1. 0 5 0. 9 0 0. 9 5 1e−06 0. 8 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 3e−05 Threshold Figure 36: FMR calibration curves on unmasked and masked images. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS 47 References [1] ISO/IEC 19794-5:2011 - Information technology - Biometric data interchange formats - Part 5: Face image data. [2] N95 Respirators, Surgical Masks, and Face Masks. https://www.fda. gov/medical-devices/personal-protective-equipment-infection-control/ n95-respirators-surgical-masks-and-face-masks. [3] NIST Special Database 32 - Multiple Encounter Dataset (MEDS-II). https://www.nist.gov/itl/iad/ image-group/special-database-32-multiple-encounter-dataset-meds. This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 [4] Patrick Grother, George Quinn, and Mei Ngan. Face in video evaluation (five) face recognition of noncooperative subjects. Interagency Report 8173, National Institute of Standards and Technology, March 2017. https://doi.org/10.6028/NIST.IR.8173. [5] Davis E. King. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10:1755–1758, 2009. http: //dlib.net. [6] Patrick Grother and Mei Ngan and Kayee Hanaoka. NIST Ongoing Face Recognition Vendor Test (FRVT) 1:1 Verification Application Programming Interface (API), April 2019. https://pages.nist.gov/frvt/api/FRVT_ ongoing_11_api_4.0.pdf. [7] Zhongyuan Wang, Guangcheng Wang, Baojin Huang, Zhangyang Xiong, Qi Hong, Hao Wu, Peng Yi, Kui Jiang, Nanxi Wang, Yingjiao Pei, Heling Chen, Yu Miao, Zhibing Huang, and Jinbi Liang. Masked face recognition dataset and application, 2020. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate” FRVT Appendix A - FACE RECOGNITION VENDOR TEST - FACE MASK EFFECTS 48 Dlib Masking Methodology This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8311 (a) wide, high coverage (b) round, high coverage (c) wide, medium coverage (d) round, medium coverage (e) wide, low coverage (f) round, low coverage Figure 37: This figure shows the Dlib facial points used to create the various synthetic masks used in this report. For wide masks, the specified Dlib facial points were used to generate a closed polygon and two additional points were interpolated between each dlib facial point used for smoothing purposes. For round masks, the specified Dlib facial points were used to generate an ellipse. The Dlib C++ toolkit version 19.19, configured with the common histogram of gradients (HoG)-based face detector and 68 face landmark shape predictor was used to generate the 68 facial landmarks. 2020/07/24 11:10:31 FNMR(T) FMR(T) “False non-match rate” “False match rate”