Alert button
Picture for Courtney Mansfield

Courtney Mansfield

Alert button

Behind the Mask: Demographic bias in name detection for PII masking

May 09, 2022
Courtney Mansfield, Amandalynne Paullada, Kristen Howell

Figure 1 for Behind the Mask: Demographic bias in name detection for PII masking
Figure 2 for Behind the Mask: Demographic bias in name detection for PII masking
Figure 3 for Behind the Mask: Demographic bias in name detection for PII masking
Figure 4 for Behind the Mask: Demographic bias in name detection for PII masking

Many datasets contain personally identifiable information, or PII, which poses privacy risks to individuals. PII masking is commonly used to redact personal information such as names, addresses, and phone numbers from text data. Most modern PII masking pipelines involve machine learning algorithms. However, these systems may vary in performance, such that individuals from particular demographic groups bear a higher risk for having their personal information exposed. In this paper, we evaluate the performance of three off-the-shelf PII masking systems on name detection and redaction. We generate data using names and templates from the customer service domain. We find that an open-source RoBERTa-based system shows fewer disparities than the commercial models we test. However, all systems demonstrate significant differences in error rate based on demographics. In particular, the highest error rates occurred for names associated with Black and Asian/Pacific Islander individuals.

Viaarxiv icon

Disfluencies and Human Speech Transcription Errors

Apr 08, 2019
Vicky Zayats, Trang Tran, Richard Wright, Courtney Mansfield, Mari Ostendorf

Figure 1 for Disfluencies and Human Speech Transcription Errors
Figure 2 for Disfluencies and Human Speech Transcription Errors
Figure 3 for Disfluencies and Human Speech Transcription Errors
Figure 4 for Disfluencies and Human Speech Transcription Errors

This paper explores contexts associated with errors in transcrip-tion of spontaneous speech, shedding light on human perceptionof disfluencies and other conversational speech phenomena. Anew version of the Switchboard corpus is provided with disfluency annotations for careful speech transcripts, together with results showing the impact of transcription errors on evaluation of automatic disfluency detection.

* Submitted to INTERSPEECH 2019 
Viaarxiv icon