Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox

Picture for Samuel R. Bowman

Comparing Test Sets with Item Response Theory


Jun 01, 2021
Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, Samuel R. Bowman

* ACL 2021 

  Access Paper or Ask Questions

What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?


Jun 01, 2021
Nikita Nangia, Saku Sugawara, Harsh Trivedi, Alex Warstadt, Clara Vania, Samuel R. Bowman

* ACL 2021 

  Access Paper or Ask Questions

Does Putting a Linguist in the Loop Improve NLU Data Collection?


Apr 15, 2021
Alicia Parrish, William Huang, Omar Agha, Soo-Hwan Lee, Nikita Nangia, Alex Warstadt, Karmanya Aggarwal, Emily Allaway, Tal Linzen, Samuel R. Bowman

* 14 pages, 10 figures 

  Access Paper or Ask Questions

What Will it Take to Fix Benchmarking in Natural Language Understanding?


Apr 10, 2021
Samuel R. Bowman, George E. Dahl

* To appear at NAACL 2021 

  Access Paper or Ask Questions

When Do You Need Billions of Words of Pretraining Data?


Nov 10, 2020
Yian Zhang, Alex Warstadt, Haau-Sing Li, Samuel R. Bowman

* 10 pages, 6 figures 

  Access Paper or Ask Questions

Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options


Oct 13, 2020
Clara Vania, Ruijie Chen, Samuel R. Bowman

* AACL 2020 

  Access Paper or Ask Questions

Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)


Oct 11, 2020
Alex Warstadt, Yian Zhang, Haau-Sing Li, Haokun Liu, Samuel R. Bowman

* accepted at EMNLP 2020 

  Access Paper or Ask Questions

Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data


Oct 09, 2020
William Huang, Haokun Liu, Samuel R. Bowman


  Access Paper or Ask Questions

Precise Task Formalization Matters in Winograd Schema Evaluations


Oct 08, 2020
Haokun Liu, William Huang, Dhara A. Mungra, Samuel R. Bowman

* Accepted to the EMNLP 2020 conference 

  Access Paper or Ask Questions

CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models


Sep 30, 2020
Nikita Nangia, Clara Vania, Rasika Bhalerao, Samuel R. Bowman

* EMNLP 2020 

  Access Paper or Ask Questions

Can neural networks acquire a structural bias from raw linguistic data?


Jul 14, 2020
Alex Warstadt, Samuel R. Bowman

* To appear in Proceedings of 42nd Annual Meeting of the Cognitive Science Society 

  Access Paper or Ask Questions

Self-Training for Unsupervised Parsing with PRPN


May 27, 2020
Anhad Mohananey, Katharina Kann, Samuel R. Bowman

* Accepted for publication at the 16th International Conference on Parsing Technologies (IWPT), 2020 

  Access Paper or Ask Questions

English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too


May 26, 2020
Jason Phang, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, Iacer Calixto, Samuel R. Bowman


  Access Paper or Ask Questions

Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?


May 09, 2020
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, Samuel R. Bowman

* ACL 2020 

  Access Paper or Ask Questions

Learning to Learn Morphological Inflection for Resource-Poor Languages


Apr 28, 2020
Katharina Kann, Samuel R. Bowman, Kyunghyun Cho

* AAAI 2020 

  Access Paper or Ask Questions

Collecting Entailment Data for Pretraining: New Protocols and Negative Results


Apr 24, 2020
Samuel R. Bowman, Jennimaria Palomaki, Livio Baldini Soares, Emily Pitler


  Access Paper or Ask Questions

jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models


Mar 04, 2020
Yada Pruksachatkun, Phil Yeres, Haokun Liu, Jason Phang, Phu Mon Htut, Alex Wang, Ian Tenney, Samuel R. Bowman


  Access Paper or Ask Questions

BLiMP: A Benchmark of Linguistic Minimal Pairs for English


Dec 02, 2019
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, Samuel R. Bowman


  Access Paper or Ask Questions

Do Attention Heads in BERT Track Syntactic Dependencies?


Nov 27, 2019
Phu Mon Htut, Jason Phang, Shikha Bordia, Samuel R. Bowman


  Access Paper or Ask Questions

Inducing Constituency Trees through Neural Machine Translation


Sep 22, 2019
Phu Mon Htut, Kyunghyun Cho, Samuel R. Bowman


  Access Paper or Ask Questions

Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs


Sep 19, 2019
Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretič, Samuel R. Bowman

* Accepted to EMNLP 2019; Added link to code+dataset 

  Access Paper or Ask Questions

Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set


Sep 15, 2019
Katharina Kann, Kyunghyun Cho, Samuel R. Bowman

* EMNLP 2019 

  Access Paper or Ask Questions

Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark


Jun 01, 2019
Nikita Nangia, Samuel R. Bowman

* ACL 2019 

  Access Paper or Ask Questions

What do you learn from context? Probing for sentence structure in contextualized word representations


May 15, 2019
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, Ellie Pavlick

* ICLR 2019 camera-ready version, 17 pages including appendices 

  Access Paper or Ask Questions

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems


May 02, 2019
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R. Bowman

* super.gluebenchmark.com 

  Access Paper or Ask Questions

Probing What Different NLP Tasks Teach Machines about Function Word Comprehension


Apr 25, 2019
Najoung Kim, Roma Patel, Adam Poliak, Alex Wang, Patrick Xia, R. Thomas McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, Ellie Pavlick

* Accepted to *SEM 2019 

  Access Paper or Ask Questions