Alert button

AR-LSAT: Investigating Analytical Reasoning of Text

Apr 15, 2021
Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Jiahai Wang, Jian Yin, Ming Zhou, Nan Duan

Figure 1 for AR-LSAT: Investigating Analytical Reasoning of Text
Figure 2 for AR-LSAT: Investigating Analytical Reasoning of Text
Figure 3 for AR-LSAT: Investigating Analytical Reasoning of Text
Figure 4 for AR-LSAT: Investigating Analytical Reasoning of Text

Share this with someone who'll enjoy it:

Analytical reasoning is an essential and challenging task that requires a system to analyze a scenario involving a set of particular circumstances and perform reasoning over it to make conclusions. In this paper, we study the challenge of analytical reasoning of text and introduce a new dataset consisting of questions from the Law School Admission Test from 1991 to 2016. We analyze what knowledge understanding and reasoning abilities are required to do well on this task. Furthermore, to address this reasoning challenge, we design two different baselines: (1) a Transformer-based method which leverages the state-of-the-art pre-trained language models and (2) Analytical Reasoning Machine (ARM), a logical-level reasoning framework extracting symbolic knowledge (e.g, participants, facts, logical functions) to deduce legitimate solutions. In our experiments, we find that the Transformer-based models struggle to solve this task as their performance is close to random guess and ARM achieves better performance by leveraging symbolic knowledge and interpretable reasoning steps. Results show that both methods still lag far behind human performance, which leave further space for future research.

* 13 pages, 5 figures  
View paper onarxiv icon

Share this with someone who'll enjoy it: