PriMeSRL-Eval: A Practical Quality Metric for Semantic Role Labeling Systems Evaluation

Semantic role labeling (SRL) identifies the predicate-argument structure in a sentence. This task is usually accomplished in four steps: predicate identification, predicate sense disambiguation, argument identification, and argument classification. Errors introduced at one step propagate to later steps. Unfortunately, the existing SRL evaluation scripts do not consider the full effect of this error propagation aspect. They either evaluate arguments independent of predicate sense (CoNLL09) or do not evaluate predicate sense at all (CoNLL05), yielding an inaccurate SRL model performance on the argument classification task. In this paper, we address key practical issues with existing evaluation scripts and propose a more strict SRL evaluation metric PriMeSRL. We observe that by employing PriMeSRL, the quality evaluation of all SoTA SRL models drops significantly, and their relative rankings also change. We also show that PriMeSRLsuccessfully penalizes actual failures in SoTA SRL models.

Towards a Predictive Patent Analytics and Evaluation Platform

The importance of patents is well recognised across many regions of the world. Many patent mining systems have been proposed, but with limited predictive capabilities. In this demo, we showcase how predictive algorithms leveraging the state-of-the-art machine learning and deep learning techniques can be used to improve understanding of patents for inventors, patent evaluators, and business analysts alike. Our demo video is available at http://ibm.biz/ecml2019-demo-patent-analytics

Assessing Regulatory Risk in Personal Financial Advice Documents: a Pilot Study

Assessing regulatory compliance of personal financial advice is currently a complex manual process. In Australia, only 5%- 15% of advice documents are audited annually and 75% of these are found to be non-compliant(ASI 2018b). This paper describes a pilot with an Australian government regulation agency where Artificial Intelligence (AI) models based on techniques such natural language processing (NLP), machine learning and deep learning were developed to methodically characterise the regulatory risk status of personal financial advice documents. The solution provides traffic light rating of advice documents for various risk factors enabling comprehensive coverage of documents in the review and allowing rapid identification of documents that are at high risk of non-compliance with government regulations. This pilot serves as a case study of public-private partnership in developing AI systems for government and public sector.

Document Chunking and Learning Objective Generation for Instruction Design

Instructional Systems Design is the practice of creating of instructional experiences that make the acquisition of knowledge and skill more efficient, effective, and appealing. Specifically in designing courses, an hour of training material can require between 30 to 500 hours of effort in sourcing and organizing reference data for use in just the preparation of course material. In this paper, we present the first system of its kind that helps reduce the effort associated with sourcing reference material and course creation. We present algorithms for document chunking and automatic generation of learning objectives from content, creating descriptive content metadata to improve content-discoverability. Unlike existing methods, the learning objectives generated by our system incorporate pedagogically motivated Bloom's verbs. We demonstrate the usefulness of our methods using real world data from the banking industry and through a live deployment at a large pharmaceutical company.

End-to-end Network for Twitter Geolocation Prediction and Hashing

We propose an end-to-end neural network to predict the geolocation of a tweet. The network takes as input a number of raw Twitter metadata such as the tweet message and associated user account information. Our model is language independent, and despite minimal feature engineering, it is interpretable and capable of learning location indicative words and timing patterns. Compared to state-of-the-art systems, our model outperforms them by 2%-6%. Additionally, we propose extensions to the model to compress representation learnt by the network into binary codes. Experiments show that it produces compact codes compared to benchmark hashing algorithms. An implementation of the model is released publicly.