Alert button
Picture for Junda Wu

Junda Wu

Alert button

FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis

Jul 31, 2023
Ziao Wang, Yuhang Li, Junda Wu, Jaehyeon Soon, Xiaofeng Zhang

Figure 1 for FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis
Figure 2 for FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis
Figure 3 for FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis
Figure 4 for FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis

In this paper, we propose FinVis-GPT, a novel multimodal large language model (LLM) specifically designed for financial chart analysis. By leveraging the power of LLMs and incorporating instruction tuning and multimodal capabilities, FinVis-GPT is capable of interpreting financial charts and providing valuable analysis. To train FinVis-GPT, a financial task oriented dataset was generated for pre-training alignment and instruction tuning, comprising various types of financial charts and their corresponding descriptions. We evaluate the model performance via several case studies due to the time limit, and the promising results demonstrated that FinVis-GPT is superior in various financial chart related tasks, including generating descriptions, answering questions and predicting future market trends, surpassing existing state-of-the-art multimodal LLMs. The proposed FinVis-GPT serves as a pioneering effort in utilizing multimodal LLMs in the finance domain and our generated dataset will be release for public use in the near future to speedup related research.

* (FinLLM 2023)@IJCAI 2023 
Viaarxiv icon

An Effective Data Creation Pipeline to Generate High-quality Financial Instruction Data for Large Language Model

Jul 31, 2023
Ziao Wang, Jianning Wang, Junda Wu, Xiaofeng Zhang

Figure 1 for An Effective Data Creation Pipeline to Generate High-quality Financial Instruction Data for Large Language Model
Figure 2 for An Effective Data Creation Pipeline to Generate High-quality Financial Instruction Data for Large Language Model
Figure 3 for An Effective Data Creation Pipeline to Generate High-quality Financial Instruction Data for Large Language Model
Figure 4 for An Effective Data Creation Pipeline to Generate High-quality Financial Instruction Data for Large Language Model

At the beginning era of large language model, it is quite critical to generate a high-quality financial dataset to fine-tune a large language model for financial related tasks. Thus, this paper presents a carefully designed data creation pipeline for this purpose. Particularly, we initiate a dialogue between an AI investor and financial expert using ChatGPT and incorporate the feedback of human financial experts, leading to the refinement of the dataset. This pipeline yielded a robust instruction tuning dataset comprised of 103k multi-turn chats. Extensive experiments have been conducted on this dataset to evaluate the model's performance by adopting an external GPT-4 as the judge. The promising experimental results verify that our approach led to significant advancements in generating accurate, relevant, and financial-style responses from AI models, and thus providing a powerful tool for applications within the financial sector.

* (FinLLM 2023)@IJCAI 2023 
Viaarxiv icon

InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding

Jun 08, 2023
Junda Wu, Tong Yu, Rui Wang, Zhao Song, Ruiyi Zhang, Handong Zhao, Chaochao Lu, Shuai Li, Ricardo Henao

Figure 1 for InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding
Figure 2 for InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding
Figure 3 for InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding
Figure 4 for InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding

Soft prompt tuning achieves superior performances across a wide range of few-shot tasks. However, the performances of prompt tuning can be highly sensitive to the initialization of the prompts. We also empirically observe that conventional prompt tuning methods cannot encode and learn sufficient task-relevant information from prompt tokens. In this work, we develop an information-theoretic framework that formulates soft prompt tuning as maximizing mutual information between prompts and other model parameters (or encoded representations). This novel view helps us to develop a more efficient, accurate and robust soft prompt tuning method InfoPrompt. With this framework, we develop two novel mutual information based loss functions, to (i) discover proper prompt initialization for the downstream tasks and learn sufficient task-relevant information from prompt tokens and (ii) encourage the output representation from the pretrained language model to be more aware of the task-relevant information captured in the learnt prompt. Extensive experiments validate that InfoPrompt can significantly accelerate the convergence of the prompt tuning and outperform traditional prompt tuning methods. Finally, we provide a formal theoretical result for showing to show that gradient descent type algorithm can be used to train our mutual information loss.

Viaarxiv icon

Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer

May 20, 2023
Kaige Xie, Tong Yu, Haoliang Wang, Junda Wu, Handong Zhao, Ruiyi Zhang, Kanak Mahadik, Ani Nenkova, Mark Riedl

Figure 1 for Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer
Figure 2 for Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer
Figure 3 for Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer
Figure 4 for Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer

In real-world scenarios, labeled samples for dialogue summarization are usually limited (i.e., few-shot) due to high annotation costs for high-quality dialogue summaries. To efficiently learn from few-shot samples, previous works have utilized massive annotated data from other downstream tasks and then performed prompt transfer in prompt tuning so as to enable cross-task knowledge transfer. However, existing general-purpose prompt transfer techniques lack consideration for dialogue-specific information. In this paper, we focus on improving the prompt transfer from dialogue state tracking to dialogue summarization and propose Skeleton-Assisted Prompt Transfer (SAPT), which leverages skeleton generation as extra supervision that functions as a medium connecting the distinct source and target task and resulting in the model's better consumption of dialogue state information. To automatically extract dialogue skeletons as supervised training data for skeleton generation, we design a novel approach with perturbation-based probes requiring neither annotation effort nor domain knowledge. Training the model on such skeletons can also help preserve model capability during prompt transfer. Our method significantly outperforms existing baselines. In-depth analyses demonstrate the effectiveness of our method in facilitating cross-task knowledge transfer in few-shot dialogue summarization.

Viaarxiv icon