Publications

2025

1. CCL 2025

   Unveiling the Linguistic Acceptability Judgments of Large Language Models in Multilingual Contexts

   Fuyu Xing, Haoyu Huang, Dawei Mo, Xinzhuo Yang, and 4 more authors

   The 24th China National Conference on Computational Linguistics (CCL 2025), Aug 2025
2. ACL 2025

   A Survey of Large Language Models in Psychotherapy: Current Landscape and Future Directions

   Hongbin Na^*, Yining Hua^* , Zimu Wang^*, Tao Shen, and 5 more authors

   Findings of the Association for Computational Linguistics: ACL 2025 (ACL 2025 Findings), Jul 2025

   Abs arXiv

   Mental health remains a critical global challenge, with increasing demand for accessible, effective interventions. Large language models (LLMs) offer promising solutions in psychotherapy by enhancing the assessment, diagnosis, and treatment of mental health conditions through dynamic, context-aware interactions. This survey provides a comprehensive overview of the current landscape of LLM applications in psychotherapy, highlighting the roles of LLMs in symptom detection, severity estimation, cognitive assessment, and therapeutic interventions. We present a novel conceptual taxonomy to organize the psychotherapy process into three core components: assessment, diagnosis, and treatment, and examine the challenges and advancements in each area. The survey also addresses key research gaps, including linguistic biases, limited disorder coverage, and underrepresented therapeutic models. Finally, we discuss future directions to integrate LLMs into a holistic, end-to-end psychotherapy framework, addressing the evolving nature of mental health conditions and fostering more inclusive, personalized care.

3. Preprint

   LMR-BENCH: Evaluating LLM Agent’s Ability on Reproducing Language Modeling Research

   Shuo Yan^* , Ruochen Li^*, Ziming Luo^* , Zimu Wang^*, and 10 more authors

   arXiv preprint arXiv:2506.17335, Jun 2025

   Abs arXiv

   Large language model (LLM) agents have demonstrated remarkable potential in advancing scientific discovery. However, their capability in the fundamental yet crucial task of reproducing code from research papers, especially in the NLP domain, remains underexplored. This task includes unique complex reasoning challenges in the intellectual synthesis of abstract concepts and the comprehension of code repositories with interdependent files. Motivated by this gap, we present LMR-BENCH, a benchmark designed to systematically evaluate the capability of LLM agents on code reproduction from Language Modeling Research. It consists of 28 code reproduction tasks derived from 23 research papers published in top-tier NLP venues over the past five years, spanning nine fundamental categories. Models are provided with a research paper, a code repository containing one or more masked functions, and instructions for implementing these functions. We conduct extensive experiments in standard prompting and LLM agent settings with state-of-the-art LLMs, evaluating the accuracy of unit tests and performing LLM-based evaluation of code correctness. Experimental results reveal that even the most advanced models still exhibit persistent limitations in scientific reasoning and code synthesis, highlighting critical gaps in LLM agents’ ability to autonomously reproduce scientific research.

4. CSCWD 2025

   MMKNet: A Multi-Modal Knowledge Network for Predicting Both Seen and Unseen Classes in Medical Imaging

   Wenqi Xu, Hong-Seng Gan^†, Shengen Wu , Zimu Wang, and 2 more authors

   2025 28th International Conference on Computer Supported Cooperative Work in Design (CSCWD 2025), May 2025
5. CSCWD 2025

   Skin Disease Classification with LVLMs: An Empirical Study

   Xinyi Zeng , Zimu Wang, Haiyang Zhang^†, Yiming Luo, and 1 more author

   2025 28th International Conference on Computer Supported Cooperative Work in Design (CSCWD 2025), May 2025
6. WS 2025

   From Posts to Timelines: Modeling Mental Health Dynamics from Social Media Timelines with Hybrid LLMs

   Zimu Wang, Hongbin Na, Rena Gao, Jiayuan Ma, and 3 more authors

   The 10th Workshop on Computational Linguistics and Clinical Psychology (CLPsych@NAACL 2025), Apr 2025
7. WS 2025

   Efficient Document-level Event Relation Extraction

   Ruochen Li , Zimu Wang, and Xinya Du

   The 10th Workshop on Representation Learning for NLP (RepL4NLP@NAACL 2025), Apr 2025
8. WIREs DMKD

   A Review on Medical Image Segmentation: Datasets, Technical Models, Challenges and Solutions

   Hong-Seng Gan, Muhammad Hanif Ramlee , Zimu Wang, and Akinobu Shimizu

   WIREs Data Mining and Knowledge Discovery, Mar 2025

   Abs HTML

   Medical image segmentation is prerequisite in computer-aided diagnosis. As the field experiences tremendous paradigm changes since the introduction of foundation models, technicality of deep medical segmentation model is no longer a privilege limited to computer science researchers. A comprehensive educational resource suitable for researchers of broad, different backgrounds such as biomedical and medicine, is needed. This review strategically covers the evolving trends that happens to different fundamental components of medical image segmentation such as the emerging of multimodal medical image datasets, updates on deep learning libraries, classical-to-contemporary development in deep segmentation models and latest challenges with focus on enhancing the interpretability and generalizability of model. Last, the conclusion section highlights on future trends in deep medical segmentation that worth further attention and investigations.

9. AAAI 2025

   Template-Driven LLM-Paraphrased Framework for Tabular Math Word Problem Generation

   Xiaoqiang Kang^* , Zimu Wang^*, Xiaobo Jin , Wei Wang, and 2 more authors

   The 39th Annual AAAI Conference on Artificial Intelligence (AAAI 2025), Feb 2025

   Abs arXiv

   Solving tabular math word problems (TMWPs) has become a critical role in evaluating the mathematical reasoning ability of large language models (LLMs), where large-scale TMWP samples are commonly required for LLM fine-tuning. Since the collection of high-quality TMWP datasets is costly and time-consuming, recent research has concentrated on automatic TMWP generation. However, current generated samples usually suffer from issues of either correctness or diversity. In this paper, we propose a Template-driven LLM-paraphrased (TeLL) framework for generating high-quality TMWP samples with diverse backgrounds and accurate tables, questions, answers, and solutions. To this end, we first extract templates from existing real samples to generate initial problems, ensuring correctness. Then, we adopt an LLM to extend templates and paraphrase problems, obtaining diverse TMWP samples. Furthermore, we find the reasoning annotation is important for solving TMWPs. Therefore, we propose to enrich each solution with illustrative reasoning steps. Through the proposed framework, we construct a high-quality dataset TabMWP-TeLL by adhering to the question types in the TabMWP dataset, and we conduct extensive experiments on a variety of LLMs to demonstrate the effectiveness of TabMWP-TeLL in improving TMWP solving performance. The code and data of this paper are available at: https://github.com/Jason8Kang/TELL.

10. COLING 2025

    Detecting Conversational Mental Manipulation with Intent-Aware Prompting

    Jiayuan Ma^*, Hongbin Na^* , Zimu Wang, Yining Hua, and 3 more authors

    The 31st International Conference on Computational Linguistics (COLING 2025), Jan 2025

    Abs arXiv HTML PDF

    Mental manipulation severely undermines mental wellness by covertly and negatively distorting decision-making. While there is an increasing interest in mental health care within the natural language processing community, progress in tackling manipulation remains limited due to the complexity of detecting subtle, covert tactics in conversations. In this paper, we propose Intent-Aware Prompting (IAP), a novel approach for detecting mental manipulations using large language models (LLMs), providing a deeper understanding of manipulative tactics by capturing the underlying intents of participants. Experimental results on the MentalManip dataset demonstrate superior effectiveness of IAP against other advanced prompting strategies. Notably, our approach substantially reduces false negatives, helping detect more instances of mental manipulation with minimal misjudgment of positive cases. The code of this paper is available at https://github.com/Anton-Jiayuan-MA/Manip-IAP.

2024

1. NeurIPS 2024

   MEQA: A Benchmark for Multi-hop Event-centric Question Answering with Explanations

   Ruosen Li , Zimu Wang, Son Quoc Tran, Lei Xia, and 1 more author

   The 38th Annual Conference on Neural Information Processing Systems (NeurIPS 2024), Dec 2024

   Abs

   Existing benchmarks for multi-hop question answering (QA) primarily evaluate models based on their ability to reason about entities and the relationships between them. However, there’s a lack of insight into how these models perform in terms of both events and entities. In this paper, we introduce a novel semi-automatic question generation strategy by composing event structures from information extraction (IE) datasets and present the first Multi-hop Event-centric Question Answering (MEQA) benchmark. It contains (1) 2,243 challenging questions that require a diverse range of complex reasoning over entity-entity, entity-event, and event-event relations; (2) corresponding multi-step QA-format event reasoning chain (explanation) which leads to the answer for each question. We also introduce two metrics for evaluating explanations: completeness and logical consistency. We conduct comprehensive benchmarking and analysis, which shows that MEQA is challenging for the latest state-of-the-art models encompassing large language models (LLMs); and how they fall short of providing faithful explanations of the event-centric reasoning process.

2. PACLIC 2024

   Domain-specific Guided Summarization for Mental Health Posts

   Lu Qian , Yuqi Wang , Zimu Wang, Haiyang Zhang, and 3 more authors

   The 38th Pacific Asia Conference on Language, Information and Computation (PACLIC 2024), Dec 2024

   Abs

   In domain-specific contexts, particularly mental health, abstractive summarization requires advanced techniques adept at handling specialized content to generate domain-relevant and faithful summaries. In response to this, we introduce a guided summarizer equipped with a dual-encoder and an adapted decoder that utilizes novel domain-specific guidance signals, i.e., mental health terminologies and contextually rich sentences from the source document, to enhance its capacity to align closely with the content and context of guidance, thereby generating a domain-relevant summary. Additionally, we present a post-editing correction model to rectify errors in the generated summary, thus enhancing its consistency with the original content in detail. Evaluation on the MentSum dataset reveals that our model outperforms existing baseline models in terms of both ROUGE and FactCC scores. Although the experiments are specifically designed for mental health posts, the methodology we’ve developed offers broad applicability, highlighting its versatility and effectiveness in producing high-quality domain-specific summaries.

3. UIC 2024

   Guardians of Discourse: Evaluating LLMs on Multilingual Offensive Language Detection

   Jianfei He , Lilin Wang , Jiaying Wang , Zhenyu Liu, and 4 more authors

   2024 IEEE 21st International Conference on Ubiquitous Intelligence and Computing (UIC 2024), Dec 2024

   Abs arXiv

   Identifying offensive language is essential for maintaining safety and sustainability in the social media era. Though large language models (LLMs) have demonstrated encouraging potential in social media analytics, they lack thorough evaluation when in offensive language detection, particularly in multilingual environments. We for the first time evaluate multilingual offensive language detection of LLMs in three languages: English, Spanish, and German with three LLMs, GPT-3.5, Flan-T5, and Mistral, in both monolingual and multilingual settings. We further examine the impact of different prompt languages and augmented translation data for the task in non-English contexts. Furthermore, we discuss the impact of the inherent bias in LLMs and the datasets in the mispredictions related to sensitive topics.

4. UIC 2024

   Language-based Audio Retrieval with Co-Attention Networks

   Haoran Sun , Zimu Wang , Qiuyi Chen , Jianjun Chen, and 2 more authors

   2024 IEEE 21st International Conference on Ubiquitous Intelligence and Computing (UIC 2024), Dec 2024

   Abs arXiv

   In recent years, user-generated audio content has proliferated across various media platforms, creating a growing need for efficient retrieval methods that allow users to search for audio clips using natural language queries. This task, known as language-based audio retrieval, presents significant challenges due to the complexity of learning semantic representations from heterogeneous data across both text and audio modalities. In this work, we introduce a novel framework for the language-based audio retrieval task that leverages co-attention mechanismto jointly learn meaningful representations from both modalities. To enhance the model’s ability to capture fine-grained cross-modal interactions, we propose a cascaded co-attention architecture, where co-attention modules are stacked or iterated to progressively refine the semantic alignment between text and audio. Experiments conducted on two public datasets show that the proposed method can achieve better performance than the state-of-the-art method. Specifically, our best performed co-attention model achieves a 16.6% improvement in mean Average Precision on Clotho dataset, and a 15.1% improvement on AudioCaps.

5. ICONIP 2024

   MonoTCM: Semantic-Depth Fusion Transformer for Monocular 3D Object Detection with Token Clustering and Merging

   Changyu Zeng , Zimu Wang, Jimin Xiao, Anh Nguyen, and 3 more authors

   The 31st International Conference on Neural Information Processing (ICONIP 2024), Dec 2024

   Abs

   Monocular 3D object detection presents significant challenges due to the inherent absence of depth and geometric information, rendering it more complex than 2D detection. This paper introduces MonoTCM, a Semantic-Depth Fusion Transformer that leverages a Token Clustering and Merging (TCM) module to enhance the efficiency and accuracy of monocular 3D object detection. The TCM module aggregates multi-scale grid-based tokens into clustering-based tokens, dynamically adjusting their shapes and sizes based on local density and distance metrics. This allows for finer granularity in critical areas while consolidating less informative regions. The aggregated tokens are subsequently decomposed into semantic and depth features, processed through dedicated transformer-based encoders, and integrated using a semantic-depth fusion decoder modeled after DETR. This approach enhances the model’s ability to capture implicit global geometric information and provides a cost-effective solution for real-time intelligent driving applications. Experimental results demonstrate the superiority of MonoTCM in enhancing detection performance compared to other advanced methods, highlighting its potential to advance the field of monocular 3D object detection.

6. EMNLP 2024

   Document-level Causal Relation Extraction with Knowledge-guided Binary Question Answering

   Zimu Wang, Lei Xia , Wei Wang, and Xinya Du

   Findings of the Association for Computational Linguistics: EMNLP 2024 (EMNLP 2024 Findings), Nov 2024

   Abs arXiv HTML PDF

   As an essential task in information extraction (IE), Event-Event Causal Relation Extraction (ECRE) aims to identify and classify the causal relationships between event mentions in natural language texts. However, existing research on ECRE has highlighted two critical challenges, including the lack of document-level modeling and causal hallucinations. In this paper, we propose a Knowledge-guided binary Question Answering (KnowQA) method with event structures for ECRE, consisting of two stages: Event Structure Construction and Binary Question Answering. We conduct extensive experiments under both zero-shot and fine-tuning settings with large language models (LLMs) on the MECI and MAVEN-ERE datasets. Experimental results demonstrate the usefulness of event structures on document-level ECRE and the effectiveness of KnowQA by achieving state-of-the-art on the MECI dataset. We observe not only the effectiveness but also the high generalizability and low inconsistency of our method, particularly when with complete event structures after fine-tuning the models.

7. INLG 2024

   MTSwitch: A Web-based System for Translation between Molecules and Texts

   Nijia Han , Zimu Wang , Yuqi Wang, Haiyang Zhang, and 2 more authors

   The 17th International Natural Language Generation Conference: System Demonstrations (INLG 2024 Demo), Sep 2024

   Abs HTML PDF

   We introduce MTSwitch, a web-based system for the bidirectional translation between molecules and texts, leveraging various large language models (LLMs). It supports two crucial tasks, including molecule captioning (explaining the properties of a molecule) and molecule generation (designing a molecule based on specific properties). To the best of our knowledge, MTSwitch is currently the first accessible system that allows users to translate between molecular representations and descriptive text contents. The system and a screencast can be found in https://github.com/hanninaa/MTSwitch.

8. WS 2024

   Knowledge Distillation from Monolingual to Multilingual Models for Intelligent and Interpretable Multilingual Emotion Detection

   Yuqi Wang , Zimu Wang, Nijia Han , Wei Wang^†, and 4 more authors

   The 14th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis (WASSA@ACL 2024), Aug 2024

   Abs HTML PDF

   Emotion detection from text is a crucial task in understanding natural language with wide-ranging applications. Existing approaches for multilingual emotion detection from text face challenges with data scarcity across many languages and a lack of interpretability. We propose a novel method that leverages both monolingual and multilingual pre-trained language models to improve performance and interpretability. Our approach involves 1) training a high-performing English monolingual model in parallel with a multilingual model and 2) using knowledge distillation to transfer the emotion detection capabilities from the monolingual teacher to the multilingual student model. Experiments on a multilingual dataset demonstrate significant performance gains for refined multilingual models like XLM-RoBERTa and E5 after distillation. Furthermore, our approach enhances interpretability by enabling better identification of emotion-trigger words. Our work presents a promising direction for building accurate, robust and explainable multilingual emotion detection systems.

9. WS 2024

   Knowledge Base-enhanced Multilingual Relation Extraction with Large Language Models

   Tong Chen, Procheta Sen , Zimu Wang, Zhengyong Jiang^†, and 1 more author

   The First International OpenKG Workshop on Large Knowledge-Enhanced Models (LKM@IJCAI 2024), Aug 2024

   Abs PDF

   Relation Extraction (RE) is an essential task that involves comprehending relational facts between entities from natural language texts. However, existing research in RE, particularly those based on large language models (LLMs), is proven to fall short in the task due to their context unawareness (lack of fine-grained understanding), schema misalignment (misaligned with human-defined schema), and world knowledge ignorance (relying solely on internal parametric knowledge). In this paper, we propose a novel framework to address the aforementioned challenges. The framework consists of two stages, including 1) entity linking and 2) relation inference, by fully leveraging the efficacy of external knowledge bases (KBs) and LLMs in this task. We conduct extensive experiments in a multilingual setting and achieve state-of-the-art performance on the experimented datasets. The LLMs with external knowledge can typically outperform those without knowledge by a significant margin, indicating the effectiveness of our proposed framework.

10. CCL 2024

    Exploring Faithful and Informative Commonsense Reasoning and Moral Understanding in Children’s Stories

    Zimu Wang , Yuqi Wang, Nijia Han, Qi Chen, and 4 more authors

    The 23rd China National Conference on Computational Linguistics: Evaluations (CCL24-Eval), Jul 2024

    Abs HTML PDF

    Commonsense reasoning and moral understanding are crucial tasks in artificial intelligence (AI) and natural language processing (NLP). However, existing research often falls short in terms of faithfulness and informativeness during the reasoning process. We propose a novel framework for performing commonsense reasoning and moral understanding using large language models (LLMs), involving constructing guided prompts by incorporating relevant knowledge for commonsense reasoning and extracting facts from stories for moral understanding. We conduct extensive experiments on the Commonsense Reasoning and Moral Understanding in Children’s Stories (CRMUS) dataset with widely recognised LLMs under both zero-shot and fine-tuning settings, demonstrating the effectiveness of our proposed method. Furthermore, we analyse the adaptability of different LLMs in extracting facts for moral understanding performance.

11. CSCWD 2024

    Generating Valid and Natural Adversarial Examples with Large Language Models

    Zimu Wang , Wei Wang^†, Qi Chen, Qiufeng Wang, and 1 more author

    2024 27th International Conference on Computer Supported Cooperative Work in Design (CSCWD 2024), May 2024

    Abs arXiv HTML

    Deep learning-based natural language processing (NLP) models, particularly pre-trained language models (PLMs), have been revealed to be vulnerable to adversarial attacks. However, the adversarial examples generated by many mainstream word-level adversarial attack models are neither valid nor natural, leading to the loss of semantic maintenance, grammaticality, and human imperceptibility. Based on the exceptional capacity of language understanding and generation of large language models (LLMs), we propose LLM-Attack, which aims at generating both valid and natural adversarial examples with LLMs. The method consists of two stages: word importance ranking (which searches for the most vulnerable words) and word synonym replacement (which substitutes them with their synonyms obtained from LLMs). Experimental results on the Movie Review (MR), IMDB, and Yelp Review Polarity datasets against the baseline adversarial attack models illustrate the effectiveness of LLM-Attack, and it outperforms the baselines in human and GPT-4 evaluation by a significant margin. The model can generate adversarial examples that are typically valid and natural, with the preservation of semantic meaning, grammaticality, and human imperceptibility.

12. EACL 2024

    FinBPM: A Framework for Portfolio Management-based Financial Investor Behavior Perception Model

    Zhilu Zhang, Procheta Sen^† , Zimu Wang, Ruoyu Sun, and 2 more authors

    The 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2024), Mar 2024

    Abs HTML PDF

    The goal of portfolio management is to simultaneously maximize the accumulated return and also to control risk. In consecutive trading periods, portfolio manager needs to continuously adjust the portfolio weights based on the factors which can cause price fluctuation in the market. In the stock market, the factors affecting the stock price can be divided into two categories. The first is price fluctuations caused by irrational investment of the speculators. The second is endogenous value changes caused by operations of the company. In recent years, with the advancement of artificial intelligence technology, reinforcement learning (RL) algorithms have been increasingly employed by scholars to address financial problems, particularly in the area of portfolio management. However, the deep RL models proposed by these scholars in the past have focused more on analyzing the price changes caused by the investment behavior of speculators in response to technical indicators of actual stock prices. In this research, we introduce an RL-based framework called FinBPM, which takes both the factor pertaining to the impact on operations of the company and the factor of the irrational investment of the speculator into consideration. For our experimentation, we randomly selected twelve stocks from the Dow Jones Industrial Index to construct our portfolio. The experimental results reveal that, in comparison to conventional reinforcement learning methods, our approach with at least 13.26% increase over other methods compared. Additionally, it achieved the best Sharpe ratio of 2.77, effectively maximizing the return per unit of risk.

13. Preprint

    Rethinking Human-like Translation Strategy: Integrating Drift-Diffusion Model with Large Language Models for Machine Translation

    Hongbin Na , Zimu Wang, Mieradilijiang Maimaiti^† , Tong Chen, and 3 more authors

    arXiv preprint arXiv:2402.10699, Feb 2024

    Abs arXiv

    Large language models (LLMs) have demonstrated promising potential in various downstream tasks, including machine translation. However, prior work on LLM-based machine translation has mainly focused on better utilizing training data, demonstrations, or pre-defined and universal knowledge to improve performance, with a lack of consideration of decision-making like human translators. In this paper, we incorporate Thinker with the Drift-Diffusion Model (Thinker-DDM) to address this issue. We then redefine the Drift-Diffusion process to emulate human translators’ dynamic decision-making under constrained resources. We conduct extensive experiments under the high-resource, low-resource, and commonsense translation settings using the WMT22 and CommonMT datasets, in which Thinker-DDM outperforms baselines in the first two scenarios. We also perform additional analysis and evaluation on commonsense translation to illustrate the high effectiveness and efficacy of the proposed method.

2023

1. EMNLP 2023

   OmniEvent: A Comprehensive, Fair, and Easy-to-Use Toolkit for Event Understanding

   Hao Peng^* , Xiaozhi Wang^*, Feng Yao , Zimu Wang, and 4 more authors

   The 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP 2023 Demo), Dec 2023

   Abs arXiv HTML PDF

   Event understanding aims at understanding the content and relationship of events within texts, which covers multiple complicated information extraction tasks: event detection, event argument extraction, and event relation extraction. To facilitate related research and application, we present an event understanding toolkit OmniEvent, which features three desiderata: (1) Comprehensive. OmniEvent supports mainstream modeling paradigms of all the event understanding tasks and the processing of 15 widely-used English and Chinese datasets. (2) Fair. OmniEvent carefully handles the inconspicuous evaluation pitfalls reported in Peng et al. (2023), which ensures fair comparisons between different models. (3) Easy-to-use. OmniEvent is designed to be easily used by users with varying needs. We provide off-the-shelf models that can be directly deployed as web services. The modular framework also enables users to easily implement and evaluate new event understanding models with OmniEvent. The toolkit is publicly released along with the demonstration website and video.

2. BIBE 2023

   Attention-based Multimodal Bilinear Feature Fusion for Lung Cancer Survival Analysis

   Hongbin Na^* , Lilin Wang^*, Xinyao Zhuang, Jianfei He, and 3 more authors

   2023 IEEE 23rd International Conference on Bioinformatics and Bioengineering (BIBE 2023), Dec 2023

   Abs HTML

   Survival analysis (SA) is an essential task that aims to predict survival status and duration, determine individual and precise treatment strategies, and assess disease intensity and direction. However, the current research on multimodal SA has identified three unique challenges: inefficient cross-modal information integration, insufficient inter-modal key features, and noisy data. In this paper, we propose a novel SA framework, named Attention-based Multimodal Bilinear Feature Fusion (AMBF)-SA, to address the aforementioned challenges. Specifically, AMBF-SA first performs feature extraction with the off-the-shelf models on each modality separately, then fuses the features between multiple sources and modalities using our proposed AMBF method, and finally outputs the survival prediction by a multi-layer perception (MLP). Experimental results on the Non-small Cell Lung Cancer (NSCLC) Radiogenomics dataset demonstrate remark performance of AMBF-SA compared with the rest of the experimented models, including the models trained with single and combined modalities under the Mean Absolute Error (MAE) and the Concordance Index (C-index) evaluation metrics, indicating the usefulness of our proposed framework.

3. Preprint

   When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks

   Hao Peng^* , Xiaozhi Wang^* , Jianhui Chen^* , Weikai Li, and 7 more authors

   arXiv preprint arXiv:2311.08993, Nov 2023

   Abs arXiv

   In-context learning (ICL) has become the default method for using large language models (LLMs), making the exploration of its limitations and understanding the underlying causes crucial. In this paper, we find that ICL falls short of handling specification-heavy tasks, which are tasks with complicated and extensive task specifications, requiring several hours for ordinary humans to master, such as traditional information extraction tasks. The performance of ICL on these tasks mostly cannot reach half of the state-of-the-art results. To explore the reasons behind this failure, we conduct comprehensive experiments on 18 specification-heavy tasks with various LLMs and identify three primary reasons: inability to specifically understand context, misalignment in task schema comprehension with humans, and inadequate long-text understanding ability. Furthermore, we demonstrate that through fine-tuning, LLMs can achieve decent performance on these tasks, indicating that the failure of ICL is not an inherent flaw of LLMs, but rather a drawback of existing alignment methods that renders LLMs incapable of handling complicated specification-heavy tasks via ICL. To substantiate this, we perform dedicated instruction tuning on LLMs for these tasks and observe a notable improvement. We hope the analyses in this paper could facilitate advancements in alignment methods enabling LLMs to meet more sophisticated human demands.

4. CCAI 2023

   Multi-level Adversarial Training for Stock Sentiment Prediction

   Zimu Wang, and Hong-Seng Gan

   2023 IEEE 3rd International Conference on Computer Communication and Artificial Intelligence (CCAI 2023), May 2023

   Abs HTML

   Stock sentiment prediction is a task to evaluate whether the investors are expecting or gaining a positive or negative return from a stock, which has a high correlation with investors’ sentiments towards the business. However, as the nature of social media, the textual information posted by ordinary people is usually noisy, inconsistent, and even grammatically incorrect, leading the model to generate unsatisfied predictions. In this paper, we improve the performance of stock sentiment prediction by applying and comparing adversarial training at multiple levels, including character, word, and sentence levels, with the utilization of three novel adversarial attack models: DeepWordBug, BAE, and Generative Adversarial Network (GAN). We also propose an effective pre-processing technique and a novel adversarial examples incorporation method to improve the prediction results. To make an objective evaluation, we select three backbone models: Embedding Bag, BERT, and RoBERTa-Twitter, and validate the models before and after adversarial training on the TweetFinSent dataset. Experimental results demonstrate remarkable improvements in the models after adversarial training, and the RoBERTa-Twitter model with word-level adversarial training performs optimally among the experimented models. We conclude that sentence-level and word-level adversarial training are the most appropriate for deep learning and pre-trained language models, respectively, and we further conduct ablation studies to highlight the usefulness of our data pre-processing and adversarial examples incorporation approaches and a case study to display the adversarial examples generated by the proposed adversarial attack models.

2022

1. EMNLP 2022

   MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction

   Xiaozhi Wang^* , Yulin Chen^*, Ning Ding, Hao Peng, and 8 more authors

   The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022), Dec 2022

   Abs arXiv HTML PDF

   The diverse relationships among real-world events, including coreference, temporal, causal, and subevent relations, are fundamental to understanding natural languages. However, two drawbacks of existing datasets limit event relation extraction (ERE) tasks: (1) Small scale. Due to the annotation complexity, the data scale of existing datasets is limited, which cannot well train and evaluate data-hungry models. (2) Absence of unified annotation. Different types of event relations naturally interact with each other, but existing datasets only cover limited relation types at once, which prevents models from taking full advantage of relation interactions. To address these issues, we construct a unified large-scale human-annotated ERE dataset MAVEN-ERE with improved annotation schemes. It contains 103,193 event coreference chains, 1,216,217 temporal relations, 57,992 causal relations, and 15,841 subevent relations, which is larger than existing datasets of all the ERE tasks by at least an order of magnitude. Experiments show that ERE on MAVEN-ERE is quite challenging, and considering relation interactions with joint learning can improve performances. The dataset and source codes can be obtained from https://github.com/THU-KEG/MAVEN-ERE.

2. ICICN 2022

   Generating Complex Questions from Knowledge Graphs with Query Graphs

   Zimu Wang

   2022 IEEE 10th International Conference on Information, Communication and Networks (ICICN 2022), Aug 2022

   Abs HTML

   Question Generation from Knowledge Graphs (KGQG) is a task that aims to generate natural language questions from subgraphs within the given Knowledge Graph. Previous research has discussed numerous KGQG approaches, generating simple and complex questions from triples and queries based on the template-based and Deep Learning-based models. However, the current KGQG research could be identified three unique challenges: determining how to represent the queries, mapping different query structures to different compositional structures in the natural language questions, and generating questions with high language variety. In this paper, we propose a novel framework to conduct the KGQG task, which consists of two stages: query graph construction and graph-to-question generation, to deal with the three identified challenges. Firstly, we propose a query graph structure that specifies the entities and relationships within a SPARQL query addition with the characteristic of each entity in order to explore an appropri-ate query graph representation. We then propose a Graph-to-Sequence (Graph2Seq) model that utilizes Gated Graph Neural Network (GGNN) to encode the constructed query graphs and an attention decoder with a Long Short-Term Memory (LSTM) network to generate natural language questions. To make an objective evaluation, we validate our framework on two compli-cated datasets designed for complex questioning and answering, namely LC-QuAD 2.0 and KQA Pro. Experimental results show a dramatic improvement in our framework compared with the Sequence-to-Sequence (Seq2Seq) baselines. The robustness of our graph structure and the Graph2Seq model in the KGQG tasks are all confirmed, accompanied by a case study to highlight the effectiveness of our proposed framework.