Publications

2023

1. Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias

   Yue Yu*, Yuchen Zhuang*, Jieyu Zhang*, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang

   Proceedings of NeurIPS (D&B Track), 2023.

   arXiv PDF Code
2. ToolQA: A Dataset for LLM Question Answering with External Tools

   Yuchen Zhuang*, Yue Yu*, Kuan Wang*, Haotian Sun, and Chao Zhang

   Proceedings of NeurIPS (D&B Track), 2023.

   arXiv PDF Code
3. Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms

   Alexander Bukharin, Yan Li, Yue Yu, Qingru Zhang, Zhehui Chen, Simiao Zuo, Chao Zhang, Songan Zhang, and Tuo Zhao

   Proceedings of NeurIPS, 2023.

   arXiv PDF Code
4. Cold-Start Data Selection for Better Few-shot Language Model Fine-tuning: A Prompt-based Uncertainty Propagation Approach

   Yue Yu, Rongzhi Zhang, Ran Xu, Jieyu Zhang, Jiaming Shen, and Chao Zhang

   Proceedings of ACL, 2023.

   Abs PDF Code

   We present PATRON, a prompt-based data selection method for pre-trained language model fine-tuning under cold-start scenarios, i.e., no initial labeled data are available. In PATRON, we design (1) a prompt-based uncertainty propagation approach to estimate the importance of data points and (2) a partition-then-rewrite (PTR) strategy to promote sample diversity when querying for annotations. Experiments on six text classification datasets show that PATRON outperforms the strongest cold-start data selection baselines by up to 6.9%. Besides, with 128 labels only, PATRON achieves 91.0% and 92.1% of the fully supervised performance based on vanilla fine-tuning and prompt-based learning respectively. Our implementation of PATRON will be published upon acceptance.

5. ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval

   Yue Yu, Yuchen Zhuang, Rongzhi Zhang, Yu Meng, Jiaming Shen, and Chao Zhang

   Proceedings of ACL Findings, 2023.

   Abs PDF Code

   With the development of large language models (LLMs), zero-shot learning has attracted much attention for various NLP tasks. Different from prior works that generate training data with billion-scale natural language generation (NLG) models, we propose a retrieval-enhanced framework to create training data from a general-domain unlabeled corpus. To realize this, we first conduct contrastive pretraining to learn an unsupervised dense retriever for extracting the most relevant documents using class-descriptive verbalizers. We then further pro- pose two simple strategies, namely Verbalizer Augmentation with Demonstrations and Self- consistency Guided Filtering to improve the topic coverage of the dataset while removing noisy examples. Experiments on nine datasets demonstrate that ReGen achieves 4.3% gain over the strongest baselines and saves around 70% of the time when compared with baselines using large NLG models. Besides, REGEN can be naturally integrated with recently proposed large language models to boost performance.

6. Local Boosting for Weakly-Supervised Learning

   Rongzhi Zhang, Yue Yu, Jiaming Shen, Xiquan Cui, and Chao Zhang

   Proceedings of KDD, 2023.

   arXiv PDF
7. DyGen: Fine-Tuning Language Models with Noisy Labels by Dynamics-Enhanced Generative Modeling

   Yuchen Zhuang, Yue Yu, Lingkai Kong, Xiang Chen, and Chao Zhang

   Proceedings of KDD, 2023.

   arXiv PDF Code
8. R-Mixup: Riemannian Mixup for Biological Networks

   Xuan Kan, Zimu Li, Hejie Cui, Yue Yu, Ran Xu, Shaojun Yu, Zilong Zhang, Ying Guo, and Carl Yang

   Proceedings of KDD, 2023.

   arXiv PDF
9. Weakly-supervised Scientific Document Classification via Retrieval-Augmented Multi-stage Training

   Ran Xu*, Yue Yu*, Joyce C Ho, and Carl Yang

   Proceedings of SIGIR, 2023. (Short Paper)

   arXiv Code
10. Neighborhood-regularized Self-Training for Learning with Few Labels

    Ran Xu, Yue Yu, Hejie Cui, Xuan Kan, Yanqiao Zhu, Joyce Ho, Chao Zhang, and Carl Yang

    Proceedings of AAAI, 2023. (Oral)

    arXiv PDF Code
11. Deep DAG Learning on Brain Networks for fMRI Analysis

    Yue Yu, Xuan Kan, Hejie Cui, Ran Xu, Yujia Zheng, Xiangchen Song, Yanqiao Zhu, Kun Zhang, Razieh Nabi, Ying Guo, Chao Zhang, and Carl Yang

    Proceedings of ISBI, 2023.

    arXiv PDF
12. A Survey on Knowledge Graphs for Healthcare: Resources, Application Progress, and Promise

    Hejie Cui, Jiaying Lu, Shiyu Wang, Ran Xu, Wenjing Ma, Shaojun Yu, Yue Yu, Xuan Kan, Tianfan Fu, Chen Ling, Joyce Ho, Fei Wang, and Carl Yang

    Proceedings of ICML (IMLH Workshop), 2023.

    arXiv PDF

2022

1. COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning

   Yue Yu, Chenyan Xiong, Si Sun, Chao Zhang, and Arnold Overwijk

   Proceedings of EMNLP, 2022. (Oral)

   Abs PDF Code

   We present a new zero-shot dense retrieval (ZeroDR) method, COCO-DR, to improve the generalization ability of dense retrieval by combating the distribution shifts between source training tasks and target scenarios. To mitigate the impact of document differences, COCO-DR continues pretraining the language model on the target corpora to adapt the model to target distributions via COtinuous COtrastive learning. To prepare for unseen target queries, COCO-DR leverages implicit Distributionally Robust Optimization (iDRO) to reweight samples from different source query clusters for improving model robustness over rare queries during fine-tuning. COCO-DR achieves superior average performance on BEIR, the zero-shot retrieval benchmark. At BERT_Base scale, COCO-DR Base outperforms other ZeroDR models with 60x larger size. At BERT_Large scale, COCO-DR Large outperforms the giant GPT-3 embedding model which has 500x more parameters. Our analysis shows the correlation between COCO-DR’s effectiveness in combating distribution shifts and improving zero-shot accuracy. Our code and model can be found at \urlhttps://github.com/OpenMatch/COCO-DR.

2. Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives

   Si Sun, Chenyan Xiong, Yue Yu, Arnold Overwijk, Zhiyuan Liu, and Jie Bao

   Proceedings of EMNLP, 2022.

   Abs PDF Code

   In this paper, we investigate the instability in the standard dense retrieval training, which iterates between model training and hard negative selection using the being-trained model. We show the catastrophic forgetting phenomena behind the training instability, where models learn and forget different negative groups during training iterations. We then propose ANCE-Tele, which accumulates momentum negatives from past iterations and approximates future iterations using lookahead negatives, as “teleportations” along the time axis to smooth the learning process. On web search and OpenQA, ANCE-Tele outperforms previous state-of-the-art systems of similar size, eliminates the dependency on sparse retrieval negatives, and is competitive among systems using significantly more (50x) parameters. Our analysis demonstrates that teleportation negatives reduce catastrophic forgetting and improve convergence speed for dense retrieval training. The source code of this paper is available at https://github.com/OpenMatch/ANCE-Tele.

3. ReSel: N-ary Relation Extraction from Scientific Text and Tables by Learning to Retrieve and Select

   Yuchen Zhuang, Yinghao Li, Junyang Zhang, Yue Yu, Yingjun Mou, Xiang Chen, Le Song, and Chao Zhang

   Proceedings of EMNLP, 2022. (Oral)

   Abs PDF

   We study the problem of extracting N-ary relation tuples from scientific articles. This task is challenging because the target knowledge tuples can reside in multiple parts and modalities of the document. Our proposed method ReSel decomposes this task into a two-stage procedure that first retrieves the most relevant paragraph/table and then selects the target entity from the retrieved component. For the high-level retrieval stage, ReSel designs a simple and effective feature set, which captures multi-level lexical and semantic similarities between the query and components. For the low-level selection stage, ReSel designs a cross-modal entity correlation graph along with a multi-view architecture, which models both semantic and document-structural relations between entities. Our experiments on three scientific information extraction datasets show that ReSel outperforms state-of-the-art baselines significantly.

4. AcTune: Uncertainty-Based Active Self-Training for Active Fine-Tuning of Pretrained Language Models

   Yue Yu, Lingkai Kong, Jieyu Zhang, Rongzhi Zhang, and Chao Zhang

   Proceedings of NAACL, 2022. (Oral)

   Abs PDF Code

   Although fine-tuning pre-trained language models (PLMs) renders strong performance in many NLP tasks, it relies on excessive labeled data. Recently, researchers have resorted to active fine-tuning for enhancing the label efficiency of PLM fine-tuning, but existing methods of this type usually ignore the potential of unlabeled data. We develop AcTune, a new framework that improves the label efficiency of active PLM fine-tuning by unleashing the power of unlabeled data via self-training. AcTune switches between data annotation and model self-training based on uncertainty: the unlabeled samples of high-uncertainty are selected for annotation, while the ones from low-uncertainty regions are used for model self-training. Additionally, we design (1) a region-aware sampling strategy to avoid redundant samples when querying annotations and (2) a momentum-based memory bank to dynamically aggregate the model’s pseudo labels to suppress label noise in self-training. Experiments on 6 text classification datasets show that AcTune outperforms the strongest active learning and self-training baselines and improves the label efficiency of PLM fine-tuning by 56.2% on average. Our implementation is available at \urlhttps://github.com/yueyu1030/actune.

5. Self-Training with Differentiable Teacher

   Simiao Zuo*, Yue Yu*, Chen Liang, Haoming Jiang, Siawpeng Er, Chao Zhang, Tuo Zhao, and Hongyuan Zha

   Proceedings of NAACL Findings, 2022.

   Abs PDF

   Self-training achieves enormous success in various semi-supervised and weakly-supervised learning tasks. The method can be interpreted as a teacher-student framework, where the teacher generates pseudo-labels, and the student makes predictions. The two models are updated alternatingly. However, such a straightforward alternating update rule leads to training instability. This is because a small change in the teacher may result in a significant change in the student. To address this issue, we propose DRIFT, short for differentiable self-training, that treats teacher-student as a Stackelberg game. In this game, a leader is always in a more advantageous position than a follower. In self-training, the student contributes to the prediction performance, and the teacher controls the training process by generating pseudo-labels. Therefore, we treat the student as the leader and the teacher as the follower. The leader procures its advantage by acknowledging the follower’s strategy, which involves differentiable pseudo-labels and differentiable sample weights. Consequently, the leader-follower interaction can be effectively captured via Stackelberg gradient, obtained by differentiating the follower’s strategy. Experimental results on semi- and weakly-supervised classification and named entity recognition tasks show that our model outperforms existing approaches by large margins.

6. Prompt-Based Rule Discovery and Boosting for Interactive Weakly-Supervised Learning

   Rongzhi Zhang, Yue Yu, Pranav Shetty, Le Song, and Chao Zhang

   Proceedings of ACL, 2022.

   Abs PDF

   Weakly-supervised learning (WSL) has shown promising results in addressing label scarcity on many NLP tasks, but manually designing a comprehensive, high-quality labeling rule set is tedious and difficult. We study interactive weakly-supervised learning—the problem of iteratively and automatically discovering novel labeling rules from data to improve the WSL model. Our proposed model, named PRBoost, achieves this goal via iterative prompt-based rule discovery and model boosting. It uses boosting to identify large-error instances and discovers candidate rules from them by prompting pre-trained LMs with rule templates. The candidate rules are judged by human experts, and the accepted rules are used to generate complementary weak labels and strengthen the current model. Experiments on four tasks show PRBoost outperforms state-of-the-art WSL baselines up to 7.1%, and bridges the gaps with fully supervised models.

7. Counterfactual and Factual Reasoning over Hypergraphs for Interpretable Clinical Predictions on EHR

   Ran Xu, Yue Yu, Chao Zhang, Mohammed K Ali, Joyce C Ho, and Carl Yang

   Proceedings of ML4H, 2022. (Best Paper Award)

   Abs PDF Code

   Electronic Health Record modeling is crucial for digital medicine. However, existing models ignore higher-order interactions among medical codes and their causal relations towards downstream clinical predictions. To address such limitations, we propose a novel framework CACHE, to provide effective and insightful clinical predictions based on hypergraph representation learning and counterfactual and factual reasoning techniques. Experiments on two real EHR datasets show the superior performance of CACHE. Case studies with a domain expert illustrate a preferred capability of CACHE in generating clinically meaningful interpretations towards the correct predictions.

8. A survey on programmatic weak supervision

   Jieyu Zhang*, Cheng-Yu Hsieh*, Yue Yu*, Chao Zhang, and Alexander Ratner

   arXiv preprint arXiv:2202.05433, 2022.

   arXiv

2021

1. Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach

   Yue Yu*, Simiao Zuo*, Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang

   Proceedings of NAACL, 2021. (Oral)

   arXiv PDF Code
2. SumGNN: multi-typed drug interaction prediction via efficient knowledge graph summarization

   Yue Yu*, Kexin Huang*, Chao Zhang, Lucas M Glass, Jimeng Sun, and Cao Xiao

   Bioinformatics, 2021.

   arXiv PDF Code
3. WRENCH: A Comprehensive Benchmark for Weak Supervision

   Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner

   Proceedings of NeurIPS (D&B Track), 2021. (Oral)

   arXiv PDF Code

2020

1. STEAM: Self-supervised taxonomy expansion with mini-paths

   Yue Yu, Yinghao Li, Jiaming Shen, Hao Feng, Jimeng Sun, and Chao Zhang

   Proceedings of KDD, 2020. (Oral)

   arXiv PDF Code
2. BOND: BERT-assisted open-domain named entity recognition with distant supervision

   Chen Liang*, Yue Yu*, Haoming Jiang*, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang

   Proceedings of KDD, 2020. (Oral)

   arXiv PDF Code
3. SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup

   Rongzhi Zhang, Yue Yu, and Chao Zhang

   Proceedings of EMNLP, 2020.

   Abs arXiv Code

   Active learning is an important technique for low-resource sequence labeling tasks. However, current active sequence labeling methods use the queried samples alone in each iteration, which is an inefficient way of leveraging human annotations. We propose a simple but effective data augmentation method to improve label efficiency of active sequence labeling. Our method, SeqMix, simply augments the queried samples by generating extra labeled sequences in each iteration. The key difficulty is to generate plausible sequences along with token-level labels. In SeqMix, we address this challenge by performing mixup for both sequences and token-level labels of the queried samples. Furthermore, we design a discriminator during sequence mixup, which judges whether the generated sequences are plausible or not. Our experiments on Named Entity Recognition and Event Detection tasks show that SeqMix can improve the standard active sequence labeling method by 2.27%–3.75% in terms of F_1 scores. The code and data for SeqMix can be found at https://github.com/rz-zhang/SeqMix.

4. Semantic-aware spatio-temporal app usage representation via graph convolutional network

   Yue Yu, Tong Xia, Huandong Wang, Jie Feng, and Yong Li

   Proceedings of IMWUT/UbiComp, 2020.

   PDF
5. Urban anomaly analytics: Description, detection, and prediction

   Mingyang Zhang, Tong Li, Yue Yu, Yong Li, Pan Hui, and Yu Zheng

   IEEE Transactions on Big Data, 2020.

   arXiv PDF
6. Understanding Urban Dynamics via State-Sharing Hidden Markov Model

   Tong Xia, Yong Li, Yue Yu, Fengli Xu, Qingmin Liao, and Depeng Jin

   IEEE Transactions on Knowledge and Data Engineering, 2020.

   PDF

2019

1. Understanding Urban Dynamics via State-sharing Hidden Markov Model

   Tong Xia*, Yue Yu*, Fengli Xu, Funing Sun, Diansheng Guo, Depeng Jin, and Yong Li

   Proceedings of WWW, 2019.

   PDF
2. Privacy-preserving cross-domain location recommendation

   Chen Gao, Chao Huang, Yue Yu, Huandong Wang, Yong Li, and Depeng Jin

   Proceedings of IMWUT/UbiComp, 2019.

   PDF