Posted by Xingchen Wan, Student Researcher, and Ruoxi Sun, Research Scientist, Cloud AI Team
Recent advancements in large language models (LLMs) have shown great promise in their ability to solve general problems with just a few examples or even no training on specific tasks. This is particularly impressive in the few-shot setup, where LLMs are given only a few question-answer demonstrations before being tested. The zero-shot setup is even more challenging, as LLMs are directly given the test question without any prior examples. While the few-shot setup has reduced the amount of data needed to adapt a model for specific tasks, it can still be difficult to generate sample prompts. For tasks like summarizing long articles or answering medical questions, it can be challenging to come up with sample answers. In such cases, models with high zero-shot performance can be useful since they don’t require manual prompt generation. However, zero-shot performance is typically weaker as LLMs lack guidance and may produce incorrect outputs.
In our paper “Better Zero-shot Reasoning with Self-Adaptive Prompting” at ACL 2023, we introduce Consistency-Based Self-Adaptive Prompting (COSP) to address this issue. COSP is an automatic zero-shot prompting method that selects and constructs pseudo-demonstrations for LLMs using only unlabeled samples and the model’s own predictions. With COSP, we bridge the performance gap between zero-shot and few-shot setups while maintaining the generality of zero-shot prompting. In our subsequent paper “Universal Self-Adaptive Prompting” (USP) accepted at EMNLP 2023, we extend this idea to a wide range of natural language understanding (NLU) and natural language generation (NLG) tasks and demonstrate its effectiveness.
Prompting LLMs with their own outputs is based on the understanding that LLMs can benefit from demonstrations and have some zero-shot capabilities. However, we need to be cautious as zero-shot solutions may be imperfect and could potentially mislead LLMs. We conducted experiments to show that adding correct demonstrations leads to correct solutions, while adding incorrect demonstrations results in incorrect answers. Therefore, we need to carefully select reliable self-generated demonstrations.
COSP leverages the observation that confident and consistent predictions are more likely to be correct. We propose using the model’s confidence in its output as a proxy for correctness. By considering high-confidence outputs and their inputs as pseudo-demonstrations, we can select robust self-generated demonstrations. We use zero-shot chain-of-thought (CoT) prompting to generate a range of possible rationales and answers, and we measure the uncertainty of the answers through entropy. Answers that have high self-consistency and certainty are more likely to be correct and are selected as pseudo-demonstrations. We use a scoring function that combines consistency, lack of repetition, and diversity to select the best self-generated demonstrations.
COSP focuses on question-answering tasks with CoT prompting, where self-consistency can be easily measured. However, it may be challenging for other tasks like open-ended question-answering or generative tasks. To address this, we introduce USP, which generalizes our approach to other NLP tasks. For classification tasks, we compute the entropy of the logit distribution to measure uncertainty. For short-form generation tasks, we use the same procedure as COSP. For long-form generation tasks, we use an overlap metric based on the average pairwise ROUGE score between different outputs to the same query.
In our experiments, COSP significantly outperforms the standard zero-shot baseline, and USP improves zero-shot performance across a wide range of tasks. We compare against baselines using self-consistency and show that USP is competitive with prompting using golden examples. Our results demonstrate the effectiveness of COSP and USP in improving zero-shot reasoning and general NLP tasks.
Note: The content has been rewritten while keeping the HTML tags intact.
Source link
