Article ; Online: Augmenting interpretable models with large language models during training.
2023 Volume 14, Issue 1, Page(s) 7913
Abstract: Recent large language models (LLMs), such as ChatGPT, have demonstrated remarkable prediction performance for a growing array of tasks. However, their proliferation into high-stakes domains and compute-limited settings has created a burgeoning need for ... ...
| Abstract | Recent large language models (LLMs), such as ChatGPT, have demonstrated remarkable prediction performance for a growing array of tasks. However, their proliferation into high-stakes domains and compute-limited settings has created a burgeoning need for interpretability and efficiency. We address this need by proposing Aug-imodels, a framework for leveraging the knowledge learned by LLMs to build extremely efficient and interpretable prediction models. Aug-imodels use LLMs during fitting but not during inference, allowing complete transparency and often a speed/memory improvement of greater than 1000x for inference compared to LLMs. We explore two instantiations of Aug-imodels in natural-language processing: Aug-Linear, which augments a linear model with decoupled embeddings from an LLM and Aug-Tree, which augments a decision tree with LLM feature expansions. Across a variety of text-classification datasets, both outperform their non-augmented, interpretable counterparts. Aug-Linear can even outperform much larger models, e.g. a 6-billion parameter GPT-J model, despite having 10,000x fewer parameters and being fully transparent. We further explore Aug-imodels in a natural-language fMRI study, where they generate interesting interpretations from scientific data. |
|---|---|
| MeSH term(s) | Learning ; Knowledge ; Language ; Linear Models ; Natural Language Processing |
| Language | English |
| Publishing date | 2023-11-30 |
| Publishing country | England |
| Document type | Journal Article |
| ZDB-ID | 2553671-0 |
| ISSN | 2041-1723 ; 2041-1723 |
| ISSN (online) | 2041-1723 |
| ISSN | 2041-1723 |
| DOI | 10.1038/s41467-023-43713-1 |
| Database | MEDical Literature Analysis and Retrieval System OnLINE |
More links
Kategorien
Order via subito
This service is chargeable due to the Delivery terms set by subito. Orders including an article and supplementary material will be classified as separate orders. In these cases, fees will be demanded for each order.