Article ; Online: Argument mining as rapid screening tool of COVID-19 literature quality: Preliminary evidence.
2022 Volume 10, Page(s) 945181
Abstract: Background: The COVID-19 pandemic prompted the scientific community to share timely evidence, also in the form of pre-printed papers, not peer reviewed yet.: Purpose: To develop an artificial intelligence system for the analysis of the scientific ... ...
Abstract | Background: The COVID-19 pandemic prompted the scientific community to share timely evidence, also in the form of pre-printed papers, not peer reviewed yet. Purpose: To develop an artificial intelligence system for the analysis of the scientific literature by leveraging on recent developments in the field of Argument Mining. Methodology: Scientific quality criteria were borrowed from two selected Cochrane systematic reviews. Four independent reviewers gave a blind evaluation on a 1-5 scale to 40 papers for each review. These scores were matched with the automatic analysis performed by an AM system named MARGOT, which detected claims and supporting evidence for the cited papers. Outcomes were evaluated with inter-rater indices (Cohen's Kappa, Krippendorff's Alpha, s Results: MARGOT performs differently on the two selected Cochrane reviews: the inter-rater indices show a fair-to-moderate agreement of the most relevant MARGOT metrics both with Cochrane and the skilled interval scores, with larger values for one of the two reviews. Discussion and conclusions: The noted discrepancy could rely on a limitation of the MARGOT system that can be improved; yet, the level of agreement between human reviewers also suggests a different complexity between the two reviews in debating controversial arguments. These preliminary results encourage to expand and deepen the investigation to other topics and a larger number of highly specialized reviewers, to reduce uncertainty in the evaluation process, thus supporting the retraining of AM systems. |
---|---|
MeSH term(s) | Artificial Intelligence ; COVID-19/diagnosis ; COVID-19/epidemiology ; Humans ; Pandemics ; Reproducibility of Results ; Research |
Language | English |
Publishing date | 2022-07-18 |
Publishing country | Switzerland |
Document type | Journal Article |
ZDB-ID | 2711781-9 |
ISSN | 2296-2565 ; 2296-2565 |
ISSN (online) | 2296-2565 |
ISSN | 2296-2565 |
DOI | 10.3389/fpubh.2022.945181 |
Database | MEDical Literature Analysis and Retrieval System OnLINE |
More links
Kategorien
Order via subito
This service is chargeable due to the Delivery terms set by subito. Orders including an article and supplementary material will be classified as separate orders. In these cases, fees will be demanded for each order.