Towards a Computational Framework for Distinguishing Critical and Conspiratorial Texts by Elaborating on the Context and Argumentation with LLMs

Abstract

The shared task of PAN 2024 addresses the need to distinguish between critical and conspiratorial texts in relation to public health measures during the COVID-19 pandemic. In early 2020, the pandemic caused a simultaneous rise in misinformation and conspiracy theories, leading to an ’infodemic’ that increased societal insecurity. This notebook introduces an experimental computational framework leveraging Large Language Models (LLMs) for contextual and argumentative elaborations to enhance the classification accuracy of a reference DeBERTa base classification model. Our approach involves automatic annotations of intent and argumentation style, hypothesizing that these features aid in differentiating between conspiracy and critical texts. Experimental results, however, reveal that DeBERTa performs best without these elaborations, achieving an MCC of 0.838 and F1-macro of 0.917. The inclusion of LLM-generated feature annotations did not surpass the baseline performance. These findings suggest that while theoretically valuable, the practical application of such elaborations requires further refinement. Future work should focus on optimizing LLM outputs and exploring alternative techniques to enhance text classification without overloading models with excessive information.