SIGNAL: Dataset for Semantic and Inferred Grammar Neurological Analysis of Language

Recently, the idea of comparison of models representations and human brain signals has been a topic of several works. Consequently, several datasets with text data and EEG representations have been published. However, most of the datasets are based on normal reading task with grammatical sentences. At the same time, in the interpretability studies of LLMs, more and more attention is paid to thoroughly designed linguistic tasks based on acceptability measures. In this paper, we present SIGNAL, a dataset for Semantic and Inferred Grammar Neurological Analysis of Language. Our dataset contains a group of sentences with a combination of a fully acceptable sentence and a grammatically or/and semantically incongruent sentences. The dataset has been approved by native speakers and later used for an EEG experiment. In total, our dataset contains recordings of 21 participants, each of whom read 600 sentences. In addition, we present a pilot study where we compare EEG analysis with simple probing experiments.