Relation is an option for processing context information

Abstract

Attention mechanisms are one of the most frequently used architectures in the development of artificial intelligence because they can process contextual information efficiently. Various artificial intelligence mechanisms such as transformer for processing natural language, image data, etc. include the attention mechanism. Since attention is a powerful component to realize artificial intelligence, various improvements have been made to enhance its performance. The time complexity of attention depends on the square of input sequence length. Methods to improve time complexity of attention are one of the hottest topics in various studies. In this study, we have devised a mechanism called relation that can understand the context information of sequential data without using attention. Relation is very simple to implement and its time complexity depends only on the length of the sequences; a comparison of the performance of neural networks with relation and attention mechanisms on several benchmark datasets shows that relation achieved the same context processing capability as attention with less computation time. Hence, relation is an ideal option for processing context information.