Unsupervised Contrastive Learning of Sentence Embeddings Through Optimized Sample Construction and Knowledge Distillation - Ding, Yan; Jin, Rize; Paik, Joon Young; Chung, Tae Sun (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2024)
Unsupervised contrastive learning of sentence embedding has been a recent focus of researchers. However, issues such as unreasonable division of positive and negative samples and poor data enhancement leading to text semantic changes still exist. We propose an optimized data augmentation method that combines contrastive learning’s data augmentation with unsupervised sentence pair modelling’s distillation. Our data augmentation uses in-sentence tokens for positive examples and text similarity for negative examples, while the distillation is conducted without supervised pairs. Experimental results on the STS task show that our method achieves a Spearman correlation of 81.03%, outperforming existing STS benchmarks.
Keywords: Augmentation methods; Contrastive learning; Data augmentation; Data enhancement; Embeddings; Negative samples; Positive examples; Semantic similarity; Text similarity; Unsupervised sentence embedding; Contrastive Learning; Distillation; Semantic Similarity; Unsupervised Sentence Embedding