1. causal attention을 bidirectional attention으로 전환하고, unsupervised manner로 masked next token prediction을 학습하여 llm을 적응시킴.

2. SimCSE 방법을 이용하여, unsupervised contrastive learning을 수행함.

We demonstrate the effectiveness of LLM2Vec by applying it to 3 popular LLMs ranging from 1.3B to 7B parameters and evaluate the transformed models on English word- and sequence-level tasks. We outperform encoder-only models by a large margin on word-level tasks and reach a new unsupervised state-of-the-art performance on the Massive Text Embeddings Benchmark (MTEB). Moreover, when combining LLM2Vec with supervised contrastive learning, we achieve state-of-the art performance on MTEB among models that train only on publicly available data.

• encoder model보다 pre-training 과정에서 일반적으로 sample efficient함.

• decoder-only LLM에 관련된 연구 혹은 생태계가 상대적으로 잘 구축되어있고, 좋은 pre-training recipes (e.g. staged pre-training)이 많이 연구됨.

• decoder-only LLM의 instruction following 연구들에 힘입어, instruction을 사용해서 universal text embedding model을 만들려는 시도들이 있음.

As shown in Figure 1, LLM2Vec consists of three simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsupervised contrastive learning. Crucially, LLM2Vec does not require any labeled data and is highly data- and parameter efficient.

Overall, our work demonstrates that decoder-only LLMs are indeed capable of producing universal text embeddings and only very little adaptation is required to reveal this ability.

Crucially, when predicting a masked token at position i, we compute the loss based on the logits obtained from the token representation at the previous position i − 1, not the masked position itself (see Figure 1).

Sheared-LLaMA-1.3B (S-LLaMA-1.3B, Xia et al., 2023), Llama-2-7B-chat (LLaMA-2-7B, Touvron et al., 2023), and Mistral-7B-Instruct-v0.2 (Mistral-7B, Jiang et al., 2023).

Specifically, we use the Wikitext-103 dataset (Merity et al., 2017) for the MNTP step and a subset of Wikipedia sentences released by Gao et al. (2021) for the unsupervised SimCSE step.

• decoder-only LLMs의 pre-training 과정에 포함되는 데이터이므로, adaptation하는 것외에 새로운 knowledge를 가르치지 않게 할 수 있음.

We fine-tune the model using LoRA (Hu et al., 2022) to predict the masked token using the representation of the previous token to maximally align our training objective with the pre-training setup of decoder-only LLMs.

We merge the MNTP LoRA weights into the base model and initialize new LoRA parameters before starting the SimCSE training, which ensures that the model retains the knowledge learned in the previous step.

We evaluate on three word-level tasks: chunking, named-entity recognition (NER), and part-of-speech tagging (POS), using the CoNLL-2003 benchmark (Tjong Kim Sang & De Meulder, 2003).

• 이전 연구와 같은 instruction set을 사용하고, query에만 이를 붙임.
  example: instruction set

  

• mean pooling 시, instruction tokens들은 제외함.

However, compared to Echo embeddings, LLM2Vec is much more efficient as Echo embeddings repeat the input and therefore double the sequence length which makes inference considerably slower (we provide a runtime comparison in Appendix E.2).

how good models are at judging the similarity of sentences that have different meanings but share the same prefix.

• 35 sentence triples $\{(q_i,s_i^+,s_i^-)\}_{i=1}^{35}$
  • $q_i=(A_i,B_i)$, $s_i^+=(A_i,C_i)$, $s_i^-=(A_i,D_i)$
  • $B_i$와 $C_i$는 서로 비슷한 의미, $D_i$는 다름.

A model that incorporates information from future tokens $(B_i,C_i,\text{or} \ D_i)$ in the representations of the prefix Ai should assign a higher similarity to the positive example.

• $\bf{H}_l^c$: the hidden representations of every token at ever layer $l$ with causal attention

• $\bf{H}_l^{bi}$: the hidden representations of every token at ever layer $l$ with bidirectional attention

we speculate that Mistral models are pre-trained with some form bidirectional attention, e.g., prefix language modeling (Raffel et al., 2020) – at least for some parts of its training.

We follow standard practice and train the models with contrastive learning using hard negatives and in-batch negatives.

The MNTP LoRA weights are merged into the base model, and the trainable LoRA weights are initialized with SimCSE weights. For LLM2Vec models that use just MNTP, the LoRA weights are randomly initialized. The training is performed for 1000 steps with a batch size of 512.

As shown in Figure 6, LLM2Vec-transformed models achieve better performance earlier in training. This observation is consistent across all three models. For S LLaMA-1.3B, the smallest of our three models, even performing just MNTP leads to considerably improved sample efficiency. These results are particularly encouraging for settings where it is hard to acquire high quality labeled data, a setting which we leave for future work.