Farrow AI is converting an open decoder-only LLM into a text-embedding model by contrastive fine-tuning on query-passage pairs. Early experiments show that information near the start of long passages is weakly represented in the pooled vectors. The team can change the attention pattern during training and wants the change best supported by published results for turning decoders into embedding models. What should it do?
- A.
Keep the causal mask and pool from the first token instead of the last, because the first token has the earliest view of the passage
- B.
Keep the causal mask and increase the number of contrastive negatives, since the weakness comes from too few negatives rather than attention
- C.
Keep the causal mask and reverse the passage order during training so that early content sits next to the pooled final token
- D.
Remove the causal attention mask during contrastive training so every token can attend to the whole passage before pooling
Show answer
Answer: D
Removing the causal mask during contrastive training lets every token see the whole passage, which NV-Embed found consistently better than keeping it.
- A. Under a causal mask the first token sees only itself, so it has no view of the passage at all.
- B. More negatives can sharpen a contrastive objective, but they do not change the one-directional information flow that limits the representation.
- C. Reversing text would help early content only by hurting late content, and it produces vectors unlike those of normal-order inputs at inference.
- D. Correct: NV-Embed reports that removing the causal mask during contrastive training, making attention bidirectional, consistently outperformed the causal mask for all pooling types.