Theoretical Analysis of Positional Encodings in Transformer Models

 (arxiv.org)

34 points | by PaulHoule 16 hours ago

1 comments

  • semiinfinitely
    12 hours ago
    Kinda disappointing that rope- the most common pe- is given about one sentence in this work and omitted from the analysis.
    [-]
    • gsf_emergency_2
      9 hours ago
      Maybe it's because ropes by themselves do nothing for the model capacity?