Editing Embedding (section)

== Technical Structure ==
Embeddings are numerical representations of words or phrases that capture their meaning and context. They are vector in that they can be represented by a list of numbers which assigns them a 'position' in a 'space' representing language and meaning. The standard embedding vectors for [[LLaMA]] each have 2048 numbers.  Each number in the list corresponds to a dimension in a vector space, and the position of the vector in the space reflects its semantic and syntactic properties. 

=== Simplified Example ===
For example, imagine a two-dimensional vector space (a plane) where the x-axis represents the gender of a word and the y-axis represents the number of a word. In this space, the word “he” would have a negative x-value and a zero y-value, meaning that it is masculine and singular. The word “they” would have a zero x-value and a positive y-value, meaning that it is neutral and plural. The word “she” would have a positive x-value and a zero y-value, meaning that it is feminine and singular. The word “them” would have a zero x-value and a negative y-value, meaning that it is neutral and singular or plural. 

The distance between two vectors in this space indicates how similar or dissimilar they are in terms of gender and number. The angle between two vectors indicates how related or unrelated they are in terms of gender and number. For example, the angle between “he” and “she” would be 180 degrees, meaning that they are opposite in gender. The angle between “he” and “they” would be 90 degrees, meaning that they are orthogonal or unrelated in gender.

Of course, this is a very simplified example of how embeddings work. In use, the values of the embedding for each [[token]] are learned as part of the [[base model]] [[training]] process, and it is not typically attempted to directly attribute meaning to the dimensions. These models can capture more nuanced and diverse aspects of language, such as sentiment, topic, style, tone, etc.