Embeddings: Turning Objects into Coordinates | COSMO - Community of Science and Math Outreach

An embedding represents an object by a vector. A word, image, song, or molecule becomes a point such as

\mathbf{x}=(x_1,x_2,\ldots,x_d)\in\mathbb{R}^d.

The coordinates are useful only when the geometry preserves relationships that matter for the task. A good representation of words might place violin near cello. A representation built to recognize color may instead place two red objects together even when their shapes differ. There is no universally best embedding; there is only an embedding that is more or less useful for a stated purpose.

Features before learned embeddings

Suppose we describe a fruit using sweetness, acidity, and mass. An apple might be $(6,5,180)$ and a lemon $(2,9,110)$ . This is already an embedding, but the coordinates were chosen by a person. A learned embedding uses data and an objective to choose its coordinates.

Raw features need care. In the fruit example, mass ranges across hundreds while sweetness ranges from 1 to 10. Euclidean distance would be dominated by mass unless we rescale the coordinates. Standardization replaces a feature $x$ with

z=\frac{x-\mu}{\sigma},

so that each feature is measured relative to its typical variation.

What does “near” mean?

An embedding does not define similarity by itself. We also need a comparison rule. Two common choices are Euclidean distance,

d_2(\mathbf{x},\mathbf{y})=\sqrt{\sum_{i=1}^{d}(x_i-y_i)^2},

and cosine similarity,

\operatorname{cos}(\mathbf{x},\mathbf{y})= \frac{\mathbf{x}\cdot\mathbf{y}}{\lVert\mathbf{x}\rVert_2\lVert\mathbf{y}\rVert_2}.

Euclidean distance compares location. Cosine similarity compares direction. If vector length contains useful information, throwing it away with cosine similarity can hurt. If length is mostly caused by document length or signal strength, direction may be the cleaner comparison.

A small evaluation experiment

Imagine three embeddings for the same collection of short articles. We want articles about the same subject to be close.

Choose a set of query articles with known subject labels.
For each query, retrieve its $k$ nearest neighbors.
Record the fraction of those neighbors that share the query label.
Repeat for Euclidean distance and cosine similarity.
Inspect failures instead of reporting only an average.

For a query $q$ , precision at $k$ is

P@k(q)=\frac{\text{relevant items among the first }k}{k}.

If the first five results contain four relevant articles, then $P@5=0.8$ . Averaging this value across queries gives a useful summary, but it can hide weak performance on rare subjects. Break the score down by category and look at actual neighbor lists.

Classroom investigation

Give students eight familiar objects and ask each group to design a three-coordinate representation. One group might describe animals by size, speed, and habitat; another might use number of legs, diet, and activity time. Compare the neighbor lists produced by each design.

The important question is not “Which coordinates are correct?” It is “Which relationships did this representation preserve, and which did it erase?” That question connects hand-built features to modern learned embeddings without treating a model as a black box.

Common mistakes

Comparing unscaled features whose units differ greatly.
Calling two points “similar” without naming the metric.
Evaluating only a two-dimensional visualization.
Treating a high average score as proof that every subgroup works well.
Assuming that semantic closeness is objective rather than task-dependent.

An embedding is a claim about what should count as nearby. Evaluating it means testing that claim against the relationships we actually care about.