Logits: What Happens Before the AI Picks the Next Word
Published on
Have you ever heard of “logits”? If not, get ready for a more technical post about how LLMs work under the hood and how they do inference to pick the next word (actually, the next token) of the sentence they’re generating.
Put very simply, logits are the raw, “uncooked,” unnormalized scores produced by the final linear projection layer of an AI model’s neural network.
Going back to the beginning: when you train an LLM from scratch, you do tokenization (splitting the words in your training set into tokens, or pieces of words, like the tokens “un” and “happy” that make up the word “unhappy”). This step happens before the neural network is trained, and you train the tokenizer on the corpus to define a fixed vocabulary, which usually ranges from 50,000 to over 100,000 tokens. Let’s say 100,000 for our example.
When you send a prompt, an instruction-tuned model will try to “complete” it by giving an answer. To make things easier, let’s use the example of completing a sentence. Take the sentence: “The sky is ____”. The LLM will process this information through dozens of attention and feedforward blocks (the heart of the Transformer), and at the end, a linear projection layer spits out a vector of 100,000 logits, the exact size of our vocabulary. In other words, it assigns a raw score to every one of the 100,000 tokens it knows in its dictionary, judging which one would make the most sense next.
Hypothetical example of logits for the next token of “The sky is ____”:
["blue" = 5.7,
"beautiful" = 4.9,
"pineapple" = -3.1,
...,
"Maria" = -2.7,
"pink" = 4.7,
...]

Since logits aren’t bounded between 0 and 1, training via gradient descent can operate in a more numerically stable way (avoiding problems with extremely small values). But those raw values aren’t intuitive for us, so at the end a mathematical function called Softmax is applied, which turns those values into percentages from 0 to 100%. In our example above, it could be something like:
["blue" = 18.7%,
"beautiful" = 8.4%,
"pineapple" = 0.003%,
...,
"Maria" = 0.004%,
"pink" = 6.9%,
...]
But before they go through that Softmax function, the system can mathematically manipulate those logits to control how the model writes. The most famous parameter for doing this is Temperature. The formula is basically softmax(logits/Temperature), like this:
If you use a Low Temperature (e.g. 0.1), the logits are divided by 0.1, which is the same as multiplying them by 10. That dramatically amplifies the difference between the scores, making the AI more conservative, factual, and predictable (the “blue” logit will crush all the others).
With a high Temperature (e.g. 1.5), on the other hand, the division shrinks the differences and the scores get closer to each other (the AI starts giving less obvious words more of a chance, becoming much more creative and diverse, but with a higher risk of hallucinating as a result).
This is also the moment where famous filters like Top-K come in (where the model ignores the thousands of low logits and focuses only on the K highest values). Then, already in probability space (after the Softmax), comes Top-P, also called nucleus sampling, which selects only the tokens whose probabilities add up to a threshold (e.g. 90%). Other filters like Repetition Penalty (which penalizes tokens that were already generated to avoid repetition) can also be applied.
After all that, the AI “rolls the dice,” picks the token, adds it to the text, and starts the cycle over. This autoregressive process repeats dozens or hundreds of times per second until the answer is complete.
If you run models locally with tools like Ollama, LM Studio, llama.cpp, or text-generation-webui, you’ve probably already seen these parameters exposed in the interface or on the command line. Now you know exactly what they’re doing under the hood: manipulating the logits before (or after) the Softmax to control the model’s behavior.
Using HuggingFace’s Transformers library, you can see the logits directly:
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3-8B")
inputs = tokenizer("The sky is", return_tensors="pt")
with torch.no_grad():
output = model(**inputs)
next_token_logits = output.logits[0, -1, :]
That “next_token_logits” vector is the one that holds the score for every token in the vocabulary. In practice, “model.generate()” does this in a loop for you, automatically applying the temperature, top_k, top_p, and repetition_penalty parameters at each step before picking the next token.
And that’s pretty much it! :) Knowing how the sausage is made under the hood helps a ton when it comes to “taming” these AIs.
Happy studying!