Section 3 — Lexical Analysis: How Source Code Becomes Meaningful Tokens

Before a compiler can understand your program, it must first learn how to read it.

---

A Computer Doesn't Read Code Like Humans Do

Look at this simple Python program:

name = "Alice"
age = 25

print(name)

You can probably understand it in less than a second.

Your brain immediately recognizes:

• name is a variable.
• "Alice" is a string.
• = assigns a value.
• print() outputs something.

Humans naturally recognize patterns because we've learned programming syntax.

A computer doesn't.

To the compiler or interpreter, the source file initially looks like nothing more than a long sequence of characters.

n a m e   =   " A l i c e "

At this point, the computer has no idea what name or print means.

It simply sees letters, quotation marks, spaces, and symbols.

So how does a programming language begin making sense of this text?

The answer is lexical analysis.

---

What Is Lexical Analysis?

Lexical analysis is the first stage of almost every compiler and interpreter.

Its job is surprisingly simple:

Read raw source code character by character and group those characters into meaningful pieces called tokens.

Instead of understanding complete statements or program logic, the lexer focuses on identifying the building blocks of the language.

Think of it as reading a sentence in English.

Consider this sentence:

The quick brown fox jumps over the lazy dog.

You naturally separate it into individual words.

The
quick
brown
fox
jumps
over
the
lazy
dog

The lexer performs a similar task for programming languages.

Instead of words, it produces tokens.

---

Figure 5. Lexical analysis scans raw source code and transforms it into a structured stream of tokens.

Caption: The lexer acts as the first reader of your program, converting characters into meaningful language elements.

SEO Alt Text: Diagram showing lexical analysis converting source code into programming tokens.

---

What Exactly Is a Token?

A token is the smallest meaningful unit recognized by a programming language.

Imagine building a house.

Before walls are constructed, individual bricks are manufactured.

Tokens are those bricks.

The parser—the next stage of the compiler—cannot build an understanding of your program until the lexer has produced these building blocks.

For example:

total = price * quantity

The lexer doesn't see a complete mathematical expression.

Instead, it produces something like this:

The parser will later examine these tokens to understand that this statement performs multiplication and assignment.

---

The Lexer Reads One Character at a Time

It might sound surprising, but lexers usually process source code one character at a time.

Suppose your source code contains:

if age >= 18:

The lexer begins reading:

i

Then:

if

It recognizes that if is a reserved keyword.

Next it skips whitespace.

Then it reads:

age

This becomes an identifier.

Then:

>=

Instead of treating > and = as separate characters, the lexer recognizes them together as a single comparison operator.

Finally it finds:

18

which becomes a numeric literal.

The resulting token stream looks like this:

Notice something important.

The lexer doesn't know what this statement does.

It only knows what each piece is.

Understanding the statement comes later.

---

Different Types of Tokens

Programming languages typically classify tokens into several categories.

1. Keywords

Keywords are reserved words with predefined meanings.

Examples:

if
else
while
for
return
class
import

You cannot normally use keywords as variable names.

For example:

if = 10

This isn't valid Python because if already has a special meaning.

---

2. Identifiers

Identifiers are names chosen by programmers.

Examples:

username
totalPrice
calculateTax
studentAge

The lexer doesn't know what these variables represent.

It only knows they are identifiers.

---

3. Operators

Operators perform actions.

Examples include:

+
-
*
/
%
=
==
!=
<
>
>=
<=
&&
||

Different programming languages support different operators, but the lexer recognizes them all as operator tokens.

---

4. Literals

Literals represent fixed values.

Examples:

100
3.14
"Hello"
True
False

These values become literal tokens.

---

5. Delimiters and Separators

These symbols organize code.

Examples include:

(
)
{
}
[
]
,
;
:
.

Although small, these characters are essential because they define structure.

---

Figure 6. A lexer classifies source code into different categories of tokens before parsing begins.

Caption: Every piece of source code belongs to a token category such as a keyword, identifier, operator, literal, or delimiter.

SEO Alt Text: Color-coded tokenization example showing keywords, identifiers, operators, literals, and delimiters.

---

Whitespace Usually Doesn't Matter

One interesting responsibility of the lexer is deciding which characters can be ignored.

For example:

total=price*quantity

and

total = price * quantity

produce almost identical token streams.

The spaces improve readability for humans, but in many languages they don't affect the tokens produced by the lexer.

Python is a notable exception because indentation carries meaning. In Python, the lexer generates special indentation-related tokens that later help define code blocks.

---

Comments Are Usually Removed

Comments help developers.

They don't help the CPU.

For example:

# Calculate total price
total = price * quantity

Most lexers recognize the comment and simply ignore it.

The resulting token stream contains only the executable statement.

This is one reason comments never make a program slower—they're typically discarded before parsing even begins.

---

Key Insight

The lexer doesn't understand your program's logic. It simply transforms a stream of characters into a stream of categorized tokens. This structured output becomes the foundation for every later stage of compilation or interpretation.

---

Why Lexical Analysis Matters

Without lexical analysis, every later stage of the compiler would need to interpret raw characters directly.

That would make parsing dramatically more difficult.

Instead, the lexer simplifies the job by producing clean, structured input.

Think of a factory assembly line.

Raw materials arrive first.

Before assembly can begin, workers sort those materials into organized bins.

Only then can the product be built efficiently.

The lexer performs that sorting process for programming languages.

---

The Journey Continues

After lexical analysis, your source code is no longer just plain text.

It has become a structured stream of meaningful tokens.

But the compiler still doesn't understand how those tokens fit together.

It knows that if is a keyword and age is an identifier, but it doesn't yet know whether the overall statement follows the language's grammar.

That responsibility belongs to the next stage: parsing.

The parser takes the token stream produced by the lexer and assembles it into a tree-like structure called an Abstract Syntax Tree (AST), allowing the compiler to understand the relationships between variables, operators, expressions, and statements.

In the next section, we'll explore how parsing transforms individual tokens into a complete structural representation of your program—the foundation for semantic analysis, optimization, and code generation.