Advanced NLP: BERT, T5 & LLaMA

By the end you'll explain how transformers read context, tell BERT (understanding) apart from GPT (generation), fine-tune a pretrained model, and run NER, question answering, and summarization with a single line of Hugging Face code.

Learn Advanced NLP: BERT, T5 & LLaMA in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

🌍 Real-World Analogy: Understanding Meaning, Not Just Words

Imagine two readers handed the sentence "I went to the bank." A dictionary-only reader sees the word "bank" and shrugs — it could be money or a river. A thoughtful reader looks at the words around it ("I fished off the …") and instantly knows which one you mean.

That thoughtful reader is a transformer. It doesn't store one fixed meaning per word — it builds a contextual embedding , a fresh vector for each word shaped by its neighbours. That is the whole reason modern NLP understands meaning , not just spelling. BERT is the detective reading the full report in both directions to understand; GPT is the storyteller writing one word at a time; T5 is the translator who reads everything, then writes a fresh version.

1 Contextual Embeddings — Words With a Meaning That Moves

An embedding is a list of numbers (a vector) that represents a piece of text so a computer can do maths with meaning. The breakthrough of transformers is that the embedding is contextual : the same word gets a different vector depending on the sentence.

You compare two vectors with cosine similarity — a score from 0 (unrelated) to 1 (same direction). This single tool powers semantic search : find the document whose vector points the same way as your question, even if it shares no exact words. Run the worked example below and read the comments — every result is stated inline.

2 BERT vs GPT — Encoder vs Decoder

BERT is an encoder . It is pretrained with masked language modeling (MLM) : random words are hidden behind a [MASK] token and BERT must guess them using context from both sides. Because it reads in both directions, it is brilliant at understanding — classification, NER, similarity, and question answering.

GPT is a decoder . It is pretrained to predict the next token using only the words to its left. That left-to-right view makes it a natural generator — chat, writing, code. T5 combines both: an encoder reads the input, a decoder writes the output, which is the perfect shape for summarization and translation.

3 Transfer Learning & Fine-Tuning

You almost never train an NLP model from zero. Instead you use transfer learning : take a model already pretrained on billions of words, then fine-tune it on your much smaller, task-specific dataset. The model already "knows" language; you only teach it your specific job.

The recipe for fine-tuning BERT on a task like NER or sentiment is short:

With a few thousand labelled examples you can reach accuracy that would have needed millions of examples to train from scratch. That is the power — and the economy — of transfer learning.

4 NER, Question Answering & Summarization in One Line

Three of the most common NLP jobs each have a ready-made pretrained model:

• Named Entity Recognition (NER) — tag each word as a person, organisation, or location.
• Question Answering (QA) — given a context and a question, return the exact answer span.
• Summarization — condense a long passage into a short one.

Hugging Face's pipeline() downloads and wires up the right model for you. Study the worked example below — each call is followed by its # Expected output .

🎯 Your Turn 1: Build a Rule-Based NER Tagger

Real NER uses BERT, but the idea is simple enough to do by hand: look each word up in a dictionary of known entities and label it. Fill in the one blank, then run it and check your output against the comment.

🎯 Your Turn 2: Semantic Search With Cosine Similarity

This is how a search box "understands" a question: turn the query and each document into vectors, then rank documents by cosine similarity. Complete the cosine formula and confirm the password document scores higher than the weather one.

🎯 Mini-Challenge: Keyword Sentiment Tagger

Time to fade the scaffolding. Only a comment outline is provided — you write the logic. Build a tiny lexicon-based sentiment tagger that decides whether a review is positive, negative, or mixed.

5 Common Errors (And How to Fix Them)

Treating a word as one fixed meaning (old word2vec thinking). "Apple" the company and "apple" the fruit are different — a model that ignores context will confuse them.

✅ Fix: Use a transformer model whose embeddings are contextual; never average away the sentence.

Using GPT for plain classification or NER. It works, but it is 10–100× slower and pricier than a fine-tuned BERT, and often less accurate on the label.

✅ Fix: Encoder (BERT) for understanding, decoder (GPT) for generation, encoder-decoder (T5) for rewriting. Use the simplest model that solves the task.

BERT caps at 512 tokens. Feed a long document straight in and it is silently truncated — the model never sees the end, so your answer or summary is wrong.

✅ Fix: Chunk long text into overlapping windows, or use a long-context model designed for it.

Fine-tuning on noisy, mislabelled, or imbalanced data. The model faithfully learns your mistakes — garbage in, garbage out — and scores look fine on a flawed test set.

✅ Fix: Clean and balance labels, hold out a trustworthy validation set, and prefer fewer correct examples over many noisy ones.

📋 Quick Reference: BERT vs GPT

Feature

BERT (Encoder)

GPT (Decoder)

Attention

Bidirectional (both sides)

Left-to-right only

Pretraining

Masked LM (fill the blank)

Next-token prediction

Best at

Understanding

Generation

Typical tasks

Classification, NER, QA, similarity

Chat, writing, code, reasoning

Sees

All tokens at once

Past tokens only

Example

bert-base-uncased

GPT-4, LLaMA, Mistral

Need to rewrite text (summarize/translate)? Use an encoder-decoder like T5 or BART — it has both.

❓ Frequently Asked Questions

Lesson 35 complete — you understand the NLP model landscape!

You can explain contextual embeddings, distinguish BERT's encoder from GPT's decoder, describe transfer learning and fine-tuning, and run NER, question answering, and summarization with Hugging Face's pipeline() . You also built a rule-based NER tagger and a cosine-similarity search by hand.

🚀 Up next: RAG Systems — combine these LLMs with a knowledge base so they answer from your documents, not just their training data.

Practice quiz

Continue this course

• Previous: Speech Recognition
• Next: RAG Systems

Related lessons

• Transformers & Attention