Semantic Keyword Clustering with Python: Using Embeddings to Cluster by Meaning

By RankTree Team · 2026-07-09

Most Python keyword clustering tutorials focus on code, not results. You run a script, generate clusters, and assume the job is done.

But when those clusters don’t match real search intent, your entire SEO strategy breaks leading to poor rankings, keyword cannibalization, and wasted content effort.

The problem is simple: semantic similarity is not the same as search intent.

TF-IDF and stemming-based clustering commonly used in Python SEO tutorials group keywords by shared words and frequency not meaning. As a result semantically identical phrases can land in separate clusters simply because they don’t share common terms.

For a search engine that has relied on semantic understanding since 2013 building content architecture on word-matching logic is a structural flaw that compounds with every page you publish.

This guide focuses on the solution: embedding-based semantic clustering. It assumes you already understand basic Python keyword clustering. If not, start with your “Keyword Clustering in Python” guide first.

From here we focus on how transformer-based embeddings solve the limitations of TF-IDF and enable more accurate intent-driven clustering.

Why Word Embeddings Outperform TF IDF for SEO Clustering?

The clustering method you choose directly impacts the quality of your keyword groups so it is important to understand how each approach works.

TF-IDF groups keywords based on shared words. For example “best SEO tools” and “top SEO software” cluster together because they share the word “SEO.”

However “best keyword clustering software” and “top tools for grouping keywords” may end up in separate clusters despite having the same intent simply because they do not share common terms.

Word embeddings solve this by grouping keywords based on meaning rather than vocabulary. Transformer models understand that “best” and “top” are similar and that “clustering” and “grouping” represent the same concept. As a result semantically similar phrases are correctly grouped together.

This matters for SEO because Google has used semantic understanding since updates like Hummingbird RankBrain and BERT.

Relying on TF-IDF today means applying outdated logic to a search engine that now prioritizes meaning over exact word matches.

Semantic Keyword Clustering vs SERP Clustering in Python

Semantic keyword clustering and SERP-based clustering solve different problems, and understanding the difference is critical for SEO accuracy.

Semantic clustering uses embeddings to group keywords based on linguistic meaning. It identifies relationships between words even when there is no direct keyword overlap.

SERP clustering, on the other hand, groups keywords based on actual Google search results. If two keywords return similar URLs, they are considered the same intent.

The key difference is this: semantic clustering understands language, while SERP clustering understands Google.

This distinction explains why embedding based clustering alone often fails in SEO workflows.

Library Selection: Sentence Transformers over spaCy for This Use Case

There are two common ways to generate semantic embeddings in Python but they are not equally effective for keyword clustering.

spaCy’s word vectors (e.g. en_core_web_md) create embeddings at the word level and then average them for multi-word keywords. This often loses context, for example “not expensive” and “expensive” can end up with similar vectors because the dominant word is the same.

Sentence Transformers generate sentence level embeddings using transformer models processing each keyword as a complete unit. 

This preserves context allowing the model to distinguish between queries like “best free keyword clustering tool” and “paid keyword clustering software” which have different intent.

For SEO clustering Sentence Transformers are more accurate especially for long-tail and comparison queries. The all-MiniLM-L6-v2 model is a strong starting point fast, reliable for short text and efficient on CPU without requiring a GPU.

Building the Semantic Clustering Pipeline

Stage 1  Load and Normalize Keywords

import pandas as pd

df = pd.read_csv("keywords.csv")

keywords = df["keyword"].dropna().str.lower().str.strip().tolist()

print(f"Processing {len(keywords)} keywords")

Lowercase normalization matters here. Transformer models are case sensitive  "Running Shoes" and "running shoes" can produce slightly different embeddings. Lowercasing everything before encoding ensures consistent vector outputs across your list.

Stage 2  Generate Embeddings

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all MiniLM L6 v2')

# batch_size controls memory usage  lower for large lists on CPU

embeddings = model.encode(

    keywords

    batch_size=64

    show_progress_bar=True

    convert_to_numpy=True

)

print(f"Embedding shape: {embeddings.shape}")

# Output: (number_of_keywords 384)

Each row in the embeddings array is the semantic fingerprint of one keyword. The 384 columns represent the 384 dimensions of the embedding space. Keywords about the same topic cluster together in this space regardless of the specific words used.

Stage 3  Cluster with AgglomerativeClustering

AgglomerativeClustering is the right algorithm for this use case for one specific reason: it does not require you to specify the number of clusters in advance. 

It discovers clusters based on the actual semantic distance between keywords controlled entirely by the distance threshold parameter.

from sklearn.cluster import AgglomerativeClustering

from sklearn.preprocessing import normalize

# Normalize embeddings for cosine distance

normalized = normalize(embeddings)

clustering = AgglomerativeClustering(

    n_clusters=None

    distance_threshold=0.35

    metric='cosine'

    linkage='average'

)

labels = clustering.fit_predict(normalized)

print(f"Total clusters found: {len(set(labels))}")

What distance threshold actually controls:

This is the most important parameter in the entire script. It defines how semantically close two keywords must be to merge into the same cluster.

• 0.25  Very tight clusters. Only near identical phrasing ends up together. Good for detecting duplicate intent pages on an existing site.

• 0.35  Balanced. True synonyms and closely related variants group together. Recommended starting point.

• 0.50  Broad clusters. Related subtopics sometimes merge. Useful for building high level topical maps before content planning.

Run at 0.35 first review 10–15 clusters manually then adjust based on what you see.

Stage 4  Assign Cluster Names and Export

import numpy as np

results = pd.DataFrame({

    'keyword': keywords

    'cluster_id': labels

})

# Name each cluster after its shortest keyword (most concise intent signal)

cluster_names = (

    results.groupby('cluster_id')['keyword']

    .apply(lambda x: min(x key=len))

    .rename('cluster_name')

)

results = results.merge(cluster_names on='cluster_id')

results = results.sort_values(['cluster_id' 'keyword']).reset_index(drop=True)

results.to_csv("semantic_clusters_output.csv" index=False)

# Summary

summary = results.groupby('cluster_name').size().sort_values(ascending=False)

print("Top 10 clusters by keyword count:")

print(summary.head(10))

The output file maps every keyword to a cluster ID and cluster name  ready for content hierarchy mapping.

Reading and Acting on the Output

The CSV output is not a content plan it’s raw input for one. Here’s how to turn it into decisions:

Large clusters (15+ keywords):
These often contain mixed intent. Review the keywords and split the cluster if informational and commercial queries are combined.

Medium clusters (2–8 keywords):
Usually tight single intent groups. These are ideal for focused cluster articles targeting one clear user goal.

Single-keyword clusters:
These may represent unique intent navigational queries or threshold issues. Review before removing.

Mapping to content structure:
Use search volume tiers to organize clusters: high-volume clusters become sub-pillar pages, mid-volume clusters become articles and the broadest cluster connects to the main pillar page. For full architecture see your .

Validating Semantic Clusters Against SERP Data

Embedding similarity and search intent are related but not identical. Transformer models group keywords by meaning while Google groups them by actual user intent. These signals often diverge especially for commercial and comparison queries.

The most reliable way to validate clusters at scale is using Jaccard similarity on SERP URLs. For each cluster compare the top 10 results for 2–3 representative keywords:

• 50%+ overlap → same intent → one page can target the cluster

• 30–49% overlap → partially aligned → requires manual review

• Below 30% → different intent → split into separate pages

This method catches errors embeddings miss. For example “keyword clustering software” and “free keyword clustering tool” are semantically similar but produce different SERPs due to different intent. Embeddings may group them but SERP overlap correctly separates them.

RankTree automates this by analyzing SERP overlap across all clusters turning a time-consuming manual task into a scalable validation step before finalizing your content plan.

The Hybrid Clustering Workflow for SEO

Relying on semantic clustering alone is a common mistake in SEO. While embedding group keywords efficiently, they do not fully capture how Google interprets intent.

The most effective approach is a hybrid workflow that combines semantic clustering for scale and SERP clustering for validation.

Step one is semantic pre-clustering. This reduces a large keyword list into manageable topic groups using embeddings.

Step two is SERP validation. Representative keywords from each cluster are analyzed using SERP overlap to confirm whether they belong on the same page.

Step three is content mapping. Validated clusters are assigned to pages based on intent, ensuring each page targets a distinct search goal.

This workflow allows you to scale clustering efficiently while maintaining SEO accuracy, preventing cannibalization and improving ranking potential.

What This Approach Cannot Do?

Semantic clustering with Sentence Transformers is powerful but not universal. Being clear about its limitations prevents the most common production errors.

Cannot detect SERP cannibalization: It has no awareness of your existing site. Different clusters may still map to pages you already rank for. Always cross check clusters against your URL structure.

Does not classify intent: Semantic similarity ≠ search intent. It won’t tell you if a query needs a blog post product page or landing page. Intent must be handled separately via manual review or SERP analysis.

Quality depends on model choice:

• all MiniLM L6 v2 works well for English SEO.

• paraphrase multilingual MiniLM L12 v2 is better for multilingual data.

• Domain specific models can outperform both in specialized niches.

Understanding keyword types especially informational vs commercial intent makes review more efficient. Instead of analyzing every cluster from scratch you focus only on detecting mixed intent within each group. 

Final Thoughts

Most SEO strategies plateau not because content is weak but because the underlying keyword architecture is flawed: overlapping intent keyword based clustering instead of meaning and slow burn cannibalization that only appears as ranking drops later.

Embedding based semantic clustering addresses this at the source by aligning clusters with how search engines interpret meaning rather than surface keyword similarity. This gives every page built on top a structural advantage that compounds over time.

The workflow typically runs in under 10 minutes on a standard laptop for most keyword sets with another 20 minutes for distance threshold tuning. A manual review step handles the edge cases algorithms miss.

FAQs

How is this different from regular keyword clustering in Python?

Traditional Python clustering (e.g. TF IDF stemming) groups keywords by shared terms or patterns. Semantic clustering uses transformer embeddings to group by meaning allowing keywords with no word overlap but identical intent to cluster together. This reduces cannibalization and improves content coverage per page.

Does SentenceTransformers require a GPU?

No, Models like all MiniLM L6 v2 run efficiently on CPU for up to ~10000 keywords. For larger datasets using a free GPU in Google Colab speeds up processing but CPU is sufficient for most SEO use cases.

What is the best distance threshold for SEO keyword clustering? 

A good starting point is 0.35. If your clusters feel too broad and mix multiple intents you can lower it to around 0.25. If too many keywords remain unclustered, increasing it to about 0.45 helps. The ideal threshold varies depending on your niche and how diverse your keyword set is.

Can this be used for non English keywords?

Yes. Swap in paraphrase multilingual MiniLM L12 v2 which supports 50+ languages. The rest of the pipeline remains the same.