Technical SEO: Crawling, Indexing & Core Web Vitals

Technical SEO: The Foundation

Technical SEO ensures the infrastructure of your site supports search engine discovery and understanding. Even the best content won't rank if Google can't crawl and index it properly.

Crawling: Letting Google Find Your Pages

robots.txt

Your robots.txt file tells crawlers which parts of your site to visit or skip. A misconfigured robots.txt is one of the most common causes of ranking drops — developers accidentally block production pages.

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://yourdomain.com/sitemap.xml

Never disallow your CSS or JavaScript files — Google needs to render them to understand your pages.

XML Sitemaps

A sitemap tells Google which pages exist and when they were last updated. For Next.js, use next-sitemap:

npm install next-sitemap

// next-sitemap.config.js
module.exports = {
  siteUrl: 'https://yourdomain.com',
  generateRobotsTxt: true,
  changefreq: 'weekly',
  priority: 0.7,
  sitemapSize: 7000,
}

Submit your sitemap to Google Search Console at yourdomain.com/sitemap.xml.

Indexing: Getting Pages into the Index

Meta Robots Tags

Control indexing at the page level:

<meta name="robots" content="index, follow" />   <!-- default: index this page -->
<meta name="robots" content="noindex, follow" /> <!-- don't index, but follow links -->

Use noindex for: thank-you pages, admin pages, search result pages, paginated pages beyond page 2.

Canonical Tags

Canonical tags tell Google which version of a page is the "master" — preventing duplicate content penalties:

<link rel="canonical" href="https://yourdomain.com/blog/my-post" />

In Next.js App Router:

export async function generateMetadata({ params }) {
  return {
    alternates: {
      canonical: `https://yourdomain.com/blog/${params.slug}`,
    },
  };
}

Rendering: CSR vs SSR vs SSG

Google can render JavaScript, but SSR (Server-Side Rendering) and SSG (Static Site Generation) are more reliable and faster for SEO. With Next.js App Router, pages are server-rendered by default — a significant SEO advantage.

Core Web Vitals: The User Experience Ranking Signals

Google uses three Core Web Vitals as direct ranking factors:

Fixing LCP

LCP is usually an image or large text block. Optimize it:

import Image from 'next/image';

<Image
  src="/hero-image.webp"
  alt="Hero image"
  width={1200}
  height={630}
  priority  // preloads image — use on above-the-fold images only
/>

Fixing CLS

CLS is caused by elements that shift layout after load — usually images without explicit dimensions or dynamically injected content:

<!-- Always set explicit width and height -->
<img src="photo.jpg" width="800" height="600" alt="..." />

Site Architecture

A clean URL structure helps both users and crawlers:

• Short and descriptive: /blog/technical-seo-guide (good) vs /post?id=12345 (bad)
• Use hyphens, not underscores
• Keep URLs lowercase
• Put the primary keyword in the URL

Key Takeaways

• robots.txt controls crawler access — a misconfiguration can de-index your entire site
• XML sitemaps accelerate discovery of new and updated pages — submit to Search Console
• Canonical tags prevent duplicate content penalties across similar pages
• Next.js App Router renders server-side by default — a major SEO advantage
• LCP, INP, and CLS are direct ranking factors — monitor them in Search Console

---

Try It Yourself: Set up next-sitemap in a Next.js 14 project. Configure custom priorities (0.9 for the homepage, 0.8 for tutorials, 0.7 for blog posts), generate a robots.txt that allows all crawlers, and verify the sitemap renders correctly at /sitemap.xml.