---
title: "Evidence-Based AI Website Reports: Real Scraping, No Guesswork"
description: "Multi-page website scraping with evidence tracking. Low temperature (0.2) for accuracy. No hallucinations. Real data only."
date: "2025-11-20"
draft: false
tags:
  - "Engineering"
  - "AI"
---

## The Problem: Generic AI Summaries

Most AI website analysis tools scrape your homepage and hallucinate the rest. You get
generic personas. Vague pain points. Messaging that could apply to any B2B SaaS company.

We fixed this with three changes: multi-page scraping, evidence tracking, and low
temperature.

## Fix 1: Multi-Page Scraping

Homepage content is marketing fluff. Product pages have the real details.

We now scrape 3-6 pages per site:

- Homepage
- /product or /products
- /platform
- /solutions
- /pricing (if available)
- /about (for company context)

We extract titles, meta descriptions, H1s, H2s, and paragraph content. Filter out cookie
banners and GDPR noise.

= 3) break;
}`}
/>

## Fix 2: Evidence Tracking

Every AI claim must cite the website content. No invented facts.

We added evidence fields to the prompt:


The AI must ground every insight in scraped content. If it cannot find evidence, it
returns "insufficient content" instead of guessing.

## Fix 3: Low Temperature (0.2)

Temperature controls randomness. Higher = more creative. Lower = more factual.

We dropped from 0.7 (default) to 0.2 for website analysis.


At 0.2, the model sticks to the facts. It repeats exact phrases from the website instead
of paraphrasing creatively.

## Fail Gracefully: Insufficient Content

If we cannot scrape enough content (bot protection, login walls, etc.), we do not fake
it.

The API returns an error with a clear message:


Better to fail honestly than generate garbage.

## Impact: Before and After

**Before (temperature 0.7, homepage only)**

"Target audience: Marketing managers and growth teams at B2B companies. They
struggle with campaign management and need better tools."


**After (temperature 0.2, multi-page, evidence)**

"Target audience: VPs of Engineering at Series B+ companies with 50-200
developers.

Evidence: From /platform: &apos;Purpose-built for engineering orgs with distributed
teams&apos; and from /customers: &apos;Used by Uber, Lyft, and Dropbox engineering
teams&apos;."


Specific. Grounded in real content. Actionable for media planning.

## Performance: 5-8 Seconds

Scraping 3 pages in parallel takes 2-3 seconds. GPT-4o analysis takes 3-5 seconds at
temperature 0.2.

Total: 5-8 seconds for an accurate, evidence-backed report.
