Web traffic forensics and website audit pipeline

Problem

The owners of a business website needed an independent, evidence-based picture of who was really visiting and of the site's overall health. Raw analytics did not separate real visitors from crawlers, datacenter traffic and the site team's own sessions, and years of accumulated users, forms, redirects and plugins had never been reviewed.

Goal

A defensible traffic breakdown and a prioritised health and security audit, produced in a reproducible way from raw logs and read-only site access, with an evidence trail behind every conclusion.

My role

Designed and built the whole pipeline: environment discovery, log collection, the analytical queries, network-ownership enrichment, traffic classification, the site audits and report generation.

Solution

A multi-stage command-line pipeline of small Python scripts that share a secret loader and thin API clients: load per-request logs into an embedded analytical database, enrich and classify the traffic, audit the site over its REST API, and render self-contained HTML reports plus an evidence file.

How it works

Discover. Inventory the site's hosting environments and domains through the host's API, and identify the edge or CDN layer from DNS.

Collect. Gather the raw per-request logs and take a read-only inventory of plugins, themes and configuration.

Load and analyse. Load hundreds of thousands of rows into DuckDB and compute daily trends and the top addresses, user agents, URLs and countries.

Enrich. Resolve the heaviest sources to their network owner through a free DNS-based lookup (no API key, cheap at volume) and reverse DNS, then classify each network by owner type (datacenter or hosting versus residential provider), because a user-agent string is self-declared and trivially faked.

Classify. An address is treated as the site team's own when a meaningful share of its requests go to endpoints that only work when signed in (admin pages, edit-mode API calls). Those addresses are excluded through a filtered view, so one stray hit never removes a real visitor, leaving likely genuine visitors separated from bots and datacenter traffic.

Audit. Authenticated, read-only checks of settings and users (including stale author accounts), forms, internal links and redirects (with a threaded crawler), and security posture (headers, TLS, exposed paths, multi-factor sign-in, plugins).

Report. A one-page executive summary, a full multi-section audit, a prioritised remediation plan and a supporting evidence file.

Technical challenges

Classifying traffic without trusting user-agent strings.

Running network-ownership lookups at volume, cheaply, over DNS.

Separating the site team's own sessions by behaviour rather than by identity.

Fast analytical queries over a large raw log.

Staying strictly read-only against a live production site.

Crawling in parallel without overloading the site.

Packaging evidence so each finding traces back to log rows.

Capabilities

Environment inventory discovery
Log collection
Traffic trends
Datacenter versus residential classification
Own-session exclusion
User, form, link, redirect and security audits
Executive summary, full audit, remediation plan and evidence pack

Technology

Impact

The site's owners got an evidence-backed breakdown of their real audience and a prioritised remediation plan, produced from raw data rather than dashboard estimates. No metrics are claimed.

Skills demonstrated

Log forensics and traffic classification
SQL for analysis
Network intelligence
API and log-collection automation
Web security and site-health auditing
Evidence-based reporting for non-technical readers