A Googlebot name in an access log is a claim, not proof of who sent the request. To understand crawler activity, combine network verification with a record of the requested page, response status and time. This guide shows how to turn those records into a small crawl report without mistaking a tool fetch or a repeated visit for a newly indexed page.
Start with a request record you can trace
Keep the request time, hostname, path, method, response status and User-Agent. A request identifier helps connect an edge response with the application event that produced it. Record the network address used for verification in restricted logs, while public reports can retain the classification and evidence version instead of the address itself.
If a trusted proxy sits in front of the application, distinguish its socket address from the client address it forwards. Do not accept an arbitrary forwarded-address header from every connection. An incorrect proxy boundary can make a verification result describe the wrong client. Preserve the origin of the address so another operator can inspect that decision.
Verify the address before counting Googlebot
Google documents both matching client addresses against its published crawler ranges and checking reverse DNS followed by forward DNS. Choose a documented method and retain the evidence needed to repeat it. A DNS check needs to confirm that the returned hostname is in the expected Google domain and resolves back to the original address; a name that merely contains the word Google is not sufficient.
For range-based checks, save the source, retrieval time and fingerprint of the range snapshot used. Keep IPv4 and IPv6 handling explicit. When evidence is missing or too old under your application's policy, report the request as unverified rather than promoting a recognizable User-Agent into proof.
Keep Search crawls and inspection fetches separate
Network identity and the purpose of a request are different questions. A request from Google may belong to ordinary Search crawling, an inspection tool or another product. Retain both the network result and crawler family so a manual inspection does not inflate the report for a discovery experiment.
The following record is illustrative, not evidence from a live crawler. Its separate fields make the classification visible. A real report must fill them from the actual request and verification process.
{
"request_id": "example-001",
"path": "/articles/sample/",
"method": "GET",
"status": 200,
"crawler_family": "googlebot_mobile",
"network_verified": true,
"verification_snapshot": "example-snapshot"
}Report requests, distinct pages and index observations
If a verified crawler downloads one article five times, that is five requests and one distinct article URL. Separate document requests from robots files, icons, images and HEAD checks. For a cohort of twenty documents, report how many of those twenty received the declared type of verified request, then retain the total requests as a separate measure.
A successful document response establishes that the server served it. It does not establish that Google selected the document for its index or sent search traffic. Use separately timestamped index observations for that question, and name the source of any clicks or impressions. When part of a cohort has not been measured, show the coverage instead of silently treating missing observations as negative results.
A short review checklist
Before sharing a crawl report, check the time zone, observation interval, verification method and exact document cohort. Exclude your own quality checks from Search crawl counts. Keep rejected paths separate from expected pages, and preserve interruptions in logging. These details make a modest, reproducible report more useful than a large counter with an unclear definition.