Nothing in a deployment log distinguishes a page that will earn traffic from one that will sit unvisited for a year. The build succeeds either way, and the difference only becomes visible months later, in a report nobody thought to open.
Publishing places a file on a server. It does not summon anyone to read it. Between the deploy and the first visitor sits a process that nobody in your engineering channel controls, and that process has a budget.
Pages arrive faster than anyone decides they should
Most guidance about site growth assumes pages are written one at a time by a person who thought about each one. That is not how a software company's site grows. Pages here are generated: an API reference that expands with every endpoint, a changelog entry per release, an integration page per connector, a comparison page per competitor, a headless CMS pushing posts from a content calendar somebody else owns.
None of that is wrong. It is how a modern product site is supposed to work. But it means the decision to add a hundred URLs is made by a template and a merge, not by a meeting, and the site's URL count crosses thresholds nobody announced.
- Documentation multiplies with the product. Every version, locale and endpoint variant can produce its own address, and old versions rarely get retired.
- Changelogs are per-release by design. Fine as a feed, expensive as a thousand indexable pages that nobody outside the product team will ever search for.
- Integration pages scale with the partner list. Each connector gets a page, and most of them are the same page with two nouns swapped.
- Marketing pages arrive from a separate system. A headless CMS with its own publishing rules can add URLs the engineering team never sees in a diff.
Crawl budget, and the things that quietly spend it
A search engine allocates a finite amount of fetching to any given site, roughly proportional to how much it trusts the site and how much load the server can take without complaining. That allocation is not published, cannot be purchased, and is not evenly distributed across your URLs.
The important consequence is that crawling is competitive within your own domain. Every fetch spent on an address that adds nothing is a fetch not spent on the page you actually need discovered this week. On a small brochure site the effect is invisible. On a site with forty thousand generated addresses it dominates everything.
| What consumes the visit | Typical origin on a product site | Cost if left alone |
|---|---|---|
| Parameter and filter variants | Docs search, tag filters, sort orders | Endless near-identical addresses |
| Redirect chains | Renamed routes, migrated docs | Several fetches to reach one document |
| Soft errors returning 200 | Client-side routing without a real status | Wasted fetches, no signal that anything is wrong |
| Superseded versions | Every prior release of the reference | Deep archives crawled ahead of new work |
| Slow server responses | Cold functions, unwarmed caches | The allocation shrinks on its own |
| Duplicate near-identical pages | Templated comparisons and integrations | Volume without distinctiveness |
Three of those six are ordinary side effects of building a product site well. That is the trap: the engineering decisions that create the crawl problem are, in isolation, entirely correct decisions.
Comparison and integration pages, generated by the hundred
Two page types deserve naming individually, because they are produced by nearly every venture-backed product in this city and they fail in the same way.
The competitor comparison
One template, a list of rival names, a feature matrix pulled from a spreadsheet. Two hundred pages from an afternoon of work.
- The body text is identical except for a name
- Most rivals nobody is actually comparing you against
- The matrix goes stale the moment either product ships
The integration page
One page per connector, generated from the same catalog that drives the app. It scales automatically, which is exactly the problem.
- Real demand exists for perhaps a dozen of them
- The rest describe connectors almost nobody uses
- New partners add pages without any review step
Both types can work. A comparison page written by someone who genuinely knows both products, against a rival people actually evaluate you beside, is among the highest-converting pages a software company owns. The same page generated four hundred times is a crawl liability with a handful of useful entries buried in it.
The test is uncomfortable and simple: for each generated page, is there a plausible query, and would the answer be different from the answer on the other three hundred and ninety-nine? If not, the page is not content. It is a row in a database that happens to have a URL.
A feature gets a new name and the URL history does not
Software companies rename things. A module becomes a product, two features merge, the pricing page reorganizes around a new set of tiers. Each rename is a considered decision inside the company and a small catastrophe in the URL space outside it.
What actually happens is that the old addresses stay reachable, because breaking them felt risky. Redirects accumulate in layers, one per rename. Documentation for the old name remains, because a customer might search for it. Within two years the site carries three vocabularies at once: what the feature was called at launch, what it was called after the repositioning, and what it is called now.
Every one of those layers is crawled. Redirect chains cost a fetch per hop. Stale documentation for a feature that no longer exists under that name gets fetched with the same enthusiasm as the page you shipped yesterday, because nothing in the markup says otherwise.
- Flatten the chains after each rename. Point the oldest address directly at the current one rather than walking it through every intermediate name.
- Retire, do not merely hide. A page removed from navigation but still returning 200 is still a crawl target. Give it a real status or a canonical destination.
- Keep one page per concept, not per name. Mention the old vocabulary inside the current page. Do not maintain a separate page for each historical label.
- Write the rename down. A dated record of what changed and when is what lets you interpret the traffic shift six weeks later instead of guessing at it.
The sitemap is an instrument, not paperwork
A sitemap is usually treated as a compliance artifact: generate it, submit it once, forget it. On a site of forty pages that is harmless. On a generated site it is the single most useful lever available, because it is the one place where you state which of your addresses you consider worth attention.
The important property is that a sitemap is a claim, not a dump. A file listing every URL your framework can produce tells a crawler nothing it could not have found on its own. A file listing the eight hundred pages you would defend in a meeting is an actual signal.
Sitemap submission and recursive parsing
For sites where the URL inventory is produced by build tooling rather than by hand.
- Upload a file or point at a URL. Either input is accepted, so a sitemap generated at build time can be submitted without an intermediate step.
- Nesting is parsed three levels deep. An index of indexes of sitemaps is followed recursively, which is what a large generated site typically produces.
- Up to 1,000 sitemaps in a single job. Enough for a documentation set split by version, locale and product area without flattening it first.
- Two jobs run at once, twenty may wait. The queue is bounded on purpose; submitting more does not make anything faster.
The three-level depth matters more than it sounds. Documentation sets routinely nest an index per product area, then per version, then per locale, and a tool that only reads the top file sees a handful of entries where there are thirty thousand. Splitting a large inventory across many small sitemaps also makes diagnosis possible: when one section stops being discovered, the file boundary tells you which section.
Direct submission, IndexNow, and the limits that shape a plan
Waiting for discovery is the default. Submission is the alternative: you name the addresses instead of hoping they are found. The mechanism is IndexNow, which notifies participating crawlers — GoogleBot and BingBot among them — that a specific address is worth a look.
The limits are the interesting part, because they turn an open-ended intention into a schedule. A thousand URLs per day per account is the tracker budget. Ten thousand URLs may be handed over in a single bulk batch, which is a queue rather than a promise: the batch is accepted, and the daily budget governs how quickly it drains.
Read against that, the daily budget is a planning input rather than a restriction. Twelve thousand URLs is a twelve-day operation. Knowing that in advance is what lets you decide the order: the pages that earn money first, the reference material second, and the four hundred generated comparisons not at all.
What the status of a batch is actually telling you
A batch in flight reports three live counters — submitted, found, failed — and keeps a per-URL log recording the bot visit with a timestamp, the status returned, and the detail of any error. That log is the only place where a discovery problem becomes concrete rather than suspected.
Failures clustered by path
Errors concentrated under one route almost always mean a routing or status-code problem in that section, not a content problem.
- Check for soft errors returning 200
- Look for a redirect introduced by a rename
- Fix the section, then resubmit the section
Visits recorded, nothing in the reports
The bot came, the page was fetched, and it did not stay. This is the failure the warning above describes, and it is a content verdict.
- Compare the page against its three hundred siblings
- Ask whether any query would want this specific answer
- Resubmitting will not change the outcome
The distinction between those two patterns is the whole value of the log. The first is an engineering defect and can be fixed this week. The second is a judgment about the page, and no amount of resubmission or sitemap tuning will overturn it.
AutoSEO — when the inventory outgrows the review
For teams whose page count grows faster than anyone's capacity to decide which pages deserve promotion.
- Discovery and prioritization run continuously. Keyword finding, backlink work and on-site suggestions operate without a scheduled review meeting.
- Indexing sits beside the analytics. Submission status and search performance are read in one place, so a discovery problem is not mistaken for a ranking problem.
- Batch input where it helps. The Stream assistant accepts keyword and URL lists in bulk rather than one line at a time.
For a portfolio — a product site, a docs subdomain, a marketing site and two acquisitions — the organizing features matter more than the submission features. Linked account groups, site tags used as a global filter and per-site sharing to specific addresses are what keep the fourth property from becoming the one nobody checks. The indexing and discovery tooling is only useful if somebody can see all of it at once.
Questions that come up during the first batch
We submitted 10,000 URLs. Why is the counter moving so slowly?
Because the bulk limit and the daily budget are different things. Ten thousand is what a single batch will accept; a thousand per day is what the account processes. The batch is a queue that drains at that rate, so ten thousand URLs is a ten-day operation by design. Nothing is stuck.
Should the sitemap contain every URL the site can produce?
No. It should contain the URLs you would defend if asked. A sitemap listing everything is a dump and carries no information. The act of leaving pages out is the signal — it tells a crawler which part of a large generated inventory you consider worth the fetch.
Our docs are nested four levels deep. Is that a problem?
Recursive parsing follows three levels, so a fourth level of nesting will not be reached. The usual fix is to flatten one layer in the build step, which typically means promoting the per-locale indexes up a level. Splitting into more sitemaps at a shallower depth is better than nesting deeper anyway, because it makes failures traceable to a section.
Is it worth queuing jobs for several sites at once?
Two sitemap jobs run concurrently and up to twenty can wait, so a portfolio can be lined up in one sitting. Ordering still matters: the queue is processed as a queue, and putting an archive subdomain ahead of the site that pays for the office is an easy mistake to make at the point of submission.
Should we delete the generated pages that nobody wants?
Usually not immediately. Take them out of the sitemap first and watch what happens over a cycle; some of them turn out to serve a query you did not anticipate. Deletion is the right answer for pages describing a feature that no longer exists under that name, where the address will never be correct again.
A calculation worth doing before the next generator ships
Before adding a page type, do the arithmetic. Count what the template will produce: connectors times locales, or rivals times product lines, or releases per year times years retained. Multiply it out honestly. A generator producing forty pages per release and shipping weekly adds two thousand addresses a year without anyone approving a single one.
| Generated page type | What multiplies it | Worth submitting |
|---|---|---|
| API reference | Endpoints × versions × locales | Current version only |
| Changelog entries | Releases per year | Rarely; one index page instead |
| Integration pages | Every partner in the catalog | The ones with real demand behind them |
| Comparison pages | Rivals × product lines | Only where you can say something specific |
| CMS blog posts | Editorial calendar | All of them, if a person wrote them |
Then ask what share of that number you would submit if submission cost you a day per thousand — because it does. If the honest answer is eight percent, generate the eight percent, or generate all of it and put only the eight percent in the sitemap. Both are defensible. Publishing forty thousand addresses and hoping the crawler sorts them out is not, and it is the default outcome when nobody runs the sum.
The reporting side closes the loop. Once a batch has drained, the same account's search analytics show whether the submitted pages ever collected an impression, which is the only measure of whether the exercise achieved anything. That comparison — submitted against actually appearing — is the number that should govern the next generator, and it is why the discovery and analytics views belong in one workspace rather than two. We cover the reporting side separately in our technical notes.
If your URL count grew by a factor nobody planned, the first useful step is an inventory rather than a submission — connect the property and run a sitemap job to see how many addresses actually exist and how many of them a crawler has ever visited. That number is usually larger and smaller than expected at the same time: more pages than anyone believed, fewer visited than anyone assumed. Where the cleanup is a project rather than an afternoon, our technical engagements describe how that work is scoped, and the panel's per-URL logs are what make the result checkable rather than asserted.
