I work on OnePassApps, the team behind app.onepassapps.com. This is a measurement of our own site, taken today over plain HTTPS - no browser, no session, no cookies, no JavaScript.
Our app host answers a robots.txt of 160 bytes and 8 non-empty lines. One User-agent: *, one Allow: /, 5 Disallow lines and one Sitemap line. Nothing in that file is a comment, and there is no Crawl-delay and no per-bot block.
Here is what those 5 rules actually name, when you go and ask the host for each one.
Two of them name a path this host does not have
/api and /api/ are both disallowed. On this host both of them answer 404, and so do /api/plans and /api/apps: four requests, four 404s, each one the same miss document, titled "Page not found" with a Cache-Control: no-store header and 14,173 bytes.
The API is not served from this host. It is served from https://api.onepassapps.com, where /api/plans answers 200 with our plans data. That host has its own robots.txt, and the deeper read of the two files is on Dev.to: https://dev.to/onepassapps/two-robotstxt-files-one-product-the-app-host-has-rules-the-api-host-has-1248-bytes-of-licence-4o08
So the two rules that look like they close the API on the app host do not close anything there, because there is no API path on the app host to close. A crawler that obeys the file is not refused anything - it gets the same 404 document any other missing path gets.
The three that do exist carry no canonical
/account, /sign-in and /sign-up are the other three Disallow lines, and all three are real pages. Each answers 200, each carries <meta name="robots" content="noindex, nofollow">, and each carries an og:url pointing at itself. The titles are "Account — OnePassApps", "Sign in — OnePassApps" and "Create an account — OnePassApps".
None of the three carries a canonical link. The word "canonical" does not appear anywhere in the HTML those three return - I checked the whole document, not just the head. A page with no canonical is not an error for a crawler, and these three ask not to be indexed anyway, but it does mean the set of pages on this site that carry a canonical link is not the set of pages a crawler can reach.
And the fourth page off the index that the file does not name
/support is the fourth page this site keeps out of the index. It answers 200 - its title is "Support and feedback — OnePassApps" - it carries noindex, nofollow, it has no canonical link either, and it is not in the sitemap. robots.txt does not name it - its own meta tag does the whole job on its own.
The map the file points at
The Sitemap: line points at https://app.onepassapps.com/sitemap.xml: 4,046 bytes, a plain urlset with no <lastmod>, no <changefreq>, no <priority> and no count attribute. 56 URLs in it, and they divide cleanly:
- 22 app routes
- 22 guide routes
- 7 studio routes
- 5 site-level pages: /, /guides, /plans, /terms, /voting
All 56 answer 200, and every one of them carries a canonical that points at its own URL.
What I did not measure, so you know what is not claimed here: one build, one network, one signed-out session, 11 November 2026. I read the app host's robots.txt and the app host's own routes; I did not re-read the API host's file line by line in this pass, and I did not test how any particular crawler interprets any of this - this is what the rules say, not what a crawler does with them. The byte counts I quote are from three requests each, and they move by a few bytes between requests because the address in the footer is obfuscated per request.
The apps themselves are here: https://app.onepassapps.com?utm_source=devto&utm_medium=post&utm_campaign=robots-rules&utm_content=app-link . Every app has a guide page beside it, and the guides are collected at https://app.onepassapps.com/guides?utm_source=devto&utm_medium=post&utm_campaign=robots-rules&utm_content=guides-link
I write here as the OnePassApps team - if any of this does not match what you see in your own browser, say so and I will go and look again.
Top comments (0)