Google has crawled by site about a week ago, and only marked 5 pages as "valid". The majority of my pages are marked as "Crawled - currently not in...
Some comments have been hidden by the post's author - find out more
For further actions, you may consider blocking this person and/or reporting abuse
It's a common pattern for new sites. Google takes quite a while to properly understand a website and determine where it should stand in terms of being a reliable source of information. I imagine a lot of it has to do with avoiding sudden manipulation, though ultimately has an effect on average people like you and me, also.
About how long did it take for you, just so I can get a rough idea?
Thanks!
The whole process? I would say around ~3 months.
I run a content blog myself, and have like 12 long-form posts published, but only 4 of them are indexed. I have great links and decent exposure already, but those pages haven't been indexed yet. So, like you, I have to wait.
I speak from experience, too. I have done sites like this before and it's always around the 3 month mark when indexing becomes more regular. Granted, you have to keep the site active and work on it regularly.
Thanks so much for the knowledge, it helps out a ton!
i also had the same issue but there is a way:
robots.txtis written correctly which finds where google bot can index the sites and also Addsitemap.xmlfile location with domain inrobots.txt.Thanks for the reply! Unfortunately, all the pages marked as that are in my site map, and are not restricted by robots.txt. The files that are rerun an error instead.
It is definitely not a robots.txt thing. If pages are blocked by robots.txt then they won't show up in Google search console at all. Google's crawler respects robots.txt so won't crawl stuff if you direct them not to.
Anyway, there isn't really anything you can do to make them index those pages quicker. The fact that they crawled them is the first step. If they were listed as duplicates, or that Google selected a different url as canonical, or something similar, would be a different story. You'll likely see the number of pages marked as "crawled currently not indexed" go down over time as Google adds them to index, or excludes them for other reasons.
Please see post edit
Even without seeing your robots.txt it was clear that wasn't the cause, as Google won't even crawl those. Just be patient with the ones marked as "crawled not currently indexed." Google doesn't instantly index the pages they crawl, and doesn't necessarily index all that they crawl. It sounds like your site is new since you mentioned a week ago. Watch for other reasons pages are excluded. The "crawled currently not indexed" status is where pages end up before either being indexed or excluded for other reasons. Some pages may stay in that status for longer than others. On my domain, there are close to 400 urls that are indexed currently, and around 80 with the status "crawled currently not indexed." Some of those will likely be indexed in the future. Others may stay in that status indefinitely.
Thanks. I read this part wrong, mistakenly thinking you sad that was the issue! “It is definitely not a robots.txt thing”.
I was worried that there was nothing I could do, although I kind of expected it. Thanks for the conformation though.
try changing last line from
Disallow: /shop/*&filter_description=To
by this it will not cause errors in future
As a rule and I am fairly experienced in this field it takes two to six weeks to fully crawl a traditional new site with no authoritative backlinks, which is made mainly in html. You are clearly using search console which will help. During this period putting a link to your sitemap in your footer can be useful.
Try generating authority both with internal and external linking. If you are client side rendering a site using a js framework Google may take longer and it is true in a rare niche small website using client side js it can take longer than six weeks. Links and content build authority. If you use a is framework let me know which one or better just link the site, but with new sites you can't just trigger a recrawl and expect Google to fetch all your content as you used to. dev.to is an authoritive backlink and googlebot can follow no-follow links
This is pretty common, especially for newer sites. Google's probably still evaluating your content. Focus on improving your site's overall quality and technical SEO. Make sure your content is unique and valuable. Double-check your internal linking and site structure. It might take a few weeks, but keep an eye on Google Search Console. You can also use an indexing tool if you want faster indexing, but use it in moderation, for it can also be harmful.
Hey, thanks for the reply!
I’ve come quite a long way since creating this post almost 3 years ago (wow).
I’ve created a few more sites since posting this question, and can say that waiting is the game to play (After checking for quality and good structure of course)!
You're welcome :)
This is a typical trend for new websites. It takes a while for Google to fully comprehend a website and decide how trustworthy a source of information it should be. It ultimately affects regular people like you and me, but I suppose a lot of it has to do with avoiding abrupt manipulation.
Only four of the twelve long-form items I've written on my own content site are searchable. Although those pages haven't been indexed yet, I already have some excellent links and some visibility. So I have to wait just as you do.
I can also talk from experience. I've worked on websites similar to this before, and indexing always becomes more consistent after three months. Of course, you have to maintain the site and update it frequently.
I know it's been a while since you posted that, but it can help others in the future.
It sounds like your site is experiencing the common issue of "Crawled - currently not indexed." While it can be frustrating, there are a few potential reasons why Google might be holding off on indexing your pages:
Quality and Relevance: Google may not index every page if it perceives them as low quality or too similar to other content (either on your site or elsewhere on the web). Make sure your content is unique, valuable, and relevant to your audience.
Crawl Budget: If your site has a lot of pages, Google may prioritize crawling and indexing certain sections based on the perceived importance. In this case, it could take time for the rest of your pages to be indexed. In this cas head over to the next point: internal linking.
Internal Linking: Ensure the pages inside the "webhosting" folder have strong internal links pointing to them from other parts of your site. Pages that are well-linked are more likely to be crawled and indexed.
Indexing Frequency: It can take anywhere from a few days to weeks for pages to be indexed. This depends on factors like the size of your site, its crawl budget, and the frequency of Google’s crawl visits.
What You Can Do to Speed It Up:
Fetch as Google: In Google Search Console, use the "URL Inspection" tool to manually request indexing of specific pages that are marked as "Crawled - currently not indexed." While there’s no guarantee, this can sometimes speed up the process.
Submit a Sitemap: If you haven’t done so, make sure you’ve submitted an XML sitemap in Search Console. It helps Google understand the structure of your site and prioritize crawling.
Improve Engagement on your pages: add a search bar, a quick form, or a button to access the content.
Improve Content & On-page SEO: Ensure that your pages inside the "webhosting" folder have high-quality content with optimized metadata (titles, meta descriptions, etc.), as this can improve their chances of being indexed.
Check for Thin Content: Avoid pages with little or no content. If you have a lot of similar pages, consider consolidating them or adding more unique, valuable information.
Monitor Your Robots.txt and Noindex Tags: You mentioned the robots.txt file and confirmed that the "webhosting" folder isn't blocked. Just double-check that none of the pages in that folder have any accidental noindex tags in the HTML or within your CMS settings.
Ultimately, it's not unusual for indexing to take some time, but by following these steps, you might be able to get Google to prioritize your pages sooner.
Expert Hack :
If you still have those pages marked as Crawled but not Indexed after few month, my hack is to change the URL of the page, and re-submit it to Google (don't forget your 301 redirection from the old URL). It works most of the time ;)
Good luck, and feel free to share any updates!