Release from Google Sandbox Only to Search the Playground
The Google Sandbox Effect has been discussed at length in ourcase study of a new website first crawled in May by Googlebot.We can now further the case study with indexing comparisonsand discuss interesting Googlebot crawler behavior afterrelease, at the 75 day mark, of the study website from thatvery confining Sandbox.
This case study is not for the faint of heart - those justlaunching a new web business on a new domain name with hopesof instant indexing and immediate traffic may find theirwebsite very lonely for two and a half months - if it is in acompetitive market segment. You may as well plan to stay inthe Google Sandbox for at least 45 days on average. If someearly release stories are to be believed, search phrasesnobody wants to play with are taken pity on by Google and senthome for early release.
Those non-competitive or obscure search phrases seem to beseen as good, quiet little children, playing by themselves inSandbox playground and are sent home early on good behavior.Googlebot probably sees good behavior as playing well withothers, like a good little baby domain and NOT beingcompetitive as some young domains can be. Throwing sand inother childrens' faces and insisting on having your siteindexed, throwing sand out of the Sandbox with your brightplastic toy shovel and bucket will not be allowed.
Now that the site discussed in this study is out of theSandbox, it still lingers on the playground, unable to escapethe community park and leave for the business world to playwith the big boys in the outside world. It does indeed taketime to grow up and be the model citizen in this new searchplayground. Though on the first full day after this first weekof being released from the sandbox, the site has gotten 68visitors referred by searches done at Google, the firstreferred search traffic coming into the site. MSN has sent 8visitors, Yahoo has sent 6, 4 came from AOL searches, 2 fromNetscape and 1 from Dogpile.
The indexing behavior of Yahoo and MSN has been nothing shortof bizarre with numbers of indexed pages increasing rapidlyover the first two months to reflect 6,941 pages indexed until8 weeks into this study and we outlined previously how numberschanged as you click through results pages first upward, thendownward to about half the total of highest numbers listedalong the top of the results pages.
It appears that Yahoo and MSN are playing on the 'slipperyslide' in this playground, climbing to the top of the ladderof results at about 10 week mark showing 8,210 and 6,941 pagesrespectively indexed, then sliding down again to 3,510 forYahoo and 373 for MSN, as of this writing two weeks later onAugust 6. Still, Yahoo will show you only 1,000 (100 pages) ofthose results and MSN will show you only 250 results, or 25pages, no matter how many they claim to index. MSNbot iscrawling the site faster and more consistently than any of theengines, yet shows by far fewer pages indexed than the others.
One of the interesting comparisons between Google and MSN inour Sandbox study is that Google will show you most of whatthey claim to have indexed after you click that link at thebottom of the first page showing only 3 or 4 results when youuse the "site:Publish101.com" query operator then go to thebottom of the page and click the link under the line reading,"In order to show you the most relevant results, we haveomitted some entries very similar to the 3 already displayed.If you like, you can repeat the search with the omittedresults included."
Go ahead and click that link, then you'll be presented withthe claimed total of indexed pages. That number has verysteadily increased since Sandbox release after 75 days fromfirst crawling of this Sandbox study site. The timing andnumbers of indexed pages at Google goes upward, and ONLYupward with VERY distinct patterns noted from raw log files.Crawling schedules seem to have been established for this siteby Google and indexing changes occur on a very regularschedule.
The first observation of Sandbox release was at noon onThursday July 28, seventy-five days from first crawling byGooglebot when a search turned up 379 pages indexed with a"site:Publish101.com" query. That number increased later thesame evening to 3,660 pages at a search done around the dinnerhour Pacific time. Oddly, the next day, Friday July 29, thenumber took a slight hop upward to 3,700 pages and on thefollowing Monday, showed 3,770 pages indexed.
That schedule and pattern have repeated on the second week ofSandbox release when a "site:Publish101.com" query produced5,660 results from from Google for the site on Thursday August4 at just after noon and then nearly doubled at around thedinner hour to 10,700 pages on that same query. A final checkjust now on Saturday shows it at 12,100 pages indexed byGoogle. It should be pointed out to those who wonder about thetotal number of pages that this is a dynamic site with a verylarge archive of articles that increases daily as newsubmissions are contributed by member authors at the site.
Those articles are added through a content management systemon a daily basis by an editor who reviews submissions andprocesses them for approvals or rejections. Those approved aremade live from the home page nightly. We've started doing thison the crawler's schedules as we've noted very regular visitsby Yahoo's Slurp crawler to the site home page just once dailyat around 5pm each evening and Googlebot visiting the homepage only once, at near 11pm nightly, so we've instituted amidnight activation of each day's new article submissions onthe home page of the site so that none of the new pages aremissed by those crawlers. MSNbot seems to hit the home pagemultiple times through the day, so timing is less importantfor MSN.
Crawler activity has been heated, with Yahoo crawling theleast and the slowest, barely seeming to attempt any updatesand the total of indexed pages has not changed for over threeweeks since it peaked at 8,210 pages indexed and then droppedto it's current level of 3,510. As previously stated, Slurpseems to be unhindered by any form of consistency in indexingor crawling behavior. MSNbot has crawled extensively andfairly regularly for weeks, but that odd indexing behavior isa serious flaw in their utility as a search tool.
It should be mentioned here that AskJeeves had been noted tocrawl the site extensively early in this case study anddisplayed a very regular and consistent crawl, but stoppedabruptly three weeks ago on july 13, after hitting most of thepages then available on the site. Teoma, their spider, hasbeen absent ever since and they have not indexed this domainat all since first crawling on May 23, over 10 weeks ago.Clearly, Teoma appears to have the longest Sandbox of all thesearch engines.
Much has been learned in this Sandbox case study about crawlerbehavior, indexing delays, robots.txt requirements and indexupdates at each of the top three search engines. Where thatknowledge leads will, of course, change as algorithms andcrawling schedules are adjusted by MSN, Yahoo and Google. Butvaluable information has been shared that may help otherwebmasters to better understand each of the factors thatdetermine the success of any website.
"Further findings in follow-up articles at the 3, 6 and 9month marks, explore search referrals gained as Google addsmore pages and rankings fluctuations begin to level.Meanwhile, we'd like to encourage others to publicly review their crawler traffic through logs to compare behavior on newdomains to verify findings and disclose indexing behavior and timing for new domains and further document SE indexing as well as crawling behavior.
Copyright © August 6, 2005
Previous Sandbox Case Study Articles:
Mike Banks Valentine is a search engine optimizationspecialist
The Latest Craze: Local Search, 7 Steps to Being #1 in Your Local Market
Anyone would agree that it is much easier to be number 1 out of 100 or 500 then 1 Million or 200 Million. With these 7 Steps you should have no problem being number 1 in your Local Market or Markets
The Myth of Search Engine Submission
Contrary to what most people think, it is not necessary to submit your site to the search engines. In the early days of the web, when search engine technology was still primitive and search engines' ability to crawl the web was somehow limited, it made sense to submit your site.
SEO and the Outsourcing of Inbound Link Building
Search Engine Optimization nowadays has a lot to do with building inbound links to your website. Building inbound links is a cumbersome tasks and webmasters have always been looking for shortcuts to do this. Webmasters buy links (as advertising as an example) or contact other webmasters to exchange links with them. The need for inbound links has created a new business opportunity in the search engine optimization industry. The outsourcing of link building emerged from the fact that many inbound links mean a high search engine ranking and/or a high Google PageRank.
How Do I Improve My Web Site Conversion Rate? Part 1
Six Reasons Why Your Alexa Rating Is Still Important
1. Additional Exposure For Your Site.
Everything You Wanted To Know About Google -- But Were Afraid To Search For!
(A Reflective look at the little search engine that soared!)
Search Engines The Masters Of The Internet Universe ? Part 1
Trillions of Billions of content pages make up the wide world of Internet. Keeping a house clean and arranged with proper placement for each household item is so big a task for each of us that it is a much despised daily chore. Ever wonder who or what keeps the Internet clean organized and keep them in arms length for you when you need it? This humongous task of making sure the right set of data is kept organized and delivered as information to the right folks when they need it are done by the Search Engines (SE). There are many popular search engines that deliver search results for countless number of searches every Nano-second. The Internet is the first place folks go for obtaining any type of information. The Internet has become the knowledge warehouse of what the world knows. Well almost true. If you have not discovered the full potential of the search engines it is not too late.
A Real Example of Search Engine Optimization (SEO) Success
The term, Search Engine Optimization (SEO), refers to a set of techniques by which web sites and web pages are constructed for maximum recognition and ranking by search engines such as Google, Yahoo, and MSN Search. Using the right techniques can guarantee top listing positions for keywords and keyphrases that are related to a site.
PageRank for Websites: Is There More to the Web?
Google's PageRank has been around for years, and in the opinions of a lot of e-business owners, it can make or break a site. Lately, with Google's fingers in every pie, it seems important to remind everyone that there is more to a website than just PageRank. PageRank is a term that relates to the algorithm that Google uses to rank a website in its search engine. Coined by Larry Page, one of the engineers of Google, PageRank has come to mean so much to webmasters and SEO's that it dictates how we market a website. But let me coin a few terms of my own. (Or, borrow them from others, perhaps?) And while some of these concepts are included in the PageRank algorithm themselves, it's often helpful to be reminded that there are many factors that a webmaster should concentrate on, and not just one overwhelming aspect. It bothers me that Google's toolbar's PageRank indicator "measures the IMPORTANCE" of a page; important to them, perhaps, especially in light of the release of the Google Toolbar for Firefox on July 7th. But lack of PageRank doesn't mean that your site isn't important. So how do you let the search engines, Google in particular, know that? This article is a collection of the phrases that indicate the behavior of search engines today.
The Value of Search Engine Marketing
Search engine marketing vs traditional offline marketing. Most often, traditional offline marketing such as TV and radio ads cater to the masses. Out of the tens of thousands of TV viewers or radio listeners, there may only be a handful of prospects who are interested in your services or products. Sometimes, having spent thousands of dollars on these advertising channels doesn't produce effective targetting at all! It all comes down to ROI and target marketing.
Link Building in Light of Vision-based Page Segmentation
The days of basing a successful link building strategy on link quantity and anchor text alone may be numbered. The link popularity theories behind PageRank and Hilltop remain important, but major search engines are continually adding new elements to their link algorithms to improve search relevance. One of these new elements is the concept of visual page segmentation which was recently proposed in a paper entitled "Block-level Link Analysis," by Deng Cai, Xaiofei He, Ji-Rong Wen and Wei-Ying, available online at http://research.microsoft.com/research/pubs/view.aspx?tr_id=690.
How to REALLY Profit from SEO
I want to give you a few more things to think about as you excel and grow in the craft of search engine marketing. If you are anything like me, you were hooked the first time you really made a difference to someone else's success. I soon realized that being able to help business owners to get results from these optimization methods and strategies could also be amazingly profitable. I found out that customers are your greatest resources and many of them are quite generous when you make an impact on their business. So the topic here is for those who want to profit from their skills (if you are not already doing very well and having an awesome time already).
How To Make Your Website More Successful? (Part II)
In part I of our series of how to make your website more successful we already showed you some important tricks to build a more successful website. This time we are going to expand the scope a little to further improve your website and to make it work harder for you on the Internet.
Website Ranking With an Internet Marketing Specialist
On the internet, competition is stronger than ever. There was a time where paying a few bucks to get in Yahoo was enough to generate substantial traffic but marketing websites on the internet got much more complex since. Google is now a major player in the search engine industry and any serious internet marketing specialist and seo expert knows how important it is to get a good website ranking in that popular search engine. Understanding Google's algorythm along with having good html and writing skills can often make the difference between being an amateur or a good internet marketing specialist. Although, many other aspects that we will cover here should be taken into consideration when comes the time to find the right internet marketing specialist for your website.
Why Search Engine Marketing Has A Passion for Web Site Usability
Watching a recent football game, I imagined two very different teams: one called ‚??The Horders‚?? and the other, ‚??The Hunters‚??. In the game, it takes planning and skill to carry a football a few yards. There‚??s interference and distractions. Scantily clad dancing girls are screaming cheers nearby.
10 Quick Ways To Kick-Start Your Profit Pulling Keywords
First, you must realize that targeting the right keywords or phrases is the 'key' to making any kind of profit from your site. Choosing the 'right' keywords (the exact keyword or phrase surfers type into the search engines to find yoursite or product) can make or break your online venture.
Google: The Ultimate Web Writer?s Style Guide
Indulge me for a moment.
Design A Spider Friendly Site
To be successful in the search engines it's important to design your web site with the spiders in mind. Using the latest in web page design is not generally the best way to go. Spiders don't view web pages like humans do, they must read the HTML in the page to see what it's about. Below you will find tips on how to best design your web site with search engines in mind.
Part I : Getting Free Hits Using These Simple Tips & Tricks
Search Engine Optimization
Whats with the Competition? Ever Heard of Cooperation?
I attended the SES Expo in San Jose, CA the other day. It was an hour from my home in Larkspur, CA just north of the Golden Gate Bridge. Along the way I was thinking this would be really neat to see, a lot of companies doing the same things really, and to see how they play together.
|© Athifea Distribution LLC - 2013|