Well, this is weird. Was it officially announced and now removed without any warning?

I didn't hear any official word from Google about this service – it is possible that they inadvertantly posted the link on their site?

Yahoo cached it:
http://216.109.125.130/search/cache?p=http%3A%2F%2Fcode.google.com%2Fapis%2Faccounts%2FAuthForWebApps.html&ei=UTF-8&fr=moz2&u=code.google.com/apis/accounts/AuthForWebApps.html&d=CjDx1jmtNBkt&icp=1&.intl=us

Good catch Ionut. I saved a copy from that just in case :)

Well it should be offical... they aslo pointed and created an offical group discussions on the API.

Where is the group discussion?

Everything appears fine to me:
http://code.google.com/apis/accounts/AuthForWebApps.html#tokenmgmt

I can't for them to release this.

Um, it loads just fine for me.....

Loaded just fine right know. Maybe was just a temporary error.

gregstoll: The Google Group is at <http://groups.google.com/group/Google-Accounts-API>.

It seems that the page is back online indeed.

The pages load for me too. Stop being so paranoid.

http://groups.google.com/group/Google-Accounts-API

Fine here too – UK.

I can confirm that it *did* disappear, saying that it would be released in April, but now it's back. (Even when the link disappeared, the page still said that it was last modified today...)

Yep, it's back. And the navigation link returned as well. Flaky indeed :)

Much more likely than "a page was out there, then it was pulled, then it was put back" is that 1-2 data centers lagged in getting the newer page/data. So if you were hitting that url on a web server with the older data, you would have seen whatever the url showed previously.

Think of google as not one mega webserver, but a bunch of regular webservers spread throughout the world at different data centers. If you happened to find a web server that got the newer versions of the url a little later than everyone else, that would explain it.

I haven't talked to anyone else at Google about it, but that's my educated guess. That would also account for why several people could see it just fine.

Best,
Matt

Thanks Matt! I think we all saw the same for a while there – Garett mentioned the page was gone, I checked and saw it gone too, and the first people checking at that time saw it gone as well – but it was then pretty quickly back live. Of course, could still have been a data center rollout thingie...

Hey Matt – you are supposed to be on vacation for another day ... ;-)

Yeah, the Matt Cutts countdown clock indicates 23 hours to go!

No-- I am sorry, I cant accept Matts response!!

" If you happened to find a web server that got the newer versions of the url a little later than everyone else, that would explain it."

Then Matt is telling us that their replication process across the farms takes more then 48 hrs-- for the world to see consistent information /viewpoints ??

After all regardless of which server we saw it on (i.e the closest DC), it was like 48 hrs ago!!

I'm still on vacation, but my wife is out of town, so I'm catching up on my surfing. :) Right after this, I'm going to go on a hike I've been meaning to do for a long time.

/pd, I always saw the correct version of the page when I tried to load it, so I never had a problem viewing the AuthSub page. That's probably because of the data center that I was routed to. Sometimes we also take data centers out of rotation to increase machines, do maintenance, etc., so there may also have been a data center that was catching up for that reason.

I'm not trying to argue, just offer an equally possible explanation. I'm saying that when you're load-balancing between several web servers and a page appears to be gone or an older version of the url, it's more likely (in my experience) that data is still propagating than that a page was pulled and then put back. There's nothing really sensitive on that page anyway, so it would surprise me if the latter happened, but not the former. :)

If I go missing, send searchers to the PG&E trail: http://kevingong.com/Hiking/RanchoSanAntonio.html
:)

I love how the "last modified" date on these pages (ie. http://code.google.com/apis/accounts/AuthForWebApps.html) is actually generated using javascript – the "last modified" date is always today's :)

And welcome batt Matt... it's been WAY too long! :)

hey Matt – not arguin ..only diffrent points of view and just go for the wildcat loop .. We can take the issue up later on :)-... hehe

Find something better to do with your time pd. He knows the process better than you. And Philipp Lenssen please hold off next time instead of jumping the gun. You seem to have forgotten the complexity of software rollouts. This is where you can't get it perfect everytime.

Hmm.. apart from allowing users to log into your service using their Google Account can anyone see a 'use' of this. Are their any Google services with data we can access now?

Oh thats an interesting comment Yu!!

I am sorry to hear that this is a "complexity of software rollouts" – and yes, its flaky.. and we have .. commented on a flakly process-- so that they can better their own processess..

Oh yeah in my world.. --SLA's normally indicate that DC'sshould get zoned in for industrial strength applications within 40 minutes or so.. thats assuming that there are no h/w failures..

oh BTW.. I have nothing else better to do ..Now what ?? Jump the gun again ??

Jason Schramm:
> Stop being so paranoid.

Yu:
> And Philipp Lenssen please hold off next time
> instead of jumping the gun.

Hmmm. I posed this as a question in my post, if you look back: did Google remove something due to a security issue? We saw this happening with Google Web Accelerator, actually – the tool got pulled shortly after launch due to a security issue. So while I never said this is likely, as you can see it is also not completely unlikely. Also, usually when something is rolled out it'll either be completely invisible to people or visible, but this time, there was a link in the navigation which was then removed... or if the data center rollout is the correct explanation, then there was an earlier version of Google AA in which the link wasn't there but the rest of the page was.

Just to echo what Matt said, I've spent the last two decades or so in System Administration helping manage large Corporate networks. While our data replication problem wasn't nearly as huge as Google's, if you looks under the covers of large-scale data replications, there are always some warts.

I.e. the 1,000+ machines in our network would all see the updates (typically unless a major change) in a few minutes. But what sucked was when a target machine was down. Yea, ideally we would have some sort of process fire up at reboot that would auto-sync, but since there are a variety of replications processes, it's hard to cover all cases ... and trust me, when you are dealing with large numbers, all sorts of oddball things occur – some examples are lack of disk space, authentication issues, permission issues, admins fiddling with downstream servers rather than modifying the golden master, etc.

So 99.9% of the time all would look the "same" to the user community, but every once in a while something would be "old" – s*it happens – and we'd usually notice it before end-users did and fix it before any squawked.

So Matt's explanation makes sense to me here ... but (on a seperate but semi-related topic), I'm not sure I buy a "bad data push" as the reason that 5,000,000,000+ spam pages (via the recent subdomain hack) showed up in their index.! ;-)

Alex : I can live with a target machine going hard down.. thats murphys law in most cases.. and then explaination makes total sense and I'll have to conceed.

However, I noted that the likly cause of this glitch was ;

==> " is that 1-2 data centers lagged in getting the newer page/data."

which to me does not make sense, because Garett reported these pages approx 36-40 hrs prior to users noticing that those pages were not "propagated" or philipp making this post.

Now, we all know that at this level of infrastructure deployment.. warts are always around, both visable and invisable.. but what is the likey cause of such 'anomlies' – after all we are all in different time zones and will get pointed to different DC's...yet users saw the same "not available" status.

So If I am doing a casual root cause analysis (Note, I am only debating the glitch/issue and not a person .. lets stick with the issue);
...then I am left to infer;

1) The pages were changed and needed to pushed out again.
2) The pages were 'pulled' and then repushed for reason best known to google only.
3) The was some 'major network' lag between DC's and for a brief window of time- all users got pointed to that DC which was not updated.
4) There was a machine that went down and all users were pointed to that machine.

Wwhat am I missing and makes sense here ?? just asking second opionions

First, I don't work at Google – only they would know for sure. I was only presenting on explanation based on my experience that *could* explain this anomoly.

I didn't see/understand where Garett noticed/reported this 36-/40 hours beforehand. Did he report this to Google ... and did he get a response (hah, hah! ;-)

BTW, Philipp strikes me as someone who is fairly careful and others saw this glitch ... so my two cents is he accurately reported this. There's quite a few oddities/burps/etc. in the big "G" if you look close enough. So I don't understand why some people gave him a hard time about this.

Splasho, you raise a good question. Nike's soccer site Joga.com is a good example of this sort of idea; rather than mess with doing their own username/password system, they let you use your Google account to log in.

alek, sounds like you've run the gauntlet of large-scale data replication before. It can be a bear to get right when machines, huh? :) I'm sure I'll have an opportunity to discuss the "5B" + bad data push issue--maybe a "here's what happened while I was gone" blog post. :)

/pd, you're right that I should have done the wildcat loop. The PG&E trail was a loooong trail, and the vista point wasn't all that vista-y. :) Maybe if I'd remembered to use bug spray or sunscreen it would have been better. Oh well. Just to close this issue out in my mind, when I've seen stuff like this in the past, it was usually because new data hadn't made it everywhere yet. But I haven't asked anyone about it, so I won't claim any special knowledge.

Matt,

Yea, be real interesting to have you do a "look back" on the "bad data push causes 5+B (subdomain'ed) results to pop up in the index" ... quite frankly, it caused quite a few chuckles in the blogoechosphere as you may have seen. If nothing else, it's entertaining to see the continual "arms race" between the spammers and the search engines as you guys battle each other.

I might suggest a
   "I'm back from vacation and plan to talk about X, Y, Z in the next few weeks – anything else I should cover that happened while I was gone?"
would be an excellent intro post since seems people are chomping at the bit to ask you questions and that would keep 'em at bay.

BTW, were you successful in cloning yourself while on vacation?!? ;-)

alek

P.S. Yea, I've done the Sysadmin thing for a while – brief background here – http://www.komar.org/bio/bio-alek.html ... and just to be clear, I'm not bucking for a job at Google or whatever... although if you can tell me why my web site (about as white-hat as they get IMHO) got clobbered like never before from your June 27th update, I'd love to know that.