DISQUS

Anonymous · 3 years ago

This is one of the most confusing posts I've ever seen.

"Members Only" content for where/what?

risherz · 3 years ago

Oh... okay, could you please tell me where I went wrong and where you get confused. Actually I kind of agree, I do need to write it up a little more properly, so I'll be updating this recipe or something later.
Yes I guess I should have rephrased "Members only" content but here's a little example of how this works.....
Go to http://shortenthis.com/zzdtbgl (thanks to bugmenot for the redirection link). See how it says Log on, subscribe to articles and pay about US$130 or something like that... and you only see the short preview of the article...

Now, switch your user agent to Googlebot 2.1 you should be able to read both the pages of the article(the complete article) and view the comments (all without logging in and without paying $130 or anything like that)

This is because the site thinks you are "Googlebot 2.1" and gives you access to the information so that you can index all of the information. But of course you know that you are only there to do something else.

You can also gain access to the yahoo rich text editor which is only meant for Internet Explorer if you are using Firefox... so you don't need to switch around when you are using Firefox to write e-mails using yahoo..

And please note that do not use this for evil purposes...

Hope this helps the confused...
Risherz

Anonymous · 3 years ago

Am i doing something wrong? I cant get it to work on www.backpackinglight.com...
any help is appreciated

Anonymous · 3 years ago

changing the user agent to googlebot will only gain you access to sites that allow googlebot to view the entire site. it's possible that site does not. try it on the example site he linked to previously in the ccomments

(not the same bugmenot)

risherz · 3 years ago

Yeah, I believe the bugmenot above is right (funny how many use bugmenot... its confusing actually) Some sites dont make all of the content accessible for search bots.

The thing is that if you make the content of the whole post accessible for bots like google bots, chances are that you are going to get more visitors. This is because now you probably have more keywords in your site so you might get more visitors. However somesites such as this one probably store all the member only content somewhere else and make sure that only members that pay can access it. And that's why they have the small description of the review's etc, so that's the only thing google bot indexes.

My point is that this trick will work for many sites, because many sites are trying to get more and more visitors come to the site and try to make money out of that. On some sites this trick won't work because they really want people to sign up and don't care if bots like google bot can access the rest of the information... sorry.

Hope this helps
Risherz

Anonymous · 3 years ago

Don't do what I did and set your UA to Googlebot and then forget that you did it. Some sites like http://del.icio.us actually block Googlebot's access entirely (allowing only access to RSS feeds and such). If you've set your UA to something else and forgotten about it, visiting such a site and getting a 404 or some other error can sort of catch you off guard. :D

Anonymous · 3 years ago

How many websites does this work for? Or more specifically, which type of website does this work for?

risherz · 3 years ago

<ul id="quote"><h6>spoonmonger wrote:</h6>How many websites does this work for? Or more specifically, which type of website does this work for?</ul>

Well... I'm not sure how many websites this trick works for because things are always being changed.

As for your second question, this trick works for a lot of tech websites where you need member subscription. For example windowsitpro.com . Mainly this trick works on websites that have a better search rating or wan't bots like the google bot to look at the complete content of the site.

So as a general rule I would say that if you do a search on say google and one of the top queries listed has what you are looking for... you click on that link and go to that site but that site doesn't have what google says it has then try changing your user agents and trying again... chances are you will be able to see the content that is only ment for subscribers and bots ;)

Hope this answers your question...
Risherz

Anonymous · 3 years ago

ho ive veen to the windowsitpro.com site and all articles seems full articles could you give us an an example for an article in this site that can't be accessed if your not a member and you can access it with the user agent thing ? and hos does it work do you click on log in after switching the agent ? could you give us morec details ? and does it work withh downloadable ebooks sites and with porn sites if so how does it work and thank you very much

Anonymous · 3 years ago

K this is totally useful. There are several tech advice sites that function like this... Asking outragous amounts of cash for simple questions / results...
Very good.

Anonymous · 3 years ago

can u list solme of those sites ?

Anonymous · 3 years ago

Hi,
The "general.useragent.extra.firefox" isn't on the list of preferences for my firefox. Is there a reason for this, and is there another way for me to get it to work, or should I just download it?

risherz · 3 years ago

<ul id="quote"><h6>kingking wrote:</h6>can u list solme of those sites ?</ul>

Well, I don't have a list of sites right now, but I'll try to put together one soon. Right now you can use this on a lot of tech sites like I said above...

<ul id="quote"><h6>bugmenot wrote:</h6>Hi,
The "general.useragent.extra.firefox" isn't on the list of preferences for my firefox. Is there a reason for this, and is there another way for me to get it to work, or should I just download it??</ul>

If you don't see the "general.useragent.extra.firefox" in the preference name then you probably have an older version of firefox. Just get the newer version from mozilla.com. I don't think there is another way to get this to work without downloading the extension. I think is it better to download the extension, this might help you switch back to the default user agent quicker.

Hope this helps :D
Я

Anonymous · 3 years ago

You really need to remove the direct link you have from this site to the example site. The way you have it now it will show up in their referrer logs and they may possibly follow it back and learn of the small exploit and close it up.

I took the liberty of creating a redirect for you:
http://shortenthis.com/zzdtbgl

risherz · 3 years ago

<ul id="quote"><h6>bugmenot wrote:</h6>You really need to remove the direct link you have from this site to the example site. The way you have it now it will show up in their referrer logs and they may possibly follow it back and learn of the small exploit and close it up.

I took the liberty of creating a redirect for you:
http://shortenthis.com/zzdtbgl</ul>
Wow, I completely overlooked the fact that they could trace this! Thanks so much bugmenot! I've corrected the link.

Great idea of redericting the url... sometimes I totally forget about those things... hopefully the people at digg didn't read the comments... because many people have posted comments on where you can test this out :( Hmm... but i still wonder if they will close this, the reason why they get a lot of visitors is probably because of this. But, you never know...

Thanks again :wink:
Я

Anonymous · 3 years ago

Ok, I downloaded the plugin for firefox and typed EXACTLY <font color="red">Googlebot 2.1</font> in the create useragent. I set it as [color=red]Googlebot 2.1[/color in the description and name, but, it wont work on ANY site I go too! Help please.. :(

Anonymous · 2 years ago

Hi. I'm a different person. Not the other guy who uses this account.

I figured I would add two helpful hints.

1) If you visit ipchicken.com , you can see what user agent your browser is sending. There are plenty of other sites that do this, but this is one of the easiest to remember. Your user agent is displayed after the "browser:" field.

This should aid people who are having problems. It will either affirm that you have switched the user agent correctly, or affirm that you haven't.

2) You can achieve something similar to the goal of the recipe by using google cache. The advantage is not having to switch your user-agent back and forth. The disadvantage is that google cache is usually quite ugly. It is also sometimes out of date.

It's been mentioned already, but some people don't seem to understand. This only works for sites that are specifically set up to allow googlebot but no one else besides members. Test that you are set up correctly on the site mentioned in the recipe. It is guaranteed to work (or so I assume. haven't tried it myself.) For any other site, this trick might work, might not.

Anonymous · 1 year ago

i really see no reason sites
should treat googlebot and normal visitors any different and so i see
no problem doing this

there should not be search results in google if you cannot actually access the information

Skippy · 3 months ago

There is a reason why they do that.
The reason why some member-only sites allow Googlebot to see the info, but not a regular person, is because the genuine Googlebot will collect the information and post it on Google, and these sites really want it because it would attract more people to become members.

timmy g · 9 months ago

thank you! firefox forever!