[pointertovoid]

Hello dear readers!

Ignoring nearly everything about websites, I'd like you to explain me why Google occults a website, or rather parts of it, in its search results.

Raw elements:
- Forum not listed, nor its "Sciences" section , but does list the "General discussions" section
- Other search engines occult the forum as Google does
- Said forum describes no drugs, explosives, misconducts... But it does contain my inventions (civilian uses) in the occulted sections
- Other forums where I put a few inventions, and links to this Saposjoint, are listed by Google
- Until recently, I didn't receive e-mails sent to myself containing an address of the Saposjoint website.

Questions:
- I browse from the semi-free country I happen to inhabit presently, called France, whose official policy is to censor some websites, under various excuses (explosives and so on).
Could you please check Google's answers from your country?
- Can the Saposjoint website be programmed (Php) in such a way that search engines occult some sections?

Thank you so much!
Marc Schaefer, aka Enthalpy, aka Pointertovoid

This post has been edited by Tripredacus: 07 March 2011 - 10:09 AM

[Tripredacus]

I can find those sections with Google. Are you only using google.fr?

I notice you only have 1 keyword tho, just your website name.

[pointertovoid]

Thanks for helping!

I get the same results through google.fr, google.com, google.de, google.es used from France, and from other search engines.

When searching through Google some text picked from threads in the occulted sections, I don't get results. Examples:
- "tinnitus, it is obnoxious" or "technological state just now than Krypton"
- But right now, Google finds "remotely operated like Predators and Reapers", which it did NOT on 4th of March (today is 7th) despite the page is from February.
Hey, Msfn has real magic effects!
- Google doesn't find any "Marc Schaefer, aka Enthalpy" on forum - though I put hundreds there.
- The occulted discussions are not the ones where I contributed nor the ones containing my name. But the occulted sections are the ones describing my inventions.

More observations:
- BAUTforum.com crashed regularly as I attempted to put links there to forum. But other forums did not.
- Now I receive emails from myself containing a link to forum, which was still impossible on 13th of February.
I had to relay to myself the sent message in order to receive the second try. Same without the antivirus nor the firewall. I checked the message rules for Outlook Express as well.
This improved as I began to experiment on it systematically. One other magic effect.
And I always received emails containing links to other forums quite normally.

I ignore how Php works... The computer hosting forum re-creates the web page each time a visitor asks for a page, as opposed to Html, isn't it?
Would it then be technically possible that forum is programmed to detect the bots sent by search engines, and produces for them expurgated pages?
Though this alone wouldn't explain the censored e-mails nor the crashes at other web sites.

This post has been edited by Tripredacus: 07 March 2011 - 10:10 AM

[Tripredacus]

pointertovoid, on 07 March 2011 - 09:41 AM, said:

I ignore how Php works... The computer hosting forum re-creates the web page each time a visitor asks for a page, as opposed to Html, isn't it?
Would it then be technically possible that forum is programmed to detect the bots sent by search engines, and produces for them expurgated pages?
Though this alone wouldn't explain the censored e-mails nor the crashes at other web sites.

It has nothing to do with PHP. Most forum software works that way where it only "creates" pages as needed, as most of the data is kept in the database.

You can try using a proxy in the US to see if you get different google results that way.

Quote

Hey, Msfn has real magic effects!

Indeed. I have removed all links and name to your forum.

This post has been edited by Tripredacus: 07 March 2011 - 10:11 AM

[pointertovoid]

Tripredacus, on 07 March 2011 - 10:08 AM, said:

I have removed all links and name to your forum.

Why? They were useful to readers willing to check Google's behaviour.

By the way, it's Sapo's forum, not mine. I'm a member there, no other relation.

[pointertovoid]

The name of the forum there is saposjoint.net

Its section called Science gives no hit in Google search, at least from here.

But its section called "General discussions" does.

[jaclaz]

It is perfectly possible that there is a "robots.txt" preventing indexing a part of the forum and not another.
http://en.wikipedia....lusion_standard

You should ask the actual Forum Administrators if such a policy is in effect and how it is set.

jaclaz

[submix8c]

Interesting... I Google what you said "General Discussions" and when I try to go to it I get "Forbidden" in IE6 but works (can go to it) fine in FireFox.

Riddle me that... Maybe that's why Google can't pick up on it?

(the website+)/Forum/viewforum.php?f=76

Fine in FF, but in IE6,
"Access forbidden!
You don't have permission to access the requested object. It is either read-protected or not readable by the server"

Try it... (I think the forum has "bugs")

This post has been edited by submix8c: 07 March 2011 - 06:13 PM

[pointertovoid]

Internet Explorer : o yes, I had forgotten that. The site doesn't accept it here neither. Its webmaster, who has a Linux background, explicitly asks to use Firefox. Maybe he got some disappointments with IE, and his opinion about M$ won't have given him patience to solve them? So he probably put something like
if navigator==IE then goto hell

I suppose this isn't the reason why search engines occult the site, because
- I see often their bots in the visitor's list of that site
- All sections of the site reject IE, but some are indexed by the search engines, while others aren't listed
- My intuition tells that the bots sent by the search engines don't rely on a browser - but I may be wrong

-----

Robots.txt (Ciao Jaclaz, grazie!) would be a very credible explanation. Thanks for the link at Wiki!
I had already tried to ask the administrator, who avoided giving a clear explanation.
Fortunately, his robots.txt is accessible from Firefox, I have it under my eyes.

It contains, among others:
Sitemap: coughcough/sitemap_index.xml
Disallow: /
Do I understand properly that sitemap_index.xml then gives the exclusive list of what the bots shall index?

The sitemap_index.xml is accessible and contains for instance
<sitemapindex etc>
<sitemap>
<loc>CoughCough/sitemap/0.xml</loc>
<lastmod>2011-03-05T12:00:00+00:00</lastmod>
</sitemap>

And the just mentioned 0.xml contains
<url>
<loc>CoughCough/viewforum.php?f=66</loc>
<lastmod>2010-11-07T17:28:04+00:00</lastmod>
<changefreq>daily</changefreq>
<priority>1</priority>
</url>

where viewforum.php?f=66 is the section "Technology" that I'm most interested in, which search engines don't list.

What would bots be supposed to do when reading this?

Thanks!

[jaclaz]

pointertovoid, on 08 March 2011 - 08:49 PM, said:

What would bots be supposed to do when reading this?

Most probably (but not really my field of knowledge):
Get to the "Sitemap" line.
From it get to sitemap_index.xml.
From it get to 0.xml.
Read the line "viewforum.php?f=66" and LEAVE FORUM #66 alone since it is disallowed as well as the whole site....
http://en.wikipedia.org/wiki/Sitemaps

But it could also be the other way round, like that the Forum Admin meant to have that forum crawled, but that *something* prevents the robots to follow the chain and the result is involuntarily to "disallow" it specifically.

The "Disallow /" should prevent ANY robot from crawling ANY page, from what I understand, no matter what the Sitemap says....


jaclaz

This post has been edited by jaclaz: 09 March 2011 - 04:02 AM

[Tripredacus]

pointertovoid, on 07 March 2011 - 10:18 AM, said:

Tripredacus, on 07 March 2011 - 10:08 AM, said:

I have removed all links and name to your forum.

Why? They were useful to readers willing to check Google's behaviour.

By the way, it's Sapo's forum, not mine. I'm a member there, no other relation.

Having all those links, so you could see that google results because they were on MSFN is too close to SEO spamming behaviour which is against the forum rules. So I either remove the links or you could be banned for spamming.

So I removed the links so you can get help with the issue. Besides, they weren't specifically necessary to help with the question.