patternMinor
Allowing googlebot to crawl a password protected page
Viewed 0 times
allowingprotectedcrawlpasswordpagegooglebot
Problem
The below I have put together as an experiment, I have profiles on my site that are password protected. I would still like them to be crawled by google.
The script is taken from here and basically does a reverse dns lookup to verify if a googlebot is real or fake. I am then using this logic to allow google entry to my dynamic page by skipping the 'restrict-access.php' include which checks to see if someone is logged in or not.
My question is..
The script is taken from here and basically does a reverse dns lookup to verify if a googlebot is real or fake. I am then using this logic to allow google entry to my dynamic page by skipping the 'restrict-access.php' include which checks to see if someone is logged in or not.
My question is..
- Is this viable? 2. is the code solid/safe?
Solution
Is this viable
Of course it is viable but as I am pretty sure somehow everyone can fake a googlebot without a detection by your script I don't recommend it. Another aspect is mentioned by 200_success. Google policies dislike such behaviour and it is highly recommend to follow their recommendations. At the end of the answer is a sample of my idea of implementation
Is this code solid/safe
General
It looks like you are developing procedural which is not recommend. OOP (object orientend programming) is the better way to go as it is a modular approach of developing software. In oop there are classes and each has specific task to solve. Wikipedia - Object-oriented-programming
Closing php tag
In the embedded source code you close the php tag which is not recommend. It might happen that a space is after a closing php tag and if you receive a
My idea of implementation
Independent of the google policies and that only x of y details should be indexed I would not make a google check but display the x of y details always and the z of y only if a function (e.g.
Sample
Such an implementation allows other search engines to crawl the x of y details while the z of y details are hidden for every search engine. You don't have to implement a googlebot verfication either.
Of course this sample has to be extended.
Of course it is viable but as I am pretty sure somehow everyone can fake a googlebot without a detection by your script I don't recommend it. Another aspect is mentioned by 200_success. Google policies dislike such behaviour and it is highly recommend to follow their recommendations. At the end of the answer is a sample of my idea of implementation
Is this code solid/safe
General
It looks like you are developing procedural which is not recommend. OOP (object orientend programming) is the better way to go as it is a modular approach of developing software. In oop there are classes and each has specific task to solve. Wikipedia - Object-oriented-programming
Closing php tag
In the embedded source code you close the php tag which is not recommend. It might happen that a space is after a closing php tag and if you receive a
headers already sent error it can be time consuming to identifiy the file where a space is after a closing php tag.My idea of implementation
Independent of the google policies and that only x of y details should be indexed I would not make a google check but display the x of y details always and the z of y only if a function (e.g.
userHasAccess($userid); ) returns true.Sample
Detail
User: John Doe
A description about John Doe.
Age: 18";
}
?>
Such an implementation allows other search engines to crawl the x of y details while the z of y details are hidden for every search engine. You don't have to implement a googlebot verfication either.
Of course this sample has to be extended.
Code Snippets
<!DOCTYPE HTML>
<html lang="en">
<head>
<title>Detail</title>
<meta name="robots" content="index">
</head>
<body>
<h2>User: John Doe</h2>
<p>
A description about John Doe.
</p>
<?php
if (userHasAccess(2))
{
echo "<p>Age: 18</p>";
}
?>
</body>
</html>Context
StackExchange Code Review Q#85892, answer score: 3
Revisions (0)
No revisions yet.