Yes, it’s been on my roadmap for a while. I also created a pull request several months ago to enter the repo but it was never accepted (it’s also my fault because I didn’t follow the verification process properly).
It amazes me that all it takes is just changing user agent to Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) and it can bypass paywalls on many sites? I thought those sites would try harder (e.g. checking if the ip address is truly belong to google), but apparently not.
Same. I thought there would be more stuff happening in the background but when I saw it’s just hijacking the google bot headers to display the html i was a bit disappointed it’s so stupidly easy.
Checking ip ownership is a moving target more likely to result in outcomes these sites don’t want (accidentally blocking google bots and preventing results from appearing on google).
Checking useragent is cheap, easier, unlikely to break (for this purpose, anyway) and the percentage of folks who know how to bypass this check is relatively slim, with a pretty small financial impact.
Google literally has an official list of IP ranges for their crawlers, complete with an API that returns the current IP ranges that you can use to automate a check. Hardly a moving target, and even if it is, it doesn’t matter if you know exactly where the target is at all times.
If you’re on Firefox on desktop/laptop, check out Bypass Paywall [0]. It was removed from the firefox add-on store due to a DMCA claim [1], but can be manually installed (and auto updates) from gitlab. The dev even provides instructions on how to add custom filters to uBlock Origin [2], so you don’t have to add another extension but still get some benefit.
If you’re on Android and use Firefox, you can use the Disable JavaScript extension to disable JS on sites with paywalls, like NYtimes. While not perfect, it works remarkably well.
That has nothing to do with the topic, you’re right, I could’ve put it more nicely. I’m just tired of “compile it yourself” to people without computers being considered an appropriate answer for a criticism of bad distribution. This app seems wonderfully useful and could have huge popularity, but it’s being artificially gatekept by the creator.
github.com
Hot