How does Pinterest get around this then? They pollute image searches like crazy, and require you to login to see anything. At least they did, I blocked them from searches so maybe it’s different now.
Ever heard of https://12ft.io/ ?
It allows you to bypass alot of pay walls by basically pretending to be a search engine trying to index a website. For SEO reasons a lot of pay walled sites allow search engines to access the whole article to index.
12ft.io leverages this to show you whole articles behind paywalls.
This is something you could also achieve by spoofing the User-Agent.
It would probably work for things like Pinterest without an account as well, but that’s something I have never tried (since I have no interest in the cancer that is Pinterest).
it 100% did, google removed over half the twitter links on its index due to dead links/login requirements, which if kept like that would basically kill all Twitter traffic since most traffic comes from search engines
How does Pinterest get around this then? They pollute image searches like crazy, and require you to login to see anything. At least they did, I blocked them from searches so maybe it’s different now.
Pinterest is cancer. They act like their content belongs to them when it’s all stolen images
Most of these sites serve the information, then put up something to block being able to view it.
Easy - detect if you’re getting accessed by a search crawler or a human. Serve a full page or just a login request.
So how can a user pretend to be a web crawler?
You’re going to need a special hat.
deleted by creator
Adding a clipboard and a ladder will make it even more official
This browser addon can spoof useragents:https://add0n.com/useragent-switcher.html
Ever heard of https://12ft.io/ ? It allows you to bypass alot of pay walls by basically pretending to be a search engine trying to index a website. For SEO reasons a lot of pay walled sites allow search engines to access the whole article to index. 12ft.io leverages this to show you whole articles behind paywalls. This is something you could also achieve by spoofing the User-Agent. It would probably work for things like Pinterest without an account as well, but that’s something I have never tried (since I have no interest in the cancer that is Pinterest).
They must have changed their paywall behavior, I just went and was able to see every image I clicked on.
The login popup appears after a few pages but you can just exit out and keep viewing. Google should be able to index the pages without access issues
Maybe that previous aggressive login screen killed their SEO before, I see much less pinterest images than I used to years ago
it 100% did, google removed over half the twitter links on its index due to dead links/login requirements, which if kept like that would basically kill all Twitter traffic since most traffic comes from search engines