The same day we connected our ChatGPT chatbots to the internet, OpenAI disconnected ChatGPT from browsing the web. OpenAI's explanation was that, and I quote "their web browsing feature allowed for users to circumvent payment walls".
This can be difficult to understand for non-technical users of ChatGPT, so I will try to explain it as simple as I can. However to understand, we must look at how most media outlet payment walls works, and why they work like this.
How most payment walls work
Most payment walls aren't really payment walls, they're simple "HTML tricks" to hide the text for users not having paid. If you don't know what a payment wall is, here's an example from Wall Street Journal.
Notice, I don't mean to pick on WSJ in particular here, the entire industry are using the same cheap click and bait tricks here, and due to traditional media desertification having happened the last couple of decades because of social media and the internet - You can kind of argue they're just doing what they have to do to survive.
However, the strategy they're using with these payment walls are unethical, immoral, and arguably in violation of Google's internally developed rules for SEO and content production. If the above company had any other name but "Wall Street Journal", they would have been blocked out of Google's search index, since it's a black hat SEO technique often referred to as "cloaking".
Cloaking implies showing one thing to the search engine, and another thing to the end user
The point is that the whole article actually is there, it is just "hidden" with HTML trickery ...
WHY?
The natural question then becomes why these companies are doing this. The reason is easily understood, they want to rank high for anything related to whatever the article's content is, while using the article in Google to "click bait" users into paying for a subscription based service. Like I said ...
This is unethical, immoral, and in violation of everything we find to be "decent human behavior"
When Google is scraping the above article, Google sees the whole article, and will use the content of the article as they're showing search results to their end users. When the end user clicks the article, he needs to pay to read it. This is flat out immoral business practice, and that OpenAI shuts off one of their primary features to accommodate for this, is quite frankly absurd. Paradoxically, we win as usual, so I shouldn't really complain, since we're getting tons of traffic from users looking for alternatives allowing them to connect ChatGPT to the web.
FYI - I couldn't imagine a gun big enough in the world to be able to coerce me into turning off this feature in our chatbots! If Wall Street Journal is of the opinion that what we're doing is "unethical", I will meet them in any court, any place on the planet, and defend my case - And highly likely win!
The alternative
The alternatives for Wall Street Journal, and other news media outlets, is to create a real payment wall, where parts of their content truly is behind a payment wall, and not just fancy HTML trickery - For then to produce high quality content, investigative journalism, such as the article you're reading now, to such attract readers into paying for more content. Today they're just an echo chamber, an extention of our governments, doing whatever they can to please those with powers, having failed their obligations towards society at large - Something clearly seen by how they've treated Julian Assange the last decade.
For you as a user there's also an alternative, our ChatGPT AI chatbots. These chatbots are connected to the internet and can search using DuckDuckGo. They will scrape whatever HTML they find, and use it in combination with ChatGPT to answer your question - Implying it will bypass WSJ's "cloaking trickery".
Conclusion
For ancient dinosaur companies and entities such as WSJ, New York Times, The Chronicle, and God knows who else to be able to dictate innovation, and the future of mankind, is simply absurd - And we will not have it. Either change your business practices, stay relevant, and survive - Or go die somewhere peacefully, like the dinosaurs did. You served us well for 250 years, but today you're prohibiting innovation, corrupt through to the bone, and there are no justifications for your existence what so ever. If you want me to prove it to you, I can prove it with two words ...
Julian Assange
Edit - We created an alternative ChatGPT search engine ourselves ... ;)
Top comments (29)
I applaud your technical explanation regarding paywalls, but I respectfully disagree with your perspective as to the reasoning for the feature. There's a lot of bias in your explanation and simplifies the aggregate of the publisher community to a singular focus of gamifying a search engine without mentioning more practical reasons like my shareholders expect profitability or I need to increase revenue to maintain operations. It also doesn't raise the possibility that Google allows an exception to their rules because of their own motives.
Machines inherently will consume more information than what we as a human sees, the fact that publishers are exposing this information to a search engine but not the user is not about ethics or morals, rather logic. The fact that you're being denied access is because you're a new use-case, and existing solutions to problems weren't designed with your considerations in mind. Businesses and technology is still adapting.
As a final point, I want to emphasize that the internet is not free as in speech, it is free as in beer. While yes, the internet does offer avenues for unlimited expression, it requires infrastructure based in a capitalistic society. It costs money to run servers and have an internet connection. Have you any context as to the cost burden forced on publishers to support models accessing its content? A significant uptick in traffic for internet resources already at scale is not a trivial expense increase.
You got speech and beer around the Wong way lad. It's not free beer, but it is free speech 😉. Free does not mean freedom from consequences, but [unfair] persecution [by state entities].
Paywalling content where the scripts to paywall can be removed (hello Firefox reader mode) is foolish. Doing so to lower cost shows a fundamental set of mental gymnastics that is not conducive to cost reduction. Which by the way public unfettered access is also a barrier to.
Amen!! 😊
I assure you, I do not have this backwards.
Not realizing this simply tells me you don’t actually understand the economics of serving resources on the internet.
This comment is the correct answer.
The article author apparently believes everything should be free. That the people who produce the content he wants to consume are in an invalid profession. They are unethical to ask to be paid for their efforts, he says! They are dinosaurs, he says! Yet he still wants to consume what they produce....just for free. Hmm. I wonder, what do you do for a living? Are you willing to do it for free?
Speaking of unethical dinosaur business models, I was forced to sign up for a Dev.to account just to leave this comment. Oh, it was "free", but I had to agree that they have access to all my tweets and other content (I used Twitter to sign up, God knows what they would have access to with the other methods!). So just to leave my opinion here, I was forced to trade away my personal data. Why?? Why put the ability to share my freedom of expression behind a personal data paywall??
I'm afraid this whole article smacks of hypocrisy.
I rest my case, but I'd like to finish with a "thank you for increasing the engagement of my article" ... ;)
Respectfully, but if ...
... is the only reason you're in business, you've got a holy duty to quit your job, and find another living!
Traditional media haven't adapted an inch since 1995, at least not in a positive direction ...
Agree, and this is largely traditional media's and social media's fault, and it is not a good thing ...
Because their tech is garbage, and they're downloading GBs of rubbish, where they could have optimised their content and servers, and ended up with a slick streamlined experience, not wasting hundreds of MB on "garbage" ...
You have some points, you are correct, however the moral implications of choosing your side could be used as a justification for Ted Bundy in its most extreme case ...
Ethics and technology are not two separate distinct concepts, they're intertwined, just like everything else in this world - And traditional media (and social media for that matter) are pure evil constructions, based upon tricking the masses into believing in whatever rubbish the've got some sort of stake in spreading, due to having been paid to spread it ...
Okay, first of all, I love the way you explained it! I do agree with what you said, but I think it's not right to think that they removed web browsing just because of a particular reason. There are a various factors that concluded to this decision, but yeah, one of the biggest was people were abusing this web integration in payment gateways!
Actually, this was what they gave themselves as their reasons. Then I realised most don't understand how payment walls works, and/or what it means, so I felt the urge to explain it. But their own statement is as follows;
Thx for the nice words :)
Sadly... All it takes to bypass WSJ Paywall is:
javascript:url=window.location.href;window.location=url.substring(0,url.indexOf('/articles'))+'/amp'+url.substring(url.indexOf('/articles'));
Just paste it in the address bar and wallah. 🙏
Why am I not surprised ... :/
While cloaking is technically implemented the same way, Google themselves provides you with directions on how to avoid getting blocked:
developers.google.com/search/docs/...
Cloaking generally refers to the idea of presenting different content to search engines - that's subtly different from merely omitting content from responses to public clients.
Yes, it's immoral, and the news internet these days is mostly paywalled.
Just add 12ft.io in front of every paywalled URL and show them who's boss. 😎
Hehe :D
Love that this was writen largely by AI itself. I 100% agree. We need new frontiers, new frontiers give us new opportunities - the previous frontier will have been saturated with exploitation of its users, new frontiers give us the opportunity to break free from that and for a short time, live with a sense of freedom.
We do produce about 15+ articles each day using AI, but this was only me 😉
oracle.ainiro.io
I thought that was a feature. It's been showing me stuff behind paywalls since before they rolled out Internet. Old articles and research papers are behind paywalls too. I've used it for school to help research and to do citations. It sucks they are, "fixing" it.
I honestly don't understand why they're fixing it. There are already high court verdicts in the US proclaiming quote "AI generated content is not possible to copyright" and it's already been established that "re-mixing content to create new content is not a copyright violating as long as no copying occurs" - And, it's the same thing we've done for centuries when citing each other. The only difference is that now there's a machine citing us ...
Copyright implies just that, "the right to copy". No copying, no copyright. The AI doesn't copy, it writes its own unique content, based upon existing content it has consumed as it was learning. If this is illegal, you cannot use any of your knowledge, since arguably every piece of knowledge you have, originates from a copyrighted book somewhere ...
This highlights the need for more transparency in search results. Users deserve to know whether an article is behind a paywall before clicking on it. OpenAI's adaptation to this situation shows their commitment to user experience when using chatgpt. Please continue to uphold!
You're correct, it is "click baiting", but OpenAI didn't remove it because of the reason you state, they removed it because of publishers complaining about "OpenAI stealing their articles".
If it's publicly available for scrapers, it should be publicly available for humans - Everything else is ipso facto click baiting ...
That Google even allows for this is quite frankly beyond my comprehension. It degrades the user experience a lot ...
Hear here!!! I respect and agree with do much of what you say here. Especially in regards to pay walls they disgust me. It was cool to learn more about them here though it just exacerbated how much I hate them.
Thx Pia, I don't mind people charging, but the bait and click parts of the way these guys are implemented is not particularly ethically I think ...
Interesting viewpoints! Disappointing that OpenAI removed web browsing, but it's not like the only option you've got to search the web with ChatGPT was through OpenAI ;)
Hehehe, nope :D
Not sure I like the framing of chatgpt as such an essential for-good, but aren't there plugins that bypass, which are trivial to author anyway?
No plugins. We're on our own. I'm not framing ChatGPT as essential, I'm framing the web as such 😉
Some comments may only be visible to logged-in visitors. Sign in to view all comments.