DEV Community

Cover image for Revived the promise made six years ago for Requests 3
TAHRI Ahmed R.
TAHRI Ahmed R.

Posted on

Revived the promise made six years ago for Requests 3

Requests is undeniably unmissable for anyone willing to start in the Python ecosystem. Yet the community is broadly unaware of how sad the situation has become.

The incident

Due to dramatics circumstances, it was partially locked and put in "maintenance mode" only, meaning
that it would no longer evolve (features speaking).

Some of you may have eared, more than six years ago that Requests was working toward a major version, namely the V3.
There was even a completed fundraiser with a substantial amount, around $30k.

At the time, there was a plan for bringing Async, and HTTP/2 support (most notably). You'll guess that none of exposed promises ever landed.

Impacts

This incident led to several impacts.

Let's be clear, this article is not about saying "Requests maintainers are bad" but rather saying we don't agree, and everyone is entitled to his choices.

For many years now, Requests has been frozen. Being left in a vegetative state and not evolving, this blocked millions of developers from using more advanced features.

The Journey into Waking Up

Initially, like many other reasonable person, I said to myself "Let's wait, Requests will eventually unblock the situation!"
Yet the years are still passing and nothing.

Let's realize something, Internet Explorer 11 just celebrated its tenth anniversary last year, and you’ll never guess that it
has support for HTTP/2 integrated!
Yet Requests is unable to do so.

I said to myself: "Why are we waiting for this indefinitely? They don't owe us anything! There's a reasonable chance we are waiting... trapped in a circular reference.. Let's act."

Then, I tried to get a firm grip on urllib3 base code, contributing this and there until I was ready to kick things up with a
proof of concept
that would have put urllib3 far ahead. Without any breaking changes.
I was delusional. This was a bit of a shock, but six months passed between my initial kick off and my formal give up, and here's why in a nutshell:

  • Every bits of idea brought to the table was dismissed, and never seriously analysed.

This felt as good as taking the coldest wind on your face while exiting your warm home. It's the hard truth of the OSS,
it's not a democracy and it's not ruled by pure logic, but humans.

No far latter on, urllib3 called out a fundraiser to help with HTTP/2 support (only) and claimed around $40k to be able to deliver. They probably meant well, but[...]

This was the last straw for me, knowing the past mistakes, and actually the best incentive to break the cycle. Everything point to another year before having the slightest sign of a PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n into our production pipes. They clearly lack the time to move forward (or incentives? or both?). No worries, as developers, we should share this burden for the greater good.

From my perspective (my opinion only), the OSS ecosystem should not fund raise money for a decade old protocol, and most of time should deliver prior to asking. Entities raising money with no solid proof of concept, that's rare!

This felt like a bit of a hostage situation, but it could only be me seeing it this way.

The Solution

So many years and various expertise have been brought to Requests and urllib3.. Do we throw all of it into the garbage disposal? Engineers should be reasonable scientists and not zeled revisionists.

We don't have to reinvent the wheel all over again, HTTP client Requests is well established and really pleasant in its usage.
We believe that Requests has the most inclusive and developer friendly interfaces.

So how to propose an attractive continuity for Requests? How hard could this be to (i) bring a true upgrade experience and (ii) keeping the compatibility as high as possible?

Sometime the OSS ecosystem act like it is trying to land a rover on the Moon, let's be more realistic and bring our vision down to firm grounds.

There's a chance offered to us to rehabilitate Requests (and urllib3) and give a fresh breath of air to us, the developers. Without having to learn something as basic as handling a HTTP client.

So, the work started, without looking back.

Just around my intent to give up on urllib3 evolution, we proposed a fork of both urllib3 and Requests, namely urllib3-future and Niquests.

They are (truly) drop-in replacements, at the time of writing, those forks are six-months old! And the whole work around this is roughly a year old. We listened to what the community needed by... triaging all the issues on major repositories (Requests, urllib3, and dependants of them) so that the upgrade would be truly useful.

Are you seeing it coming? Great! Let's look at the feature table comparison against requests, httpx and aiohttp!

Feature niquests requests httpx aiohttp
HTTP/1.1
HTTP/2
HTTP/3 over QUIC
Synchronous
Asynchronous
Thread Safe N/A
Task Safe N/A
OS Trust Store
Multiplexing Limited
DNSSEC
Customizable DNS Resolution
DNS over HTTPS
DNS over QUIC
DNS over TLS
Multiple DNS Resolver
Network Fine Tuning & Inspect Limited Limited
Certificate Revocation Protection
Session Persistence
In-memory Certificate CA & mTLS Limited Limited
SOCKS 4/5 Proxies
HTTP/HTTPS Proxies
TLS-in-TLS Support
Direct HTTP/3 Negotiation N/A N/A N/A
Happy Eyeballs
Package / SLSA Signed

And you would think... performance..? are bad?

Look at the performance comparison against them!

Scenario: Fetch a thousand requests using 10 tasks or threads, each with a hundred requests using a single pool of 10 connection.

High-Level APIs

Client Average Delay to Complete Notes
requests 987 ms ThreadPoolExecutor. HTTP/1.1
httpx 735 ms Asyncio. HTTP/2
niquests 470 ms Asyncio. HTTP/2

Simplified APIs

Client Average Delay to Complete Notes
requests core 643 ms ThreadPoolExecutor. HTTP/1.1
httpx core 550 ms Asyncio. HTTP/2
aiohttp 220 ms Asyncio. HTTP/1.1
niquests core 210 ms Asyncio. HTTP/2

Did you give up on HTTP/2 due to performance concerns? Think again! Multiplexing and response laziness open up a wide range
of possibilities!

And everything you are witnessing is in fact "Requests"-extended and you have nothing to learn again to get started. We kept 98% of the test cases from both Requests and urllib3, then extended them to cover newer features.

We are looking forward to see you at the Niquests repository! Let's say it's a promotional offer, get everything and more for $̶4̶0̶k̶ $0!

  • #1 You depends on Requests? Discover Niquests instead.
    • Drop-in replacement, simple patch needed, tiny effort required.
  • #2 You depends on urllib3? Discover urllib3.future instead.
    • Drop-in replacement, no patch required, atom-like sized effort required.

By visiting the GitHub repository and documentation you will be able to learn more about this, and hopefully make a decisive switch!

Top comments (0)