Chromium Spelunking: Churl

At the end of my last post I indicated I would try to build a simple executable to fetch a URL, similar to curl. churl sounds like a suitably silly name for it.

This post is going to be a little on the long side -- it's leaning into the "lightly edited lab notebook" format, in hopes of highlighting some of the places I got stuck.

An Executable

I want to focus on my goals here -- understanding the network stack, specifically proxies and QUIC -- so I don't want to get distracted with things like how to set up Ninja to create a new executable. So, I went looking for something that was near to what I wanted and which I could copy/paste and modify. I got lucky! There is a quic_client executable that seems quite close, really. However, it's specific to QUIC and I want to make something that begins at URLRequest. Still, this gives me an easy way to create a new executable in net/BUILD.gn and do a simple "Hello World":

int main(int argc, char* argv[]) {
  DLOG(ERROR) << "Hello, world.";                                                                                                                                                                                  
}

Getting to URLRequestContext

From the previous post, step one is going to be getting a URLRequestContext up and running, and that requires a builder.

ThreadPoolInstance

Naively trying to create such a thing gets me an error because I haven't initialized a thread pool. The network stack does seem to use a lot of the task-posting magic, so I suppose that's necessary! A bit of guessing based on base/test/task_environment.cc leads me to

base::ThreadPoolInstance::Set(std::make_unique<base::internal::ThreadPoolImpl>("my histogram"));

and on to the next error. I am hoping to only have to make two or three of these "wild guesses" before things start working. Beyond that point, when something doesn't work it will be almost impossible for me to figure out which of the n>3 things I do not understand might be malfunctioning.

SequencedTaskRunner

Next up, I need a SequencedTaskRunner to run tasks on that thread pool:

[0508/164407.189292:FATAL:post_task_and_reply_impl.cc(158)] Check failed: has_sequenced_context || !post_task_success. 

Checking sequenced_task_runner.h shows a lot of docs about the type, but not how to construct it. The bottom of the doc comments mentions some "theoretical implementations", suggesting that this is an abstract base class, and in fact a few methods are = 0. So, where are the implementations? Ordinarily I'd use CodeSearch for this, but it seems that the 91 subclasses are more than its little brain can handle.

I continued on a little down this path, but I am learning that when I find myself reading the Chromium source, I'm looking in the wrong place. In fact, the nice Threading and Tasks document includes snippets that create task runners:

auto thread_runner = base::ThreadPool::CreateTaskRunner({base::TaskPriority::USER_VISIBLE});

The same error occurs, I think because while this creates a new task runner, it does not set it as the default. It looks like CreateDefaultHandle is an RAII wrapper for setting a task runner as the default, so let's try

base::SequencedTaskRunner::CurrentDefaultHandle active_thread_pool(thread_runner);

This call itself fails:

[0508/171849.025900:FATAL:sequenced_task_runner.cc(99)] Check failed: task_runner_->RunsTasksInCurrentSequence(). 

The comment docs for RunsTasksInCurrentSequence suggest that it is meant to be called within a task, so maybe I need to run the remainder of churl in a task in this runner?

void Task() {
  net::URLRequestContextBuilder url_request_context_builder;
  url_request_context_builder.set_user_agent("Dustin's Experiment");

  auto url_request_context = url_request_context_builder.Build();

  DLOG(ERROR) << "Hello, world." << url_request_context;
}

int main(int argc, char* argv[]) {
  base::ThreadPoolInstance::Set(std::make_unique<base::internal::ThreadPoolImpl>("my histogram"));

  auto thread_runner = base::ThreadPool::CreateSequencedTaskRunner({base::TaskPriority::USER_VISIBLE});
  thread_runner->PostTask(FROM_HERE, base::BindOnce(Task));
}

Indeed, that compiles, but since main exits immediately after posting the task, it doesn't print "Hello, world."

Referring to the Threading and Tasks document again, I tried base::RunLoop().RunUntilIdle(), but this seems to require that it be called with the current loop set, and that seems a bit circular. Lower down I see base::RunLoop::Run, which will call until a QuitClosure is run. That sounds better, but ultimately has the same issue:

[0508/173608.004631:FATAL:single_thread_task_runner.cc(44)] Check failed: handle. Error: This caller requires a single-threaded context (i.e. the current task needs to run from a SingleThreadTaskRunner). If you're in a test refer to //docs/threading_and_tasks_testing.md.

Further down the document:

// To block until all tasks posted to thread pool are done running:
base::ThreadPoolInstance::Get()->FlushForTesting();

But that just hangs without actually running the task. Even further down the document, there's a section entitled "Using ThreadPool in a New Process", and it turns out that this is what I'm trying to do. It uses

base::ThreadPoolInstance::CreateAndStartWithDefaultParams("process_name");

instead of the initialization I was using. I don't really understand the difference, but this appears to make things work and gets the next error from net::URLRequestContextBuilder::Build:

[0508/174429.973862:FATAL:configured_proxy_resolution_service.cc(800)] Check failed: proxy_config_service. 

Proxy Config Service / Proxy Resolution Service

Proxies! At least that's relevant to my interests.

Following the traceback for that failure:

#6 0x7f00db30f76f logging::CheckError::~CheckError()
#7 0x7f00dd28f8f7 net::ConfiguredProxyResolutionService::CreateUsingSystemProxyResolver()
#8 0x7f00dd63b1b7 net::URLRequestContextBuilder::CreateProxyResolutionService()
#9 0x7f00dd637485 net::URLRequestContextBuilder::Build()

it looks like Build is passing a NULL unique_ptr for proxy_config_service to CreateProxyResolutionService, and adding some debug prints confirms that. In fact, the debug prints confirm that BUILDFLAG(IS_LINUX) is not true, despite building this on a Linux system. I don't know a quick way to fix that, so hopefully I can work around it.

Commenting out the build conditionals in url_request_context_builder.cc gets a different error, I think to do with ProxyConfigService::CreateSystemProxyConfigService requiring a SingleThreadTaskRunner when we're using a SequencedTaskRunner, so maybe that's a dead-end.

Maybe I can provide a stubbed-out implementation of these types? It looks like ProxyConfigService can create a ProxyResolutionService, or I can just specify a ProxyResolutionService directly. Looking for subclasses of both abstract base classes, I found a local ProxyConfigServiceDirect which just always returns DIRECT. That looks useful, so I'll copy-paste it (and add the necessary namespaces):

// Config getter that always returns direct settings.
class ProxyConfigServiceDirect : public net::ProxyConfigService {
 public:
  // ProxyConfigService implementation:
  void AddObserver(Observer* observer) override {}
  void RemoveObserver(Observer* observer) override {}
  net::ProxyConfigService::ConfigAvailability GetLatestProxyConfig(
      net::ProxyConfigWithAnnotation* config) override {
    *config = net::ProxyConfigWithAnnotation::CreateDirect();
    return CONFIG_VALID;
  }
};

void Task() {
  net::URLRequestContextBuilder url_request_context_builder;
  url_request_context_builder.set_proxy_config_service(
    std::make_unique<ProxyConfigServiceDirect>());
  // ..
}

and with that, I've successfully built a UrlRequestContext!

Checking In

OK, so I've made some progress today. A lot of this has been trial-and-error. There are two problems with this approach. First, it means I don't really learn how things work -- for example, I don't know why so many options for setting up a task runner failed. But I can add an item to my task list to learn more about that some day, and carry on with my current project.

Second, the distinction between "OK" and "Error" in this kind of trial-and-error is not always clear. In this case, I thought I had finished the creation of the thread pool successfully, but it turned out I had to revisit this and do it a different way. That was down to luck, and with a few more repetitions of trial-and-error, the universe of unknown things that could be wrong is so large that no amount of luck is enough to make progress.

The tools also let me down a bit here: CodeSearch was overwhelmed and unable to give me answers. I omitted it above, but I fell back to git grep. Like a pocket-knife, it's a tool that isn't fancy but is always there and always works. Another go-to tool that I used here was debug prints (in this case, DLOG(ERROR) << "HERE 1"; and incrementing the number). It's not especially cool, but it provides reliable, ground-truth information.

The code itself let me down a little, too. Classes in base seem to be well-commented, but with deep technical detail and not answers to questions like "how do you create an instance". Maybe that's OK -- it's a lot to ask from code comments -- but it's a reminder to me that I need to look outside of the code for higher-level documentation.