Chromium Spelunking: Threads and Tasks

In the last post, I "solved" a problem with incorrect usage of task runners, but I still don't feel like I understand how these things work. I happen to have this simple little binary with few moving pieces, so this seemed like a good time to try to learn a bit more.

The Question

I have the following code, which is working fine (meaning it crashes because there's no delegate):

base::ThreadPoolInstance::CreateAndStartWithDefaultParams("churl");
auto task_runner = base::ThreadPool::CreateSingleThreadTaskRunner({base::TaskPriority::USER_VISIBLE});
task_runner->PostTask(FROM_HERE, base::BindOnce(Task));

The "Threading and Tasks" document says

A task that can run on any thread and doesn’t have ordering or mutual exclusion requirements with other tasks should be posted using one of the base::ThreadPool::PostTask*() functions.

My Task() is such a task, so I tried changing the last line to

base::ThreadPool::PostTask(FROM_HERE, base::BindOnce(Task));

This fails. When run directly, it fails with a SIGSEGV from a NULL pointer, but when run under gdb it fails with the familiar Check failed: has_sequenced_context || !post_task_success. I don't know why this doesn't appear when run directly -- maybe the SIGSEGV occurs while trying to print the message? But I'll let that mystery remain.

Deep Background

So, I've misunderstood something here. Perhaps it's time to just start gathering information in hopes I can piece together the big picture. I did this in the first post in the series, but this time I won't include all of my findings inline.

However, my process may be interesting. I began with another re-read of "Threading and Tasks", this time jotting down a few notes for each section but more importantly keeping a running list of questions. As I came across an answer to a question, I'd include that in the notes and cross off the question. I find that this helps me to focus my thinking a little bit: when I have a question, I can write it down and set it aside; and when I find new information, I can compare it to the list of questions in case it answers one or two.

It also means that, after working for a while, any remaining questions on the list are good questions to ask someone with more expertise.

Here's a partial list of questions I jotted down:

• Why does base::ThreadPool::PostTask fail?
• What is a "Sequence" and how is it different from a "Virtual Thread"?
• Can there be multiple task queues per thread?
• How are tasks from multiple queues in a thread handled?
• When a function takes a callback or delegates, what runner/queue does it use to call those?
• How do the PostTask..AndReply methods work?
• If there is one SequenceManager per thread, how do thread pools work? Are there multiple SequenceManagers all competing for tasks from the same TaskQueue?
• How do the "current defaults" work for SequencedTaskRunner and SingleThreadTaskRunner?

Once I finished reading the document, I started looking at the code itself. I've learned that, in general, this is the slowest way to learn things about Chromium: most source files are sparsely commented and not ordered in a way that helps a newcomer to understand them. I suspect IDEs make this worse, especially in .cc files, as they happily index the functions in a file regardless of the order they appear. But there are usually a few useful tidbits of information to be found. I used code search to find the declarations of various classes I'd read about, spending more time on those that had more to offer.

In the end, I had a few questions left, and I took those to Gabriel Charette (gab@) [1]. His answers prompted some more reading and more questions (and a link to some non-public resources), but helped me make progress quickly! Asking the right person the right questions is an effective way to navigate a low-information environment, but it is difficult to do well. I'm working on it!

Answers

Why does base::ThreadPool::PostTask fail?

I'll answer this at the end.

What is a "Sequence" and how is it different from a "Virtual Thread"?

They are different names for the same thing. A virtual thread is conceptually a sequence of tasks executed one after the other, with the effects of one task being visible to the next. Tasks in a sequence may execute on separate threads. A Sequence implements this, by managing a TaskQueue (and a heap of delayed tasks). More on this below.

Can there be multiple task queues per thread?

Yes. Leading to the followup question:

How are tasks from multiple queues in a thread handled?

SequenceManager handles selecting tasks from those multiple TaskQueues using a TaskQueueSelector.

SequenceManager's doc comment suggests that it multiplexes tasks "into a single backing sequence", but this does not appear to be the case. Instead, ThreadControllers call SequenceManager::SelectNextTask (an override of a method from SequencedTaskSource). That returns the best task to run next, and that task gets run.

When a function takes a callback or delegates, what runner/queue does it use to call those?

This question was sort of off-topic. It's usually written somewhere in the docs for the class, and usually one of the "named threads" like the UI thread or IO thread.

How do the PostTask..AndReply methods work?

• PostTask - just run a task (on the task runner associated with the method receiver).
• PostDelayedTask - similar to PostTask, but running the task after a delay. This adds a lot of complexity to the implementation, but is pretty simple to use.
• PostTaskAndReply - post a task in the runner associated with the method receiver, and when that completes post a second task in the sequence that called PostTaskAndReply. This provides a way to do work on a background thread and then resume the current sequence when it is done.
• PostTaskAndReplyWithResult - like PostTaskAndReply but pass return value of first task callback as an argument to the second callback. This provides an even more RPC-like call out to a background thread.

All of these imply some kind of TaskRunner (plain, sequenced, or single-threaded). Task runners are just the gateway to send tasks into the scheduler, and the kind of task runner dictates the execution mode for the task.

If there is one SequenceManager per thread, how do thread pools work? Are there multiple SequenceManagers all competing for tasks from the same TaskQueue?

There's quite a bit of complicated work done to support both prioritizing lots of tasks in a single (named) thread and distributing work evenly in threads pools. SequenceManager is only used for named threads.

One of the key abstractions that bridges both named threads and thread pools is Sequence. This contains a queue of tasks that must be executed sequentially.

In a single thread, lots of Sequences can exist at the same time, and SequenceManager takes care of treating them all fairly.

In a thread pool, Sequences are kept in a priority queue shared among all the workers, and popped from that queue one by one. Popping sequences, rather than tasks, from the queue supports the robust guarantee that the tasks in a sequence are executed sequentially. Still, a worker thread only executes one task from a sequence, re-queuing the sequence if it's not empty.

How do the "current defaults" work for SequencedTaskRunner and SingleThreadTaskRunner?

In a named thread, these are fixed and are just defaults for the SequenceManager. They can all be the same object, since the named thread is a single thread and thus implicitly runs things sequentially.

In a pool, the entire "environment" in which a task runs is set up and torn down by TaskTracker::RunTask, and this includes the CurrentDefaultHandle for both SequencedTaskRunner and SingleThreadTaskRunner.

Which of these is set depends on the "execution mode" of the current task. So, if the current task was run via a SequencedTaskRunner, then its execution mode is kSequenced, and only the SequencedTaskRunner::CurrentDefaultHandle will be set during execution of the task.

This makes sense when you consider that a task may be invoking functions down a dependency chain. In churl, we're invoking URLRequest functions which call into more specific //net libraries. All of those may need to schedule callbacks in the "current" sequence, and it's possible that one of the deeply buried dependencies needs to be single-threaded or (in the case that began this whole journey!) sequenced. Using the different CurrentDefaultHandles at least makes such a thing crash immediately if the requirement isn't met, rather than subtly introducing race conditions. It'd be nice if that could be caught at compile time [2]!

Why does base::ThreadPool::PostTask fail? for real this time

Briefly: my top-level Task includes its dependencies, so it doesn't match the description doesn’t have ordering or mutual exclusion requirements in the text quoted at the top of this post.

The base::ThreadPool::PostTask API is a shortcut to base::ThreadPool::CreateTaskRunner(...)->PostTask. Which is to say, it runs the task in the thread pool's TaskRunner. A TaskRunner makes no guarantees about ordering, meaning that there is no "current" SequencedTaskRunner when the task is running.

The base::ThreadPool::CreateSingleThreadTaskRunner({}) method gets a new SingleThreadTaskRunner (actually, it may be a singleton, but that's not important) that knows how to schedule work into the thread pool. Posting the task on that runs the task in the thread pool, in a single-threaded context.

Conclusion

I feel like I understand the situation quite a bit better now:

• I can avoid making mistakes that would lead to concurrency errors.
• I can understand and debug task-related errors like the one at the top of this post.
• I can reason about the behavior of code that uses tasks.

There's a lot more detail about jobs, efficiency (both in terms of CPU time and power consumption), responsiveness, fairness, object ownership, and so on that was interesting to learn about, but ultimately doesn't serve these goals, so I've omitted it here.

I'll be making a CL to update and expand the documentation somewhat, in hopes of making this more efficient for the next person. Leaving things better than you found them is a good habit to cultivate!

Next time, we'll get back to the task at hand: loading a URL. I promise.

[1] All of the errors in this post are mine, not Gabriel's! All of them. All mine.

[2] The distinction between "Sequenced" and "SingleThread" is basically what Rust's Send bound represents. Rust's async support is conceptually similar to task scheduling, in that an executor schedules polls of various futures. Some executors poll all futures in a single thread, meaning that the futures do not need to be Send. But more "advanced" executors use a pool of worker threads and work-stealing to allow futures to be polled from multiple threads (so the futures must be Send), but guarantee that it won't happen simultaneously (so the futures need not be Sync). I don't know of an executor that can handle both Send and non-Send futures.