Java introduces a groundbreaking feature: Virtual Threads, designed to address the limitations of traditional threading models and make high-concurrency applications more accessible and efficient. In this blog, we'll dive deep into the why, what, and how of virtual threads, compare them with other concurrency models, and explore practical use cases with coding examples.
What are Virtual Threads?
Virtual threads are lightweight threads that are managed by the Java runtime rather than the OS, that reduce the effort of writing, maintaining, and debugging high-throughput concurrent applications. They provide a similar programming model to traditional threads but with much lower resource overhead, enabling the creation and management of a large number of concurrent tasks more efficiently.
There are two kinds of threads, platform threads and virtual threads.Like a platform thread, a virtual thread is also an instance of java.lang.Thread. However, a virtual thread isn't tied to a specific OS thread and when code running in a virtual thread calls a blocking I/O operation, the Java runtime suspends the virtual thread until it can be resumed. The OS thread associated with the suspended virtual thread is now free to perform operations for other virtual threads.
Why Use Virtual Threads?
Use virtual threads in high-throughput concurrent applications, especially those that consist of a great number of concurrent tasks that spend much of their time waiting.
Traditional threads, or platform threads, in Java are directly mapped to operating system (OS) threads. While they are powerful, they come with several limitations:
Resource Heavy: Each platform thread consumes a significant amount of memory (typically 1MB stack size by default).
Scalability Issues: Managing a large number of threads can lead to high context-switching overhead, making it difficult to scale applications efficiently.
Complexity: Writing scalable and maintainable multi-threaded code is complex and error-prone.
These limitations hinder the development of highly concurrent applications, especially those that need to handle tens of thousands or even millions of concurrent tasks, such as web servers or real-time data processing systems.
Virtual threads are not faster threads; they exist to provide scale (higher throughput), not speed (lower latency). Virtual threads are suitable for running tasks that spend most of the time blocked, often waiting for I/O operations to complete. However, they aren't intended for long-running CPU-intensive operations.
How Do Virtual Threads Work?
Virtual threads decouple the application-level concurrency from the OS-level threading model. This decoupling allows the JVM to manage thousands or millions of virtual threads efficiently by multiplexing them onto a smaller number of platform threads.
When the Java runtime schedules a virtual thread, it assigns or mounts the virtual thread on a platform thread, then the operating system schedules that platform thread as usual. This platform thread is called a carrier. After running some code, the virtual thread can unmount from its carrier. This usually happens when the virtual thread performs a blocking I/O operation. After a virtual thread unmounts from its carrier, the carrier is free, which means that the Java runtime scheduler can mount a different virtual thread on it.A virtual thread cannot be unmounted during blocking operations when it is pinned to its carrier. A virtual thread is pinned when it runs code inside synchronized block. So it is recommended to use ReentrantLock.
Creating Virtual Threads:
Creating virtual threads in Java 21 is straightforward. Here’s an example:
Thread.startVirtualThread(() -> {
// Simulate some work
System.out.println("Running in virtual thread: " + Thread.currentThread());
});
Virtual threads can also be created with ExecutorService which is easy to manage
ExecutorService myExecutor=Executors.newVirtualThreadPerTaskExecutor());
Future<?> future = myExecutor.submit(() -> System.out.println("Running thread"));
future.get();
System.out.println("Task completed");
Can I create 1,000,000 Virtual Threads?
Yes, I tried creating 1,000,000 and it took 5 seconds to create and run them.
public static void main(String[] args) throws InterruptedException {
Instant start = Instant.now();
Set<Long> vThreadIds = new HashSet<>();
var vThreads = IntStream.range(0, 1_000_000)
.mapToObj(i -> Thread.ofVirtual().unstarted(() -> {
vThreadIds.add(Thread.currentThread().getId());
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
})).toList();
vThreads.forEach(Thread::start);
for (var thread : vThreads) {
thread.join();
}
Instant end = Instant.now();
System.out.println("Time =" + Duration.between(start, end).toMillis() + " ms");
System.out.println("Number of unique vThreads used " + vThreadIds.size());
}
Output:
Time = 4482 ms
Number of unique vThreads used 1000000
Memory Efficiency: Virtual threads use a much smaller stack size than platform threads, often just a few kilobytes compared to megabytes. This reduction in stack size allows the JVM to manage a much larger number of virtual threads without exhausting memory.
JVM Management: The JVM handles the scheduling of virtual threads onto a smaller number of platform threads, reducing the overhead of context switching and improving efficiency.
Scalability: By decoupling application-level concurrency from OS-level threads, virtual threads can scale to handle hundreds of thousands of concurrent tasks, making them ideal for modern applications requiring high concurrency.
Why Can’t I Create 100,000 Normal Threads?
Creating 100,000 normal (platform) threads is not feasible due to their heavy memory consumption and the OS's limitations in handling such a large number of threads. Each platform thread typically uses around 1MB of memory for its stack. Creating 100,000 threads would require around 100GB of memory just for the stacks, which is impractical for most systems.Try: I tried to create Executors.newFixedThreadPool(100000) and got OutOfMemoryError. Please do try, its interesting.
Using Virtual Threads vs. Other Concurrency Models
Virtual Threads and Parallel Streams:
Parallel streams abstract the complexity of parallel processing but are limited by the number of available cores. Virtual threads can handle a larger number of concurrent tasks by efficiently managing the available resources, making them suitable for tasks that involve I/O operations and need higher concurrency.Note: Parallel streams should never be used where IO operation is involved.
Virtual Threads and CompletableFutures:
CompletableFutures offer a way to handle asynchronous computations but can become complex when dealing with many interdependent futures. Virtual threads simplify this by allowing a straightforward threading model without the overhead of managing numerous futures.
Virtual Threads and Reactive WebFlux:
Reactive programming with WebFlux is powerful for I/O-bound applications but requires a different programming model. Virtual threads offer a simpler, more intuitive model while achieving similar scalability for many concurrent I/O tasks.I like reactive programming.
Virtual Threads in One Request Per Thread Model
The "one request per thread" model is a common pattern where each incoming request is handled by a separate thread(tomcat). This model is simple and intuitive but scales poorly with platform threads due to their high resource usage. Virtual threads can revolutionize this model by making it feasible to handle thousands or even more of concurrent requests efficiently.Simple and Scalable: This example sets up an HTTP server where each request is handled by a new virtual thread, using the Executors.newVirtualThreadPerTaskExecutor(). This approach combines the simplicity of the one request per thread model with the scalability of virtual threads.Low Overhead: Virtual threads allow the server to handle a massive number of concurrent connections without the resource overhead associated with platform threads.
How Virtual threads are so efficient?
Assume a scenario where we have one carrier thread(platform thread) and three virtual threads. We will focus on what happens when a virtual thread performs an I/O operation and how the other virtual threads are managed.
VirtualThread-1 starts executing on CarrierThread-1. At some point, VirtualThread-1 needs to perform a blocking I/O operation (e.g., reading from a file or a network socket).VirtualThread-2 and VirtualThread-3 are in a ready-to-run state but are not currently running.
When VirtualThread-1 hits the blocking I/O operation, it cannot continue execution until the I/O is complete. The JVM detects that VirtualThread-1 is about to block and performs a context switch.
The JVM unlinks VirtualThread-1 from CarrierThread-1. VirtualThread-1 is now parked and placed in a waiting state. CarrierThread-1 becomes available to run other virtual threads.
The JVM scheduler selects VirtualThread-2 to run next. VirtualThread-2 is now linked to CarrierThread-1 and starts executing on this carrier thread.
VirtualThread-2 runs on CarrierThread-1 until it either completes or hits another blocking operation. When VirtualThread-2 also performs a blocking operation, a similar unlinking process occurs. CarrierThread-1 would then be free to run VirtualThread-3.
Once the I/O operation that blocked VirtualThread-1 completes, the virtual thread is ready to resume execution. The JVM scheduler finds an available carrier thread for VirtualThread-1.
If CarrierThread-1 is available, VirtualThread-1 is re-linked to CarrierThread-1. If CarrierThread-1 is busy, the JVM scheduler will find another available carrier thread. VirtualThread-1 resumes execution from the point where it was blocked. The execution continues until the virtual thread completes or hits another blocking operation.
Virtual threads have their stacks stored in the heap, unlike platform threads that use OS-provided stack space. This allows the JVM to efficiently manage and switch stacks without the overhead of OS-level thread context switching.
Virtual threads leverage a continuation-based model. When a virtual thread is unlinked from a carrier thread, its state is captured as a continuation. This state can be stored and resumed later without needing a one-to-one mapping with carrier threads.
The JVM's scheduler ensures that carrier threads are not idle when there are runnable virtual threads. This efficient scheduling minimizes the time a carrier thread is idle and maximizes CPU utilization.
What is Continuation-based model
A continuation is a mechanism that allows a computation to be paused and resumed at a later point. In the context of virtual threads, continuations enable the JVM to suspend and resume the execution of virtual threads efficiently.
Pausing Execution: When a virtual thread encounters a blocking operation (e.g., waiting for I/O), the JVM can pause the execution of the thread. This is done by saving the current state of the virtual thread, including its call stack and local variables.
Resuming Execution: Once the blocking operation completes, the JVM can resume the execution of the virtual thread from the exact point it was paused. The saved state is restored, and the computation continues as if it were never interrupted.
Encountering a Blocking Operation: When a virtual thread performs a blocking operation (such as waiting for an I/O operation), the JVM detects this and initiates the process of unlinking the virtual thread from its current platform thread.
Unlinking from Platform Threads: The JVM suspends the execution of the virtual thread by creating a continuation. The current state of the virtual thread is saved and the virtual thread is unlinked from the platform thread. The platform thread is then free to execute other tasks.
Moving to the Heap: The state of the virtual thread, now encapsulated as a continuation, is stored in the heap. This means that while the virtual thread is waiting for the I/O operation to complete, it does not consume the resources of a platform thread.
Completing the I/O Operation: Once the I/O operation completes, the JVM reactivates the virtual thread. The saved continuation is retrieved from the heap, and the virtual thread is linked back to an available platform thread.
Resuming Execution: The virtual thread resumes execution from the point where it was paused, continuing with its task as if it was never interrupted.
The JDK's virtual thread scheduler is a work-stealing ForkJoinPool that operates in FIFO mode. The parallelism of the scheduler is the number of platform threads available for the purpose of scheduling virtual threads. By default it is equal to the number of available processors. A virtual thread can be scheduled on different carriers over the course of its lifetime, i.e the scheduler does not maintain affinity between a virtual thread and any particular platform thread.
Conclusion: When to Use Virtual Threads
When to Use Virtual Threads:
High Concurrency Needs: When your application requires handling a large number of concurrent tasks, such as a web server handling many simultaneous connections.
I/O-Bound Tasks: Ideal for applications with many I/O-bound tasks, where threads spend most of their time waiting for I/O operations to complete.
Simplifying Thread Management: When you want to simplify concurrency management without dealing with the complexities of asynchronous programming or reactive frameworks.
Request per Thread Model: Perfect for web servers and other applications following the request per thread model, allowing each request to be handled independently and efficiently.
When Not to Use Virtual Threads:
Heavy Computational Tasks: For CPU-bound tasks that require intensive computation, traditional threading models or ForkJoinPool or parallel streams might be more efficient.
Limited Threading Needs: If your application only requires a manageable number of threads, the benefits of virtual threads might not be significant.
Credits: Java,Oracle,openjdk official documentation and Java Youtube channel.
Top comments (0)