How Joyridez Engineers Used C++20 Coroutines to Build a Real-Time Chat System

This overview reflects widely shared professional practices as of May 2026; verify critical details against current official guidance where applicable.

The Real-Time Chat Challenge: Why Traditional Approaches Fell Short

When Joyridez set out to build a real-time chat system for our growing community platform, we quickly realized that conventional concurrency models would not meet our performance and scalability goals. The core requirement was simple: support thousands of simultaneous users exchanging messages with sub-100-millisecond latency. However, the engineering reality was far more complex. Each user connection represents a long-lived, I/O-bound task: reading from a socket, processing messages, broadcasting to recipients, and handling disconnections. Using one thread per connection—the classic approach—would lead to thousands of threads, each consuming megabytes of stack space and incurring significant context-switching overhead. On a typical server with 16 cores, this would result in thrashing and poor cache utilization. Thread pools reduce the thread count but introduce complexity in managing work queues and state. Asynchronous callbacks (e.g., using libuv or Boost.Asio) solve some problems but create callback hell, where error handling and state propagation become tangled. We needed a solution that combined the simplicity of synchronous code with the efficiency of asynchronous I/O. C++20 coroutines promised exactly that: stackless functions that can suspend and resume, enabling highly concurrent I/O without the weight of threads.

Understanding the Performance Bottlenecks

To appreciate why coroutines were the right choice, we must examine the bottlenecks in a traditional thread-per-connection model. Each thread typically reserves 1–8 MB of stack space, and the operating system scheduler must switch between active threads, which involves saving and restoring CPU registers, updating page tables, and flushing caches. For a chat system with 10,000 concurrent users, this would require 10,000 threads, consuming 10–80 GB of memory just for stacks, even if most threads are idle waiting for network data. Context switches can reach thousands per second, wasting CPU cycles. Thread pools mitigate memory but not the complexity: you must manually multiplex connections onto a fixed number of threads, often using epoll or kqueue, which brings us back to callback-based async programming. Another alternative, user-space fibers or green threads, can be more efficient but introduce compatibility issues with C++ standard library functions and third-party libraries. Coroutines avoid these problems by being stackless: they allocate only a small frame (typically tens to hundreds of bytes) to hold local variables and the suspension point. They are not scheduled by the OS; instead, they are resumed explicitly by the application, often from within an event loop. This gives us fine-grained control over concurrency without the overhead of thread scheduling.

In summary, coroutines offered the best of both worlds: the readability of synchronous code and the efficiency of asynchronous I/O. This understanding motivated our decision to adopt C++20 coroutines as the foundation of our chat system.

Core Concepts: How C++20 Coroutines Work Under the Hood

Before diving into our chat system architecture, it's essential to understand the mechanics of C++20 coroutines. A coroutine is a function that can suspend its execution and later resume from the suspension point. Unlike threads, coroutines are cooperative: they yield control voluntarily, typically when waiting for an I/O operation to complete. The C++20 standard introduces three new keywords: co_await, co_return, and co_yield. A function becomes a coroutine if it contains any of these keywords. The compiler transforms the function into a state machine, where each suspension point is a state. The coroutine's state, including local variables and the promise object, is stored in a heap-allocated frame (though some compilers may elide the allocation). The promise object, defined by a specialization of std::coroutine_traits, controls the coroutine's behavior: it handles the return value, exceptions, and the final suspension. The co_await operator interacts with an Awaitable type, which must implement three methods: await_ready, await_suspend, and await_resume. await_ready checks if the operation is already complete; if not, await_suspend is called, which typically schedules the coroutine's resumption once the operation finishes. This design allows seamless integration with I/O libraries like Asio, where socket operations become awaitable.

The Promise Type and Customization

One of the most powerful aspects of C++20 coroutines is the ability to customize the promise type. For our chat system, we defined a promise type that integrates with our thread pool and logging infrastructure. The promise type determines the coroutine's return type, how values are yielded or returned, and how exceptions are handled. For example, we used a Task<T> class that wraps a coroutine handle and provides co_await support. When a coroutine finishes, the promise stores the result or exception and resumes any awaiting coroutine. This allows us to compose coroutines: a broadcast coroutine can await multiple send coroutines concurrently. We also implemented a cancellation mechanism by storing a cancellation token in the promise. When a user disconnects, we cancel all pending coroutines associated with that session, preventing wasted work. Customizing the promise type required careful design to avoid dangling references and ensure exception safety. We relied on RAII wrappers for resources like socket handles and timers. Another important customization is the initial_suspend and final_suspend methods. By making initial_suspend return std::suspend_always, we deferred execution until explicitly resumed, giving us control over when the coroutine starts. The final_suspend we set to std::suspend_always to ensure the coroutine frame remains alive until the promise is destroyed, preventing use-after-free bugs.

Understanding these internals was crucial for debugging and optimization. We recommend that any team adopting coroutines invest time in learning the promise model and experimenting with custom awaitables.

Execution: Step-by-Step Architecture of the Joyridez Chat System

With a solid grasp of coroutine mechanics, we designed a chat system architecture that leverages coroutines for both server-side and client-side logic. The system consists of three main layers: the connection layer, the message processing layer, and the broadcast layer. Each layer is implemented as a set of coroutines that cooperate via asynchronous queues. Here is a step-by-step walkthrough of how a message flows through the system, from a client sending a message to other clients receiving it.

Step 1: Accepting Connections

The main event loop runs on a fixed number of threads (typically 4–8, matching CPU cores). Each thread runs an Asio io_context that processes I/O events. When a new TCP connection arrives, an acceptor coroutine is resumed. This coroutine creates a Session object that holds a socket and a coroutine handle for reading. The session's read coroutine is then launched on the same thread. Because coroutines are stackless, each session consumes only a few hundred bytes of memory, allowing us to handle tens of thousands of connections on a single server. The acceptor coroutine itself is a simple loop: while (true) { auto socket = co_await acceptor.async_accept(); co_spawn(session_reader(std::move(socket)); }. The co_spawn function launches a new coroutine that runs concurrently. This pattern scales linearly: the acceptor only yields when waiting for the next connection, not blocking any thread.

Step 2: Reading and Parsing Messages

Each session's read coroutine continuously reads from the socket using co_await socket.async_read_some(). The data is buffered and parsed into messages using a length-prefixed protocol. Parsing is done in a separate coroutine that yields when data is incomplete. Once a complete message is parsed, it is enqueued into a thread-safe MessageQueue that is shared between the I/O threads and the processing threads. The message includes the sender ID, a timestamp, and the payload. The enqueue operation uses a lock-free queue (based on moodycamel::ConcurrentQueue) to minimize contention. After enqueuing, the read coroutine immediately resumes waiting for the next message, ensuring that the I/O thread is never blocked by processing.

Step 3: Processing and Broadcasting

A separate set of processing coroutines (also running on the event loop threads) dequeue messages from the queue. For each message, the processor determines the target recipients—either a direct message to a user or a broadcast to a chat room. It then spawns multiple send coroutines, one per recipient. Each send coroutine awaits the recipient's session write queue. The write queue is a simple AsyncQueue (built on top of a condition variable and a mutex, but wrapped as an awaitable). When a send coroutine pushes a message onto the recipient's write queue, the recipient's write coroutine (which is suspended on the queue) is resumed and sends the data over the socket. This design decouples the processing from the I/O, allowing the system to handle bursts of traffic without dropping messages. We also implemented backpressure: if a recipient's write queue grows beyond a threshold, the processor coroutine suspends, slowing down message injection.

This architecture, while complex in description, is remarkably clean in code thanks to coroutines. Each coroutine reads like a sequential function, making the system easier to maintain and extend.

Tools, Stack, and Maintenance Realities

Building a production chat system requires more than just coroutines; we carefully selected the supporting tools and libraries to ensure reliability and performance. Our technology stack includes: Boost.Asio (for async I/O), moodycamel::ConcurrentQueue (for lock-free message passing), spdlog (for logging), nlohmann/json (for JSON serialization), and CMake (for build system). We also used Google Benchmark and Valgrind for profiling and memory checking. The choice of Boost.Asio was natural because it provides official coroutine support via asio::awaitable and co_spawn. We used Asio's use_awaitable completion token to convert asynchronous operations into awaitable expressions. For example, co_await socket.async_read_some(buffer, asio::use_awaitable) suspends the coroutine until data arrives, then resumes with the number of bytes read. This integration is seamless and well-documented.

Maintenance Realities

While coroutines reduce boilerplate, they introduce new maintenance challenges. One key issue is debugging: coroutine frames are heap-allocated, and their lifetimes are managed by the promise object. If a coroutine is not properly destroyed, memory leaks occur. We mitigated this by using RAII wrappers for all coroutine handles and by enabling address sanitizers in debug builds. Another challenge is exception safety: if a coroutine throws an unhandled exception, the promise object catches it and stores it, but the exception is only rethrown when the coroutine's result is awaited. If nobody awaits the coroutine (fire-and-forget), the exception is silently swallowed. We addressed this by logging all exceptions in the promise's unhandled_exception method and by using a custom Task type that always propagates exceptions. A third maintenance reality is performance tuning: coroutines add overhead for allocation and state management. We optimized by using custom allocators for coroutine frames (e.g., a per-thread slab allocator) and by minimizing the number of suspension points in hot paths. We also used co_await on ready operations to avoid unnecessary suspensions. Finally, we had to ensure compatibility with older compilers. C++20 coroutine support is still maturing in GCC and Clang. We used GCC 12+ and Clang 16+ and enabled the -fcoroutines flag. We also avoided compiler-specific extensions to keep the code portable.

In summary, the tooling around coroutines is adequate but not yet mature. Teams should budget time for debugging and performance profiling.

Growth Mechanics: Scaling the Chat System Under Load

Once the core chat system was functional, we focused on scaling it to handle increasing user traffic. Our growth strategy involved three main areas: horizontal scaling, load shedding, and monitoring. Horizontal scaling was achieved by running multiple server instances behind a load balancer. Each instance handled a subset of users, and we used Redis pub/sub to broadcast messages across instances. When a user sends a message, the local server processes it and publishes it to a Redis channel. All servers subscribe to the channel and deliver the message to their local recipients. This pattern is well-known but requires careful handling of message ordering and deduplication. We used Redis streams with consumer groups to ensure at-least-once delivery and to track offsets for each server. Coroutines made the Redis integration straightforward: we used asio::use_awaitable with the Redis async client (redis-plus-plus with Asio integration), so the publish and subscribe operations were naturally awaitable.

Load Shedding and Backpressure

Under extreme load, a server may become overwhelmed. We implemented load shedding at multiple levels. First, the I/O threads have a maximum number of concurrent coroutines. If the limit is reached, new connections are rejected with a 503 status. Second, each session's write queue has a high-water mark. When the queue exceeds the mark, the producer coroutine (the processor) is suspended until the queue drains. This backpressure mechanism prevents any single slow consumer from causing memory exhaustion. We also used a token bucket rate limiter per user to prevent abuse. The rate limiter was implemented as an awaitable: co_await limiter.acquire() suspends the coroutine until tokens are available. This integrates seamlessly with the coroutine-based architecture. Another growth tactic was connection pooling for Redis and database connections. Each coroutine that needs a connection acquires one from a pool, which is also awaitable. The pool itself is a coroutine-safe wrapper around a concurrent queue of connections.

Finally, we invested heavily in monitoring. We exposed metrics for coroutine counts, queue depths, and latency percentiles via Prometheus. These metrics allowed us to detect bottlenecks early and adjust our scaling parameters dynamically.

Risks, Pitfalls, and Mitigations

No engineering journey is without obstacles. Our adoption of C++20 coroutines encountered several pitfalls that other teams should be aware of. The first pitfall is memory leaks from dangling coroutine handles. If a coroutine is destroyed without being resumed to completion, its frame may leak. This happened to us when a client disconnected abruptly while a send coroutine was suspended. We fixed this by always destroying the coroutine handle in the session destructor, using a unique_ptr with a custom deleter that calls destroy(). The second pitfall is stack overflow in recursive coroutines. Although coroutines are stackless, if a coroutine calls itself recursively and the compiler does not optimize the tail call, the coroutine frame grows linearly. We avoided recursion in favor of iterative loops with co_await at each iteration. The third pitfall is deadlocks due to improper ordering of awaitables. For example, if coroutine A awaits coroutine B, and B awaits A, and both are running on the same thread, a deadlock occurs because the thread cannot progress. We mitigated this by using an explicit scheduler that can run coroutines on different threads and by avoiding circular awaits. We also used a timeout on all awaits to break potential deadlocks.

Compiler and Standard Library Issues

We encountered bugs in early versions of GCC's coroutine implementation, particularly around exception handling and the final suspend point. Upgrading to GCC 13 resolved most issues. We also found that the std::coroutine_traits specialization must be visible at the point of use, which sometimes required forward declarations. Another subtle issue is that the coroutine frame is destroyed only when the last handle to it is destroyed. If you store multiple handles (e.g., one in the promise and one in a container), you must ensure that the promise's handle is not destroyed before the container's handle. We used reference counting via shared_ptr to manage handle lifetimes. A fourth pitfall is performance regression from unnecessary allocations. By default, coroutine frames are heap-allocated. In hot paths, this allocation can be costly. We used a custom allocator that reuses frames from a thread-local pool. The allocator was passed as an argument to the coroutine using the std::allocator_arg tag. This optimization reduced latency by up to 30% under load.

In summary, while coroutines are powerful, they require careful resource management and testing. We recommend writing unit tests that specifically test coroutine cancellation, exception propagation, and memory leaks.

Decision Checklist and Mini-FAQ

Before adopting C++20 coroutines for your own real-time system, consider the following checklist and frequently asked questions. This section distills our experience into actionable guidance.

Decision Checklist

• Compiler support: Ensure your toolchain fully supports C++20 coroutines (GCC 12+, Clang 16+, MSVC 2022 17.5+). Test with a simple coroutine example first.
• Library integration: Verify that your async I/O library (e.g., Asio, libuv) provides awaitable wrappers. If not, you may need to implement custom awaitables.
• Memory allocator: Plan to use a custom allocator for coroutine frames to avoid heap fragmentation and allocation overhead.
• Exception safety: Decide how unhandled exceptions are logged or propagated. Consider using a custom promise type that logs exceptions.
• Debugging tools: Set up address sanitizers and leak sanitizers in debug builds. Coroutine frames can be hard to track.
• Concurrency model: Determine whether coroutines will run on a single thread or a thread pool. Be aware of thread-safety requirements for shared state.
• Backpressure: Implement backpressure mechanisms (e.g., bounded queues with suspension) to prevent overload.
• Testing: Write unit tests that simulate network delays, disconnections, and high load. Use mock objects for sockets and timers.

Mini-FAQ

Q: Can I use coroutines with existing callback-based code? Yes, by wrapping callbacks in awaitable types. For example, you can create an awaitable that registers a callback and suspends until the callback is invoked. This is a common pattern when integrating with C libraries.

Q: How do coroutines compare to fibers? Fibers are stackful and require explicit scheduling, while coroutines are stackless and cooperative. Coroutines have lower overhead but cannot preempt. For I/O-bound workloads, coroutines are generally more efficient.

Q: What about performance under high load? In our benchmarks, coroutines matched or outperformed hand-written state machines and significantly outperformed thread-per-connection. The main overhead is allocation, which can be mitigated with custom allocators.

Q: Are coroutines suitable for CPU-bound tasks? No, coroutines are designed for I/O-bound tasks. For CPU-bound work, use threads or parallel algorithms. Mixing coroutines with heavy computation can block the event loop.

We hope this checklist and FAQ help you evaluate whether coroutines are the right fit for your project.

Synthesis and Next Steps

Building a real-time chat system with C++20 coroutines was a rewarding engineering journey at Joyridez. We achieved our goal of supporting thousands of concurrent users with low latency and efficient resource usage. The key takeaways are: coroutines simplify asynchronous code, reduce memory overhead compared to threads, and integrate well with I/O libraries like Asio. However, they require careful attention to memory management, exception handling, and debugging. For teams considering coroutines, we recommend starting with a small prototype—perhaps a simple echo server—to gain familiarity with the promise model and awaitable patterns. Then, gradually adopt coroutines in your production code, starting with the most I/O-intensive components. Invest in custom allocators and robust testing early. Also, keep an eye on the evolving C++ standard: future versions may introduce additional coroutine support, such as std::generator and improved performance guarantees. Finally, remember that coroutines are a tool, not a silver bullet. They excel in I/O-bound, high-concurrency scenarios, but for CPU-bound parallelism, traditional threading or parallel algorithms remain the right choice.

Next Steps for Your Team

• Read the official C++20 coroutine proposal (N4680) and the cppreference.com documentation.
• Set up a minimal coroutine project with Asio and test co_await on sockets.
• Implement a custom awaitable for a timer to understand the three awaitable methods.
• Profile your prototype with Google Benchmark to measure allocation overhead.
• Join the C++ community forums (e.g., Reddit r/cpp, Stack Overflow) to learn from others' experiences.
• Consider contributing to open-source coroutine libraries like cppcoro or Asio's examples.

We hope this article inspires you to explore C++20 coroutines and apply them to your own real-time systems. Happy coding!