How We Scaled Notifications with Fanout

• Home
• Posts
• Careers
• Home
• Posts
• Careers



Loading content

#

How We Scaled Notifications with Fanout



Engineering at Patreon

14 hours ago

New

7

Share

#

How We Scaled Notifications with Fanout

New

14 hours ago

At Patreon, we send billions of notifications per year to fans whenever creators publish posts, start livestreams, or perform other fan-facing actions.

Notifications are one of the most critical tools for creators to engage with their fans. But they can be difficult to scale, especially when evolving older, high-traffic systems. By early 2025, the growth of our largest creators had exposed severe performance bottlenecks in our notification platform. So we rebuilt the platform around fanout: a new layer for generating recipient-specific notifications at scale before handing them off to our existing delivery systems.

The redesign gave us horizontal scalability, stronger channel isolation, and better observability into how notifications move through the platform. Patreon’s highest-volume notifications saw significant performance improvements:

• For large creators, push and in-app feed notifications are now 80% faster, and email notifications are 55% faster.
• When a creator’s audience grows 4x, end-to-end latency now increases by 33% for push and in-app feed and 60% for email, compared with a 186% increase on the legacy platform.

Getting this refactor right was an interesting challenge.

The limits of our legacy notification system

Fanout is a well-understood pattern in distributed systems, and plenty of companies already run mature notification platforms built around it. The challenge was retrofitting a fanout architecture into a 13-year-old codebase at our given scale.

For a long time, our notification system had done its job well enough. Patreon had grown significantly while our engineering team remained relatively small, and investing in dedicated fanout infrastructure was easy to defer when the existing system continued to work. But the launch and growth of free memberships changed the scale of the problem by dramatically increasing the number of fans per creator.

Under the hood, post notifications followed a simple flow. When a creator published a post, a single asynchronous task fetched all recipients, filtered them by user-specific notification settings, generated notification payloads for each recipient, and handed those payloads off to the delivery systems for in-app feed, push, and email.

That simplicity eventually became the bottleneck. For our largest creators, one task could be responsible for generating millions of notifications. By the beginning of 2025, this task consistently timed out, which meant we couldn’t reliably deliver notifications for large creator audiences.

The existing flow had two major limitations:

1. It did not scale horizontally. Notification generation is recipient-specific, which makes it expensive at large audience sizes. Keeping all of that work inside one asynchronous task meant the system had a hard ceiling. 2. It did not isolate channels. In-app feed, push, and email were coupled tightly such that a failure in one channel could block the others. One broken channel could take down the whole notification flow, much like an old string of Christmas lights where one broken bulb turns off the entire strand.

As more product teams needed to send high-volume notifications, they began solving these problems with bespoke scaling patterns. That worked in the short term but it increased platform complexity and made notifications harder to build, operate, and debug.

We set out to build a shared notification fanout platform that could provide horizontal scalability, channel isolation, and better observability while layering cleanly onto our existing architecture.

Designing the fanout platform

We wanted to solve notification generation without rewriting the delivery systems that already powered in-app feed, push, and email. The new platform would sit in front of those delivery systems as a fanout layer: splitting large recipient lists into smaller batches, generating recipient-specific notifications, and forwarding them to the right channel for delivery.

We designed this fanout platform around six requirements:

1. Developer friendly: Engineers should not need to solve scalability and reliability from scratch for every high-volume notification. Notification logic should be structured to minimize usage errors while flexible enough to accommodate a variety of use cases. 2. Horizontally scalable: The platform should automatically split recipient lists into smaller batches for asynchronous processing. 3. Isolated: In-app feed, push, and email should run independently so failures in one channel do not cascade to the others. 4. Priority-based: Time-sensitive notifications should be processed separately from standard notifications using dedicated queues and worker pools. 5. Observable: Engineers should be able to debug issues, inspect filtering and delivery outcomes, and measure latency across the notification lifecycle. 6. Extendible: The platform should support future capabilities (e.g. recipient list generation) without requiring another major refactor.

The Notification Factory abstraction

One of the first design challenges was separating platform orchestration from notification-specific business logic. Each notification has its own filtering rules, eligibility checks, and payload generation logic, but the fanout platform needed a consistent way to execute that logic inside asynchronous task handlers.

To solve this, we introduced a notification factory abstraction. A factory acts as the blueprint for how the platform should process a notification: which priority it should use, which notification setting applies, how recipients should be filtered, and how channel-specific payloads should be generated. The pattern is built on top of our home-grown distributed task engine and shares many of the same concepts.

The Notification model (task arguments) defines the payload with any unique data specific to a particular notification and shared for all recipients. It’s linked to a subclass of BaseNotificationFactory (task handler) that contains configurations and methods to filter recipients and generate notifications.

Below is a simplified version of the factory interface.



Two more design considerations shaped the factory:

• Associating notifications with factories: A fanout task only receives a serialized notification payload, so the platform needs a way to determine which business logic to execute. We solved this with a registry that maps each notification model to its corresponding factory class.
• Balancing flexibility and guardrails: The factory needed to support a wide range of notification types while preventing common implementation mistakes and inefficient data-fetching patterns. Patreon also has multiple user record types and filtering mechanisms, so the interface keeps recipients generic while providing clear hooks for user settings and notification-specific eligibility checks.

A single API for fanout

With the factory abstraction in place, we also wanted to simplify how product teams invoked the notification platform.

Previously, we had separate APIs for sending notifications through each channel. That meant notification flows could diverge across in-app feed, push, and email, causing product teams to think about channel-specific behavior earlier than necessary.

As part of the redesign, we replaced those entry points with a single API: send\_fanout\_notifications

Product teams pass in a notification payload, a recipient list, and the timestamp used to measure delivery latency. From there, the platform handles the rest.

send\_fanout\_notifications looks up the notification’s factory, determines the batch size and priority, and fans out work separately for each channel. Processing each channel independently gives us channel isolation (failures or delays in one channel do not block the others)



How livestream notifications use fanout (example)





How the fanout architecture works

The redesigned platform introduces a two-stage fanout process.

The first stage scales notification generation. The platform splits the full recipient list into smaller batches, and each batch is handled by a FanoutNotifications task. That task filters recipients, runs notification-specific eligibility logic, and uses the notification factory to generate channel-specific payloads.

The second stage scales delivery handoff. Once notifications have been generated, the platform fans them out again into dedicated delivery tasks for in-app feed, push, and email.

Compared to the legacy platform, the main architectural change is the new FanoutNotifications task handler. It sits between the API entry point and the existing channel delivery systems, turning one large unit of work into many smaller tasks that can be processed independently.





Making notifications debuggable

Notifications can fail or be skipped for many reasons. A recipient may have disabled a notification setting, failed an eligibility check, or have an undeliverable email address.

With the legacy platform, understanding what happened to a specific notification often required stitching together logs and data from multiple systems. Without comprehensive per-recipient traces, investigations could take engineers several hours.

We wanted the new platform to make common notification questions easy to answer:

• How long did notifications take to send?
• How large was the original recipient list?
• How many recipients were filtered out, and why?
• How many notifications were successfully sent for each channel?
• Why did a specific user receive, or not receive, a particular notification?

To answer these questions, we introduced timing and logging data models that are propagated through the platform. The platform records timestamps at each stage of the notification flow and emits metrics and events for both aggregate and per-recipient outcomes. Developers can also attach logging props to include relevant entities in traces and events, making it easier to debug notification behavior end to end.



Migrating 200+ notifications

The notification factory pattern changed how developers build and maintain notifications, so the migration could not be handled with a simple codemod. Each notification had its own business logic, data dependencies, eligibility rules, and channel-specific behavior that needed to be preserved.

To speed up the migration, we built AI skills to help engineers convert existing notifications to the new fanout pattern. The skills were grounded in our migration documentation and exemplary hand-written migration pull requests. Developers could then run /migrate-notif-fanout <notif\_name> to generate a first-pass migration.

These AI skills were not a replacement for engineering judgment though. AI can hallucinate, miss subtle business logic, or produce redundant code that needs cleanup.

However, these skills were effective at handling the repetitive structure of the migration: creating notification models, wiring factories, updating call sites, and following the expected platform patterns.

Leveraging AI significantly accelerated the migration. In a large cross-functional effort, we migrated more than 200 notifications and completed 80% of the work in six weeks.

Performance and product impact

The notification fanout platform significantly improved performance for notifications with large recipient lists. By moving recipient-specific generation into horizontally scalable fanout tasks, we can now reliably generate millions of notifications per creator without timing out.

That reliability also improved developer velocity. Product teams no longer need to design bespoke scaling solutions for every high-volume notification. The platform has supported critical launches such as Lives, Quips, and large creator acquisitions in 2025. Without the fanout redesign, those launches would have run into the same scaling bottlenecks as the legacy system and required additional engineering time to build one-off solutions.

The platform also made notifications easier to operate. With better observability, alerting, and debugging tools, teams can independently investigate notification issues instead of relying on platform engineers. Timing metrics also give us a clearer view into where time is spent, helping us tune infrastructure to balance speed and cost.

What made the migration hard

The hardest parts of this project came from evolving a live, legacy system and coordinating a large, cross-team migration.

For nine months, we maintained two notification systems in parallel. Every platform change had to be compatible with both the legacy path and the new fanout path, which increased the surface area for bugs and incidents. The longer the migration ran, the more operational risk we carried. We saw a couple of minor incidents caused by compatibility issues which reinforced the importance of finishing the migration as quickly and safely as possible.

Coordination was the other major challenge. At first, we tried a bottom-up approach where product teams prioritized their own notification migrations. That got us about 20% of the way over 6 months. Progress slowed down as migrations competed with product roadmaps and other team priorities.

The turning point was getting engineering leadership alignment to finish the migration in Q1 2026. With a clear deadline and shared company priority, we completed the remaining migrations in 6 weeks.

In total, the migration involved 10 teams and more than 30 engineers. We used Notion trackers and automations to monitor progress, communicated frequently with stakeholders, and split the work across teams. Our team took on about half of the migrations, then handed them off to notification owners for rollout and cleanup. This helped balance the load so teams with the most notifications were not overwhelmed. We also hosted a notification migration week event with office hours, leaderboards, prizes, and a happy hour to accelerate and celebrate our progress.

We also built in a buffer by finishing handoffs two weeks before the overall deadline. That gave us time to support teams that needed extra help and scan the codebase for stragglers. The final sweep surfaced 4 net-new migrations and several incomplete ones we might have otherwise missed, allowing us to finish the migration with minor impact to timeline.

Lessons learned

• Large migrations need explicit prioritization. Leadership alignment was critical to finishing the migration in a reasonable timeframe. Without clear priorities and deadlines, cross-team migrations compete with every team’s roadmap and are easy to deprioritize. This was especially true for the long tail of low-volume notifications, where the scaling wins were less obvious than they were for high-volume notifications.
• Platform work does not end at the API. A large part of maintaining a platform is helping teams understand and debug what happened. Before this project, a significant portion of our on-call work involved answering notification questions and investigating delivery issues for other teams. By improving observability, alerting, dashboards, and debugging tools, we enabled product engineers and support teams to investigate more issues independently while reducing the on-call burden on platform engineers.
• Design for the next bottleneck, not just the current one. We could not solve every notification scaling problem in one project. Recipient list generation, for example, is still an area we want to improve. But by designing the fanout platform with future capabilities in mind, we reduced the likelihood of needing another large refactor as notification volume continues to grow.

What’s next

The fanout platform solved notification generation at scale, but it also made the next bottleneck clearer: recipient list generation. As Patreon grows, notification recipient lists will continue to get larger, and efficiently fetching millions of recipients will become increasingly important. We plan to bring recipient list generation into the notification factory so the platform can own more of the end-to-end notification flow.

We are also improving notification settings from both the developer and fan experience sides. For developers, we want to make it easier to create and manage new settings. For fans, we want to make notification preferences easier to understand and control.

About the author

Lingene Yang is a backend engineer at Patreon. Outside of work, she loves photography and going on long walks.

Join the team

Interested in joining Patreon? We're hiring! Browse our open positions here.

7

Share

---

#

How We Scaled Notifications with Fanout



Engineering at Patreon

14 hours ago

New

7

Share

#

How We Scaled Notifications with Fanout

New

14 hours ago

At Patreon, we send billions of notifications per year to fans whenever creators publish posts, start livestreams, or perform other fan-facing actions.

Notifications are one of the most critical tools for creators to engage with their fans. But they can be difficult to scale, especially when evolving older, high-traffic systems. By early 2025, the growth of our largest creators had exposed severe performance bottlenecks in our notification platform. So we rebuilt the platform around fanout: a new layer for generating recipient-specific notifications at scale before handing them off to our existing delivery systems.

The redesign gave us horizontal scalability, stronger channel isolation, and better observability into how notifications move through the platform. Patreon’s highest-volume notifications saw significant performance improvements:

• For large creators, push and in-app feed notifications are now 80% faster, and email notifications are 55% faster.
• When a creator’s audience grows 4x, end-to-end latency now increases by 33% for push and in-app feed and 60% for email, compared with a 186% increase on the legacy platform.

Getting this refactor right was an interesting challenge.

The limits of our legacy notification system

Fanout is a well-understood pattern in distributed systems, and plenty of companies already run mature notification platforms built around it. The challenge was retrofitting a fanout architecture into a 13-year-old codebase at our given scale.

For a long time, our notification system had done its job well enough. Patreon had grown significantly while our engineering team remained relatively small, and investing in dedicated fanout infrastructure was easy to defer when the existing system continued to work. But the launch and growth of free memberships changed the scale of the problem by dramatically increasing the number of fans per creator.

Under the hood, post notifications followed a simple flow. When a creator published a post, a single asynchronous task fetched all recipients, filtered them by user-specific notification settings, generated notification payloads for each recipient, and handed those payloads off to the delivery systems for in-app feed, push, and email.

That simplicity eventually became the bottleneck. For our largest creators, one task could be responsible for generating millions of notifications. By the beginning of 2025, this task consistently timed out, which meant we couldn’t reliably deliver notifications for large creator audiences.

The existing flow had two major limitations:

1. It did not scale horizontally. Notification generation is recipient-specific, which makes it expensive at large audience sizes. Keeping all of that work inside one asynchronous task meant the system had a hard ceiling. 2. It did not isolate channels. In-app feed, push, and email were coupled tightly such that a failure in one channel could block the others. One broken channel could take down the whole notification flow, much like an old string of Christmas lights where one broken bulb turns off the entire strand.

As more product teams needed to send high-volume notifications, they began solving these problems with bespoke scaling patterns. That worked in the short term but it increased platform complexity and made notifications harder to build, operate, and debug.

We set out to build a shared notification fanout platform that could provide horizontal scalability, channel isolation, and better observability while layering cleanly onto our existing architecture.

Designing the fanout platform

We wanted to solve notification generation without rewriting the delivery systems that already powered in-app feed, push, and email. The new platform would sit in front of those delivery systems as a fanout layer: splitting large recipient lists into smaller batches, generating recipient-specific notifications, and forwarding them to the right channel for delivery.

We designed this fanout platform around six requirements:

1. Developer friendly: Engineers should not need to solve scalability and reliability from scratch for every high-volume notification. Notification logic should be structured to minimize usage errors while flexible enough to accommodate a variety of use cases. 2. Horizontally scalable: The platform should automatically split recipient lists into smaller batches for asynchronous processing. 3. Isolated: In-app feed, push, and email should run independently so failures in one channel do not cascade to the others. 4. Priority-based: Time-sensitive notifications should be processed separately from standard notifications using dedicated queues and worker pools. 5. Observable: Engineers should be able to debug issues, inspect filtering and delivery outcomes, and measure latency across the notification lifecycle. 6. Extendible: The platform should support future capabilities (e.g. recipient list generation) without requiring another major refactor.

The Notification Factory abstraction

One of the first design challenges was separating platform orchestration from notification-specific business logic. Each notification has its own filtering rules, eligibility checks, and payload generation logic, but the fanout platform needed a consistent way to execute that logic inside asynchronous task handlers.

To solve this, we introduced a notification factory abstraction. A factory acts as the blueprint for how the platform should process a notification: which priority it should use, which notification setting applies, how recipients should be filtered, and how channel-specific payloads should be generated. The pattern is built on top of our home-grown distributed task engine and shares many of the same concepts.

The Notification model (task arguments) defines the payload with any unique data specific to a particular notification and shared for all recipients. It’s linked to a subclass of BaseNotificationFactory (task handler) that contains configurations and methods to filter recipients and generate notifications.

Below is a simplified version of the factory interface.



Two more design considerations shaped the factory:

• Associating notifications with factories: A fanout task only receives a serialized notification payload, so the platform needs a way to determine which business logic to execute. We solved this with a registry that maps each notification model to its corresponding factory class.
• Balancing flexibility and guardrails: The factory needed to support a wide range of notification types while preventing common implementation mistakes and inefficient data-fetching patterns. Patreon also has multiple user record types and filtering mechanisms, so the interface keeps recipients generic while providing clear hooks for user settings and notification-specific eligibility checks.

A single API for fanout

With the factory abstraction in place, we also wanted to simplify how product teams invoked the notification platform.

Previously, we had separate APIs for sending notifications through each channel. That meant notification flows could diverge across in-app feed, push, and email, causing product teams to think about channel-specific behavior earlier than necessary.

As part of the redesign, we replaced those entry points with a single API: send\_fanout\_notifications

Product teams pass in a notification payload, a recipient list, and the timestamp used to measure delivery latency. From there, the platform handles the rest.

send\_fanout\_notifications looks up the notification’s factory, determines the batch size and priority, and fans out work separately for each channel. Processing each channel independently gives us channel isolation (failures or delays in one channel do not block the others)



How livestream notifications use fanout (example)





How the fanout architecture works

The redesigned platform introduces a two-stage fanout process.

The first stage scales notification generation. The platform splits the full recipient list into smaller batches, and each batch is handled by a FanoutNotifications task. That task filters recipients, runs notification-specific eligibility logic, and uses the notification factory to generate channel-specific payloads.

The second stage scales delivery handoff. Once notifications have been generated, the platform fans them out again into dedicated delivery tasks for in-app feed, push, and email.

Compared to the legacy platform, the main architectural change is the new FanoutNotifications task handler. It sits between the API entry point and the existing channel delivery systems, turning one large unit of work into many smaller tasks that can be processed independently.





Making notifications debuggable

Notifications can fail or be skipped for many reasons. A recipient may have disabled a notification setting, failed an eligibility check, or have an undeliverable email address.

With the legacy platform, understanding what happened to a specific notification often required stitching together logs and data from multiple systems. Without comprehensive per-recipient traces, investigations could take engineers several hours.

We wanted the new platform to make common notification questions easy to answer:

• How long did notifications take to send?
• How large was the original recipient list?
• How many recipients were filtered out, and why?
• How many notifications were successfully sent for each channel?
• Why did a specific user receive, or not receive, a particular notification?

To answer these questions, we introduced timing and logging data models that are propagated through the platform. The platform records timestamps at each stage of the notification flow and emits metrics and events for both aggregate and per-recipient outcomes. Developers can also attach logging props to include relevant entities in traces and events, making it easier to debug notification behavior end to end.



Migrating 200+ notifications

The notification factory pattern changed how developers build and maintain notifications, so the migration could not be handled with a simple codemod. Each notification had its own business logic, data dependencies, eligibility rules, and channel-specific behavior that needed to be preserved.

To speed up the migration, we built AI skills to help engineers convert existing notifications to the new fanout pattern. The skills were grounded in our migration documentation and exemplary hand-written migration pull requests. Developers could then run /migrate-notif-fanout <notif\_name> to generate a first-pass migration.

These AI skills were not a replacement for engineering judgment though. AI can hallucinate, miss subtle business logic, or produce redundant code that needs cleanup.

However, these skills were effective at handling the repetitive structure of the migration: creating notification models, wiring factories, updating call sites, and following the expected platform patterns.

Leveraging AI significantly accelerated the migration. In a large cross-functional effort, we migrated more than 200 notifications and completed 80% of the work in six weeks.

Performance and product impact

The notification fanout platform significantly improved performance for notifications with large recipient lists. By moving recipient-specific generation into horizontally scalable fanout tasks, we can now reliably generate millions of notifications per creator without timing out.

That reliability also improved developer velocity. Product teams no longer need to design bespoke scaling solutions for every high-volume notification. The platform has supported critical launches such as Lives, Quips, and large creator acquisitions in 2025. Without the fanout redesign, those launches would have run into the same scaling bottlenecks as the legacy system and required additional engineering time to build one-off solutions.

The platform also made notifications easier to operate. With better observability, alerting, and debugging tools, teams can independently investigate notification issues instead of relying on platform engineers. Timing metrics also give us a clearer view into where time is spent, helping us tune infrastructure to balance speed and cost.

What made the migration hard

The hardest parts of this project came from evolving a live, legacy system and coordinating a large, cross-team migration.

For nine months, we maintained two notification systems in parallel. Every platform change had to be compatible with both the legacy path and the new fanout path, which increased the surface area for bugs and incidents. The longer the migration ran, the more operational risk we carried. We saw a couple of minor incidents caused by compatibility issues which reinforced the importance of finishing the migration as quickly and safely as possible.

Coordination was the other major challenge. At first, we tried a bottom-up approach where product teams prioritized their own notification migrations. That got us about 20% of the way over 6 months. Progress slowed down as migrations competed with product roadmaps and other team priorities.

The turning point was getting engineering leadership alignment to finish the migration in Q1 2026. With a clear deadline and shared company priority, we completed the remaining migrations in 6 weeks.

In total, the migration involved 10 teams and more than 30 engineers. We used Notion trackers and automations to monitor progress, communicated frequently with stakeholders, and split the work across teams. Our team took on about half of the migrations, then handed them off to notification owners for rollout and cleanup. This helped balance the load so teams with the most notifications were not overwhelmed. We also hosted a notification migration week event with office hours, leaderboards, prizes, and a happy hour to accelerate and celebrate our progress.

We also built in a buffer by finishing handoffs two weeks before the overall deadline. That gave us time to support teams that needed extra help and scan the codebase for stragglers. The final sweep surfaced 4 net-new migrations and several incomplete ones we might have otherwise missed, allowing us to finish the migration with minor impact to timeline.

Lessons learned

• Large migrations need explicit prioritization. Leadership alignment was critical to finishing the migration in a reasonable timeframe. Without clear priorities and deadlines, cross-team migrations compete with every team’s roadmap and are easy to deprioritize. This was especially true for the long tail of low-volume notifications, where the scaling wins were less obvious than they were for high-volume notifications.
• Platform work does not end at the API. A large part of maintaining a platform is helping teams understand and debug what happened. Before this project, a significant portion of our on-call work involved answering notification questions and investigating delivery issues for other teams. By improving observability, alerting, dashboards, and debugging tools, we enabled product engineers and support teams to investigate more issues independently while reducing the on-call burden on platform engineers.
• Design for the next bottleneck, not just the current one. We could not solve every notification scaling problem in one project. Recipient list generation, for example, is still an area we want to improve. But by designing the fanout platform with future capabilities in mind, we reduced the likelihood of needing another large refactor as notification volume continues to grow.

What’s next

The fanout platform solved notification generation at scale, but it also made the next bottleneck clearer: recipient list generation. As Patreon grows, notification recipient lists will continue to get larger, and efficiently fetching millions of recipients will become increasingly important. We plan to bring recipient list generation into the notification factory so the platform can own more of the end-to-end notification flow.

We are also improving notification settings from both the developer and fan experience sides. For developers, we want to make it easier to create and manage new settings. For fans, we want to make notification preferences easier to understand and control.

About the author

Lingene Yang is a backend engineer at Patreon. Outside of work, she loves photography and going on long walks.

Join the team

Interested in joining Patreon? We're hiring! Browse our open positions here.

7

Share

---

Loading content

Loading content

Loading content