Enterprise Streaming Architecture with Apache Pulsar

Apache Kafka has been the default choice for enterprise event streaming for the better part of a decade. Its proven scalability, robust ecosystem, and widespread adoption make it a safe choice. But the streaming landscape is evolving, and Apache Pulsar — originally developed at Yahoo and now an Apache Software Foundation top-level project — addresses several architectural limitations that enterprises encounter as they scale their streaming infrastructure.

This is not an argument that Pulsar should replace Kafka everywhere. Rather, it is an examination of where Pulsar’s architectural choices solve genuine enterprise problems that Kafka’s architecture makes difficult, and how CTOs should evaluate streaming infrastructure for their specific requirements.

Architectural Differentiators

Pulsar’s architecture diverges from Kafka in several fundamental ways that have significant enterprise implications.

Separated Compute and Storage: Kafka couples message brokers with storage — each broker manages a set of partitions stored on local disk. This coupling means scaling compute and storage must happen together, which is inefficient when an organisation needs more storage (high retention requirements) without more compute, or more compute (high throughput) without more storage.

Pulsar separates these concerns. Brokers handle message routing and protocol management while Apache BookKeeper handles persistent storage independently. This separation enables independent scaling: add more brokers for throughput, add more BookKeeper nodes for storage capacity. For enterprises with varying workload profiles, this flexibility can significantly reduce infrastructure costs.

Native Multi-Tenancy: Enterprise streaming platforms serve multiple teams, applications, and business units. Kafka supports multi-tenancy through ACLs and quotas, but these were bolted on rather than built in. Pulsar was designed from the ground up for multi-tenancy with a hierarchical namespace model: tenants contain namespaces, which contain topics. Each level supports independent access controls, resource quotas, and configuration.

This native multi-tenancy is particularly valuable for organisations running shared streaming infrastructure. Rather than deploying separate Kafka clusters for different teams (which is operationally expensive) or sharing a single cluster with limited isolation (which creates noisy-neighbour problems), Pulsar provides strong isolation within a single cluster.

Geo-Replication: Enterprises operating across multiple data centres or cloud regions need streaming infrastructure that replicates data across locations. Kafka supports replication through MirrorMaker, but it operates as a separate process that can introduce lag and requires independent monitoring and management.

Pulsar provides built-in geo-replication at the namespace level. Configuring cross-data-centre replication is a namespace-level setting, not a separate infrastructure component. This simplifies operations significantly for global enterprises that need data to flow between regions with minimal latency and operational overhead.

Unified Messaging Model: Kafka provides a log-based messaging model that excels at ordered, replayable event streaming but does not natively support traditional messaging patterns like shared subscriptions with acknowledgment-based consumption. Applications that need message queuing semantics alongside event streaming must use separate systems (Kafka plus RabbitMQ, for example).

Pulsar supports both models natively. Topics can be consumed as ordered logs (exclusive or failover subscriptions) or as shared queues (shared subscriptions with per-message acknowledgment). This unification reduces the number of messaging systems an enterprise must operate and simplifies architectures that need both patterns.

Enterprise Architecture Patterns

Several architecture patterns leverage Pulsar’s specific capabilities for enterprise scenarios.

Multi-Region Event Mesh: For organisations operating across multiple cloud regions or data centres, Pulsar’s geo-replication enables an event mesh architecture where events published in any region are automatically available in all other configured regions. This supports use cases like global inventory synchronisation, cross-region event-driven workflows, and disaster recovery.

The architecture deploys Pulsar clusters in each region with geo-replication configured at the namespace level. Applications publish and consume locally, with Pulsar handling cross-region replication transparently. Replication can be configured per namespace, allowing organisations to replicate only the data that needs to be global while keeping region-specific data local.

Tiered Storage for Event Archiving: Enterprise compliance requirements often mandate long-term retention of event data. Storing months or years of events on high-performance SSD storage is expensive. Pulsar’s tiered storage feature automatically offloads older data to cheaper object storage (S3, GCS, Azure Blob Storage) while maintaining transparent access. Applications can query historical data through the same API, with Pulsar handling the retrieval from the appropriate storage tier.

This pattern is particularly valuable for financial services organisations that need to retain transaction events for regulatory compliance, or for analytics use cases that need access to historical event streams for model training and backtesting.

Function-Based Stream Processing: Pulsar Functions provide a lightweight, serverless stream processing framework built into the Pulsar platform. For organisations that need simple transformations, filtering, and routing of streaming data, Pulsar Functions eliminate the need for a separate stream processing framework like Kafka Streams or Apache Flink.

This does not replace dedicated stream processing for complex use cases — Flink and Spark Streaming provide capabilities that Pulsar Functions does not. But for the large volume of simple stream processing tasks that exist in any enterprise streaming environment, built-in functions reduce operational complexity significantly.

Migration and Coexistence Considerations

For organisations with established Kafka deployments, the question is not whether to replace Kafka wholesale but whether Pulsar addresses specific needs that justify its introduction.

Kafka Protocol Compatibility: Pulsar provides a Kafka protocol handler (KoP — Kafka on Pulsar) that allows existing Kafka clients to communicate with Pulsar brokers without modification. This enables gradual migration: applications can be moved to Pulsar infrastructure without code changes, while new applications can use the native Pulsar client to access Pulsar-specific features.

Coexistence Architecture: Many enterprises will operate both Kafka and Pulsar. Pulsar’s Kafka connector and IO connectors enable bidirectional data flow between platforms. A common pattern is to introduce Pulsar for new use cases that benefit from its specific capabilities while maintaining Kafka for established workloads.

Operational Considerations: Pulsar’s operational model differs from Kafka’s. The separation of brokers and BookKeeper means more components to manage. ZooKeeper (used by both platforms, though Kafka is moving away from it with KIP-500) adds another dependency. Organisations evaluating Pulsar should assess their operational capacity for managing a more distributed architecture.

The managed service landscape for Pulsar is maturing. StreamNative provides managed Pulsar services, and Datastax has integrated Pulsar into their Astra Streaming platform. These managed options reduce the operational burden but introduce vendor dependencies that must be evaluated.

Strategic Evaluation Framework

For CTOs evaluating streaming infrastructure, the decision between Kafka and Pulsar (or the decision to use both) should be driven by specific architectural requirements:

Choose Kafka when: the organisation has established Kafka expertise and infrastructure, the use cases are primarily event streaming with log-based consumption patterns, the ecosystem integrations (Kafka Connect, Schema Registry, KSQL) are important, and multi-tenancy and geo-replication requirements are modest.

Choose Pulsar when: multi-tenancy with strong isolation is a primary requirement, built-in geo-replication across multiple regions is needed, the use cases span both event streaming and traditional messaging patterns, independent scaling of compute and storage is important for cost optimisation, or long-term event retention with tiered storage is a requirement.

Choose both when: the organisation has distinct use cases that map to each platform’s strengths, and the operational capacity exists to manage both platforms effectively.

The streaming infrastructure decision is a foundational architectural choice that influences many subsequent decisions about event-driven architecture, data integration, and real-time analytics. CTOs should approach it with the same rigour applied to database selection or cloud provider choice — evaluating specific requirements against platform capabilities rather than defaulting to the most popular option.