Protobuf Everywhere: How Lupyd Uses Protocol Buffers Across Web, Mobile, and Databases
Lupyd Systems & Architecture #Protobuf #Protocol Buffers #Architecture
Protobuf Everywhere: How Lupyd Uses Protocol Buffers Across Web, Mobile, and Databases
Why Lupyd, Firefly, and Noon are binary-first across client HTTP endpoints, Protobuf Web Tokens, database storage, and cryptographic workflows without touching gRPC.
S
Sri Pranav Tene, CEO
Lupyd
Sept 29, 2026
•
9 min read
On this page (12 sections)
- Separating Protobuf from gRPC
- The Binary-First Architecture
- Protobuf Web Tokens in Firefly
- Form Schemas and Submissions in Noon
- Unified Cross-Platform Code Generation
- Built-in Schema Validation Without Extra Libraries
- Client HTTP Endpoints Over Plain HTTP
- Storing Protobuf Wire Formats in the Database
- Deterministic Message Encoding in Firefly E2EE
- Performance and Resource Improvements
- Real-World Limitations and Quirks
- Conclusion
- Send them over a standard HTTP/1.1 or HTTP/2 POST request using `fetch()` or `curl`.
- Stream them over WebSockets or WebRTC data channels.
- Store them directly on disk or in a database column.
- Feed them into cryptographic hashing and encryption routines.
- No Key Name Duplication: In JSON, field names like `"conversation_id"` and `"sender_user_id"` are repeated in every record. Protobuf replaces these with small numeric field tags (varints).
- Direct Memory Decoding: Binary decoding reads fixed-width bytes and varints directly into typed structures, bypassing heap allocations and string scanning.
- Compile-Time Contracts: Data structures are locked into strongly typed interfaces across every platform, eliminating field mismatches and silent serialization bugs.
View Full Visual
When developers hear Protocol Buffers, they usually assume gRPC comes with it. The two tools originated from Google and are frequently paired together, but they address entirely different problems. Protocol Buffers is a binary serialization format; gRPC is an RPC transport framework. Across Lupyd and its sub-projects, Firefly and Noon, we use Protobuf throughout our web, desktop, and mobile clients, client-facing HTTP endpoints, authentication tokens, database records, and cryptographic workflows, all without running gRPC on our client connections.
Separating Protobuf from gRPC
Protobuf does not require HTTP/2 streaming, special RPC stubs, or bespoke networking gateways. At its core, Protobuf takes structured data defined in a schema file and encodes it into compact binary bytes. What you do with those bytes is up to you:
Adopting gRPC means accepting its full networking stack: HTTP/2 framing, bidirectional streaming models, and specialized tooling like gRPC-Web proxies for browser support. For many architectures, especially web and mobile client applications, that infrastructure adds operational complexity without proportional benefit.
Decoupling the serialization format from the transport layer gives us the benefits of compact binary data and generated types while keeping our transport layer standard, stateless HTTP.
Protocol Buffers
A schema-driven binary serialization format. Compiles to native classes across languages, packs data into minimal byte lengths, and validates types during decoding.
gRPC
A network transport framework built on HTTP/2 streams. Handles connection multiplexing, RPC routing, and client-server method invocation.
The Binary-First Architecture
Lupyd and its core systems, including Firefly and Noon, are built around a binary-first architecture. In our stack, binary is the native format for in-memory operations, network transmission, and persistent storage.
Text formats such as JSON are treated as auxiliary encodings. They exist exclusively at the outer boundaries of our systems when we need to provide backwards compatibility for third-party integrations, external webhooks, or public developer APIs. Within our own client-to-server and internal communication paths, data remains binary from start to finish.
Avoiding text-based serialization eliminates the recurring costs of string parsing:
Protobuf Web Tokens in Firefly
Most web services rely on JSON Web Tokens (JWT) for stateless authentication. A standard JWT consists of three separate JSON objects (header, payload, and signature) that are each base64url-encoded and joined by dots:
jwt-token.txt Standard JWT
// Standard JWT: text serialized to JSON, then re-encoded to base64url strings
eyJhbGciOiJFUzI1NiJ9.eyJzdWIiOiJ1c3JfODkiLCJleHAiOjE3Mzg5... .MEQCIG3x...Every time a client attaches a JWT to a request and every time a server verifies it, both sides must perform string splitting, base64 decoding passes, and JSON string parsing before they can even inspect the user's identity or verify the signature.
In Firefly, we replaced JWT with a custom Protobuf Web Token (PWT). A PWT defines the authentication envelope directly as a Protobuf message:
token.proto Protocol Buffers
syntax = "proto3";package firefly.auth.v1;
message ProtobufWebToken {
bytes user_id = 1;
int64 issued_at = 2;
int64 expires_at = 3;
uint32 key_version = 4;
repeated string scopes = 5;
bytes signature = 6;
}
With PWT, authentication operates purely on raw bytes. The user identifier is stored as compact bytes, the timestamps and key versions are encoded as varints, and the cryptographic signature (such as an Ed25519 signature) sits directly in a byte field.
There are no base64 string conversions, no string-concatenation routines, and no JSON parsing passes on auth checks. The token decodes in a single step directly into native memory, and the signature verifies immediately over the serialized claims buffer.
Form Schemas and Submissions in Noon
Lupyd's anonymous feedback and voting platform, Noon, relies on RSA Blind Signatures (RFC 9474) to decouple a participant's identity from their submission.
In Noon, both the form creation schema and the respondent's form submission are modeled in Protobuf. This choice is critical for the cryptographic guarantees of blind signatures.
During the Noon protocol, the respondent computes a cryptographic digest of their submission before blinding:
m = SHA256(SubmissionPayload || Nonce)
If `SubmissionPayload` were encoded as JSON, differences in client platforms (key ordering, comma spacing, or integer vs float representation) would produce different byte sequences for identical answers. That discrepancy would invalidate the unblinded signature when the user submits their response to the verification server.
Using Protobuf ensures that:
1. Form Definitions Are Structured: Questions, input types, validation constraints, and options are compiled into typed models across the Noon form builder and client renderer. 2. Submissions Are Byte-Deterministic: The respondent's answers serialize into an exact binary buffer regardless of whether the form is submitted from a browser, a desktop client, or a mobile device. The SHA-256 hash matches precisely, and the blind signature verifies without issue. 3. Wire-Level Answer Validation: When a response arrives, the Protobuf decoder validates that required fields and answer types match the form schema without needing separate validation libraries.
Unified Cross-Platform Code Generation
Lupyd maintains clients across multiple platforms and languages:
Maintaining data models manually across four distinct languages is error-prone. A typo in a property name or a mismatched type definition in one client can quietly cause runtime issues.
With Protobuf, all data structures are defined once in version-controlled `.proto` files:
chat\_message.proto Protocol Buffers
syntax = "proto3";package lupyd.messaging.v1;
message ChatMessage {
string message_id = 1;
string conversation_id = 2;
string sender_id = 3;
bytes encrypted_body = 4;
int64 timestamp_ms = 5;
optional string reply_to_id = 6;
}
Running our build pipeline generates native types and serialization functions across all targets:
When an API field changes or a new message type is added, updating the `.proto` file updates the models across every repository. Type mismatches become compile-time errors instead of production bugs.
Built-in Schema Validation Without Extra Libraries
In JSON-based APIs, verifying incoming data requires dedicated schema libraries such as Zod, Joi, or custom validator functions. Teams write and maintain validation logic to check:
This approach duplicates effort: backend engineers define validation in Go or Rust, while frontend engineers define identical schemas in TypeScript. If the two definitions fall out of sync, requests fail unexpectedly.
Protobuf handles schema validation directly during deserialization. The decoder reads binary wire tags, maps them to known field types, and constructs the object. If an incoming byte stream contains malformed data or type mismatches, the decode operation fails immediately.
This eliminates thousands of lines of handwritten validation boilerplate. The schema defined in the `.proto` file is the sole contract, enforced automatically at the binary decoding boundary.
Client HTTP Endpoints Over Plain HTTP
Instead of running gRPC-Web proxies or custom streaming protocols for client communication, Lupyd's API endpoints use standard HTTP POST requests with a Protobuf payload.
In the web client, sending a message uses the standard browser `fetch()` API:
api-client.ts TypeScript
// Encode client payload to binary bytes
const body = ChatMessage.encode({
messageId: "msg_90412",
conversationId: "conv_441",
senderId: currentUserId,
encryptedBody: ciphertext,
timestampMs: BigInt(Date.now()),
}).finish();// Standard HTTP POST request
const res = await fetch("/api/v1/messages/send", {
method: "POST",
headers: {
"Content-Type": "application/x-protobuf",
"Accept": "application/x-protobuf",
},
body: body,
});
// Decode binary response
const buffer = await res.arrayBuffer();
const ack = MessageAck.decode(new Uint8Array(buffer));
This architecture retains full compatibility with existing web infrastructure:
Storing Protobuf Wire Formats in the Database
Many modern web applications store semi-structured data or event logs inside relational databases using JSON columns (such as PostgreSQL's `JSONB`).
While flexible, JSON storage comes with operational costs:
In Lupyd, complex event objects, form records, and internal messages are stored directly as raw Protobuf wire format bytes (using binary columns such as PostgreSQL `BYTEA` or SQLite `BLOB`).
This provides three clear operational benefits:
1. Compact Storage: Binary wire format is significantly smaller than text JSON, reducing table size and improving buffer cache hit ratios. 2. Faster Reads: Decoding raw binary bytes in application memory takes microseconds, avoiding the string scanning and allocation overhead of JSON parsers. In cases where the server merely relays stored events to a client, it can stream the raw bytes directly to the HTTP response without deserializing them at all. 3. Safe Schema Evolution: Protobuf is explicitly designed for forward and backward compatibility. When we add a new optional field with a new tag number, older server binaries or background workers continue reading existing rows without errors or database migration scripts.
Deterministic Message Encoding in Firefly E2EE
Lupyd's end-to-end encryption engine, Firefly, relies on mathematical guarantees: digital signatures, cryptographic hashes, and MLS (Messaging Layer Security) epoch state transitions.
In cryptography, signatures and hashes must be computed over exact, unambiguous byte sequences. A single modified byte invalidates a signature.
JSON serialization is inherently non-deterministic across platforms and programming languages:
If a client running JavaScript signs a JSON message and a recipient running Rust or Kotlin verifies that signature, minor discrepancies in key ordering or formatting will cause signature verification to fail.
Firefly eliminates this ambiguity by using Protobuf for pre-encryption serialization:
Firefly Serialization Pipeline
1. Pre-Encryption Serialization: The message payload is encoded to Protobuf binary bytes. Because Protobuf fields are ordered by tag number and packed predictably, the resulting byte array is deterministic across Web, Rust, and mobile runtimes. Signatures are calculated directly over these bytes.
2. Ciphertext Packaging: After encryption via MLS, the ciphertext, epoch identifier, and routing metadata are packaged into an outer Protobuf envelope. This envelope is transmitted over standard HTTP to Lupyd relays, which route the bytes blindly to recipient devices.
Performance and Resource Improvements
Across our client and server workloads, adopting binary Protobuf yields consistent performance improvements:
Real-World Limitations and Quirks
While Protobuf provides clear advantages, it comes with practical trade-offs that teams should consider before adopting it broadly:
1. No Native UUID Type
Protobuf does not include a dedicated UUID primitive. Applications that use UUIDs must choose between two approaches:
2. Dynamic Types are Unwieldy
Protobuf supports unstructured data via types like `google.protobuf.Struct` and `Any`. However, using these types requires manual packing, unpacking, and runtime type checking. If an application relies heavily on dynamic, arbitrary key-value dictionaries, Protobuf adds complexity without providing its core benefit: static compile-time safety.
3. Default Values and Field Presence in Proto3
In Proto3, fields originally defaulted to zero-values (empty string, `0`, `false`) and were omitted from the wire when unset. This made it difficult to determine whether a field was intentionally set to `0` or simply not provided.
Later releases reintroduced the `optional` keyword to Proto3, restoring explicit field presence tracking while maintaining backward compatibility. Teams must still be deliberate about when to rely on default zero-values versus explicit `optional` presence checks.
Conclusion
Protocol Buffers is not tightly coupled to gRPC. It is a capable, compact serialization format that functions effectively over plain HTTP, inside databases, and across client applications.
By separating the data format from the transport framework, Lupyd, Firefly, and Noon maintain standard, stateless web infrastructure while operating on a unified binary foundation across every layer of the stack.
Key Takeaways
Continue Reading
Related Articles
Advanced Security Security Architecture Aug 15, 2026 • 6 min read Firefly Chat Suite: An Open-Source Alternative for Safe Team Chat An overview of the Firefly chat suite: open-source tools, private channels, and easy-to-run clients for privacy-conscious teams. S Sri Pranav Tene, CEO Read 
Encryption Encryption Technology Apr 16, 2026 • 8 min read How Noon Delivers Honest Team Feedback With Truly Anonymous Surveys Meet Noon: honest workplace surveys and public feedback forms where mathematical proofs make it impossible to trace who submitted what. S Sri Pranav Tene, CEO Read 
Secure Communication Messaging Security Mar 31, 2026 • 6 min read How Message Verification Proves Your Chat Was Not Changed in Transit How cryptographic hash checks guarantee that nobody in the middle altered, reordered, or tampered with the message you received. M Meenakshi Karnataka, Project Lead Read
Open Source
Explore Lupyd on GitHub
Inspect our open-source repositories, cryptography libraries, and client tools on GitHub.
Protobuf Everywhere: How Lupyd Uses Protocol Buffers Across Web, Mobile, and Databases