Earlier this year, the Cloud Native Computing Foundation (CNCF) announced with great fanfare the graduation of the CloudEvents project. CloudEvents is a forward-thinking specification that aims to standardize the publication of event metadata, significantly enhancing interoperability between different platforms, services, and systems. As the only event metadata model in the IT field that is bound to all mainstream messaging protocols and encodings, CNCF CloudEvents particularly focuses on two aspects: delivering events without compromising protocol functions and supporting event migration across multiple protocols without losing any metadata information.
To delve further into the development trajectory of CloudEvents and the deep philosophies behind it, we interviewed Mr. Clemens Vasters, who is not only the Lead Architect in messaging and stream processing at Microsoft, but also one of the active proponents of the CloudEvents project.
InfoQ: Can you share some insights about the entire development process of CloudEvents from its inception to its recognition by the CNCF?
Clemens Vasters: The project started at the end of 2017 in a Seattle conference room, where a group of individuals convened by Google came together to discuss the commonalities in building interoperable event schemas. Due to the participation of many renowned enterprises, we encountered some difficulties in determining the governance model and scope. These discussions not only fostered trust-building but also helped us find the rhythm for collaboration. After a series of explorations, we reached two core consensuses: first, to define a set of minimalist yet practical rules; second, not to reinvent the wheel, especially regarding protocols or encoding, but rather to integrate existing resources. The market’s memory of the standardization of SOAP and WS-* and the subsequent debate between SOAP and REST was fresh in our minds. We realized that when defining the envelope model for event data, we should avoid making the same mistakes. Unlike SOAP/WS-* back then, which put all bets on a single data encoding (XML) and attempted to simplify application protocols into basic transport channels, adding new semantics on top, including end-to-end security, CloudEvents took an entirely opposite approach. We believe users should be able to handle events and their data in their favorite encoding, which is why we established a minimal abstract type system. You can represent an “online” CloudEvent as a separate datagram and encode it in the way you choose, whether it’s JSON, XML, Apache Avro, Google Protobuf, or AMQP, all of which have their respective official “format” specifications.
We refer to the communication that independently contains data as a “structured event”. To smoothly integrate CloudEvents support into existing applications, CloudEvent can be directly mapped onto the current messaging model used by the application, making CloudEvents’ metadata attributes become extended headers of the corresponding protocol. We have implemented bindings such as HTTP, MQTT, AMQP, NATS, and Kafka, and have also created more vendor-specific bindings. This way, you can take advantage of all the benefits and features provided by the protocol/platform you are currently using while transmitting standardized events.
In the development and design process of the CloudEvents specification, we especially focused on ensuring interoperability between different event routing protocols, such as MQTT, HTTP, Kafka, and AMQP. Particularly since events often pass through multiple hops and are forwarded, starting with devices sending events via MQTT or HTTP, following which they are forwarded to Kafka and then inserted into AMQP queues. We have carefully studied this process and ensured that events can be mapped between local protocol messages and structured formats without losing information or causing ambiguity. The decision-making in the design of CloudEvents attribute names, such as not using separators and solely using lowercase Latin letters, was made after thoroughly analyzing the interoperable character sets.
Ultimately, we defined the metadata for CloudEvent:
- What type it is: type
- Where it’s from: source
- What it’s about: subject
- What’s the event ID: id
- When it happened: time
- How the data is encoded: datacontenttype
- What schema the data of this content type follows: dataschema
- Which version of CloudEvents it belongs to: specversion
The current specification version is 1.0, and after completing this version, the focus shifted to expanding this core specification and developing further formats and bindings. The working group has deliberately avoided drafting a “2.0” version but is committed to preserving the core specification to ensure it becomes a foundation everyone can trust. In all standards-making work, patience and stability are crucial, and CloudEvents’ recognition by the CNCF proves that this sort of patience is worthwhile.
Since the implementation of CloudEvents, it has gained widespread acceptance in the industry. During my time at Microsoft, I noticed an increasing number of enterprise customers incorporating CloudEvents into their designs even before discussing certain aspects of their schemes, which is a very positive signal.
For Microsoft, CloudEvents is an event model we use for integrating unoccurred events across various platforms. CloudEvents play a crucial foundational role in the interoperability ecosystem of event-driven applications. A key to its subsequent development is establishing a metadata model that declares CloudEvents and their contents and associates these CloudEvents with application endpoints. In this way, event producers can declare potential triggering events in advance and accurately, making it possible to build applications on this basis. We are eager to see event flows become “type-safe,” allowing consumers to predict the types of events they can receive from a stream or topic. We strive to achieve a level of type safety in event flows by adding generics and templates, aligning with popular programming languages.
We have spent years on this initiative called “xRegistry”, eventually defining a highly generic, version-aware, extensible metadata registry model. A notable feature of this model is the complete symmetry between document formats and resource-oriented APIs. We plan to implement this API through OpenAPI, while the document formats will be represented using JSON and Avro schemas. We hope that the document formats will support XML and that the APIs can be expressed through RPC bindings or other methods. The specific registries defined in xRegistry are version-aware and support serialization and validation of schemas (such as JSON Schema, Avro Schema, Protos, etc.); they also serve as a message metadata registry, declaring CloudEvents as well as message templates for MQTT, AMQP, Kafka, NATS, HTTP, etc., and binding their payloads to the schema registry; and an endpoint registry used for cataloging application network endpoints bound to the message definition registry.
We also have a draft of the registry, containing contract definition documents such as OpenAPI and AsyncAPI. As with all the work in this working group, our principle is to create only what is necessary but to be very thorough in the work we choose. We have also given considerable thought to the scope of our work, believing that we should not standardize relationships between event channels until we can accurately describe individual event channels. This is also why we are currently not considering higher-level contract models that would explain, for example, if you send an event to channel A, the possible response that might be generated by channel B.
Having a formal contract model that can reflect an activity graph of multiple pillar event streams would be very beneficial. The International Telecommunication Union has some early technologies in this field, although so far it has not been able to realize this concept. The LF AsyncAPI work provides a simple contract model from the perspective of a direct connection party for event streams. The prototype code generator we are testing is capable of creating templated AsyncAPI and OpenAPI documents by leveraging endpoint or message group definitions in xRegistry. We believe that xRegistry, as an infrastructure, will complement our work well.
In the field of integration platform implementation, Azure DevOps, and Azure platform solution architecture, experts’ technical expertise is particularly prominent. By participating in various conferences and user groups, they continuously enhance their understanding of these topics, and also write articles for industry professional media to share knowledge and insights. Over the past fourteen years, their professional capabilities have also been recognized by Microsoft, and they have been awarded the honor of Microsoft Azure MVP multiple times.
