Next generation of data processing and movement platform at Netflix

Netflix engineering recently posted on a technology blog how they used data mesh architecture and principles as the next generation of platform and data processing to unlock more opportunities and business use cases.

Data mesh is the new paradigm shift in data management that allows users to easily import and use data without transporting it to a centralized location like a data lake. It focuses on decentralization and distribution of data ownership across different business domains. Each domain manages and governs your data as a product, making it discoverable and trustworthy based on required business service level objectives (SLOs).

The data mesh architecture in Netflix is ​​made up of two main layers. One is the control plane (controllers) and the other is the data plane (data pipes). It is defined in the blog post simply as:

The controller receives user requests and implements and orchestrates pipelines. Once deployed, the pipeline does the real heavy lifting of data processing. Provisioning a pipeline involves different resources. The controller delegates responsibility to corresponding microservices to manage its lifecycle.

The following diagram shows the high-level architecture of the data mesh in Netflix:

Netflix High-Level Data Mesh Architecture

The main operations are carried out in pipelines. Pipeline reads data from various resources, applies algorithms, and transports it to the destination. Controllers are responsible for finding out the resources associated with the pipes and calculating the correct settings. The following diagram shows an example of the pipeline architecture:

Pipeline high-level architecture

The data sources are domain data related to each business unit and processed by different processors in the system. Engineers primarily use Apache Flink for real-time data processing. The connectors are the main element to start the data transfer. They monitor data sources and produce change data capture (CDC) events on the data mesh. Apache Kafka acts as the core component for the data movers in the data mesh. The data schema and catalog are very important in providing searchability and visibility of the data across different business domains. Netflix uses Apache Avro as the standard scheme between domains.

See also  Sound sick? New AI technology could tell if it's COVID

Many companies have already started to switch to this paradigm based on their needs. Chris Damour, one of the readers mentioned in the comments of the blog post:

We have also implemented the features to track data lineage so our users can get a better idea of ​​overall data usage. I’d love to see a blog on that, data provenance has been a real struggle in our data meshes, we get “same thing”.

Companies are now planning and changing their data platform around this architecture, including Intuit and Zalando. Many cloud providers like Amazon, Google, and Microsoft also provide data mesh solutions and services for their customers.

Leave a Comment