Every company relies on some form of workflow organization, where they devise a mechanism or tool that helps them utilize the resources to the best of their ability, meet customer demands, facilitate easy software development, and more.
Not just big companies like Netflix, but every company has its unique set of problems and ideas to handle them. So, there is never a "one solution fits all" approach here.
However, if a company invests a significant amount of money to build their own workflow engines, and then they are open-sourcing it, that's good news! 🥳
Meet Netflix's Workflow Orchestrator 'Maestro'
If you are a cloud computing enthusiast, the term orchestrator may remind of Kubernetes or microservices. But, Maestro is not that.
As the official blog describes:
Maestro is a general-purpose, horizontally scalable workflow orchestrator designed to manage large-scale workflows such as data pipelines and machine learning model training pipelines.
Companies like Netflix need to collect data, process it, store it, transform it, verify, and monitor it. Anything that deals with data engineering, the same processes are involved.
In other words, the decisions to recommend you of your favorite shows, analyzing what works best, what doesn't, management of media, and any back-end trick, everything is data-driven (heaps of it). And, that is facilitated by using such a tool.
One could say Maestro is at the core of Netflix's platform, making all the gears work in sync together for its engineers to work comfortably, partners to get all insights needed, and customers to access the service seamlessly.
Netflix used Meson (another of its orchestrator) before it introduced Maestro in 2022 to address the scalability issue. But, it was not open-sourced.
Now, Netflix seems to be confident that its solution can help others, and in return, use the help of the open-source community to improve it further. So, they decided to open-source the entire code and make it available on GitHub under the Apache 2.0 license.
Some users in the Hacker News discussions seem to think that this is purely a PR move, and they did not need to re-invent the wheel when there are options like Apache Airflow.
Some are skeptical if Netflix would maintain it or abandon it later for something else, like Conductor (Netflix's older tech).
While others believe that this is a good move, and it could help the open-source community re-purpose the code to individual use-cases for different business logic.
I am not a data engineer, so I am not aware of how this is technically beneficial over existing solutions. So, I would let you find that out from their detailed announcement blog post.
💭 What do you think about Netflix's decision to open-source their latest workflow engine? Let me know your thoughts in the comments below!
More from It's FOSS...
- Support us by opting for It's FOSS Plus membership.
- Join our community forum.
- 📩 Stay updated with the latest on Linux and Open Source. Get our weekly Newsletter.
