Clarisights is a SaaS platform that integrates, processes and visualizes marketing data for mid to large enterprises that run lots of campaigns across different social media. As part of its processing, Clarisights would pull in data from Google, Facebook, LinkedIn, custom databases on a regular basis. They run >10 million jobs to pull and process this data! These jobs had dependencies - some of them could run on after some others finished. They quickly outgrew simple cronjobs. Then they used Airflow to represent job dependencies. The static scheduling methods simply did not work for them - they had to adjust their schedules based on SLAs and failures.
So they ended up Mozek - a declarative engine for scheduling workloads. Do check out the full talk by Ashu, their CTO, linked below.
If you’d like to get an invite to join the next SaaS Engineering talk, please register here. If you would like to nominate someone to give a talk in this forum, shoot me a mail at siva@fylehq.com.