![]() We want to focus more on what kind of data we need and want! Airflow’s User Survey 2022 Today, we want to focus less on the technicality behind data orchestration. But this makes Airflow also less flexible. Also, one wants to mitigate bugs like infinite running jobs when two tasks would be called in a cycle. So Airflow’s choice to abstract the relation between tasks through a DAG is relatable. Tasks that have dependencies between each other. Back in 2015, it was completely normal to think about workflow orchestration in terms of tasks. To mitigate this problem, you can simply install Airflow on WSL or simply use Docker.Īirflow’s abstractions are also outdated. There were workarounds for some Airflow versions but they were too hacky. Undeniably, testing Airflow pipelines is far from easy, e.g., mocking requires knowledge of Airflow’s internal machinery.Īnd if you thought one second about running Airflow on Windows, forget it. ![]() Also, getting to know the most important operators takes time and patience. Another restriction which is posed by XComs is that it should be only used to exchange metadata. XComs which is responsible for the handling of task cross-communication feels unintuitive. ![]() Moreover, the approach that Airflow follows is not the most intuitive one. While Airflow’s documentation is verbose, it is still lacking details in a lot of areas like testing. You have to invest a lot of time when onboarding junior data engineers. There is no data asset awareness.įurthermore, Airflow’s learning curve is very steep. For instance, Airflow’s tasks are working like black boxes, Airflow does not know what happens inside these black boxes. Task: average revenue per manager aggregationĪirflow tries to market itself as a data orchestration tool but Airflow is a job scheduler with extras.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |