I have not used celery + rabbitmq but I assume that combo is like sidekiq + redis, or any other job queue + worker system.
Airflow packages those things together and adds some additional features
- UI with Graph, gantt, logs and other views of the workflow
- Users and permissions
- Places to store config
- Mechanisms for passing small data between tasks
- Various "sensors" for triggering workflows
- Various operators that interact with common data-oriented systems (bigquery, snowflake, s3, you name it). These are basically libraries that expose a config-forward API.
Probably the main selling point is the pre-made operators, but in short it is a complete solution with bells and whistles that aligns itself with the data ecosystem.
Airflow packages those things together and adds some additional features - UI with Graph, gantt, logs and other views of the workflow - Users and permissions - Places to store config - Mechanisms for passing small data between tasks - Various "sensors" for triggering workflows - Various operators that interact with common data-oriented systems (bigquery, snowflake, s3, you name it). These are basically libraries that expose a config-forward API.
Probably the main selling point is the pre-made operators, but in short it is a complete solution with bells and whistles that aligns itself with the data ecosystem.