AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |
Back to Blog
Airflow docker connection12/9/2023 ![]() ![]() I have not been able to test and explore all of these images personally, but I tested the Spark image provided by Datamechanics and Bitnami Spark which works for the ETL workflow provided here. In our case, I was able to try the images provided by Bitnami Spark and Datamechanics, although these companies have the highest downloads on the Dockerhub, there are still many different options. ![]() Interestingly, different companies and individuals have published Spark images on Dockerhub, you can pull the images, and try them out to see if they address your development use case. You can aggregate the amount of RAM and CPU that would be needed by different running applications on your machine thereby managing resources.You can isolate them with Docker containers Even if the application requires different versions of dependencies to run independently. You have the ability to package multiple applications on the same machine, for example, you can run Spark, Kafka, and Airflow on the same machine, and all your deployments will work as expected.You can build, and deploy your development on AWS, Azure, GCP or any cloud platform, and expect that you would get the same result.You can share your development with your co-worker with all the necessary dependencies included, and set up the full application within a few minutes.Some of the advantages of using Docker container to run Spark jobs are For example, problems that relate to a different environment, dependencies issues e.t.c, thereby leading to fast development, and deployment to production. This is very important because, with Docker images, we are able to solve problems we encountered in development. In this blog post, we set up Apache Spark and Apache Airflow using a Docker container, and in the end, we ran and scheduled Spark jobs using Airflow which is deployed on a Docker container. ![]()
0 Comments
Read More
Leave a Reply. |