Speaker
Magnus Karl Moritz Hagdorn
(Charité Universitätsmedizin Berlin)
Description
Task farms can be used to solve embarrassingly parallel workloads where a number of independent tasks need to be performed. This presentation introduces taskfarm, a python client/server framework that was designed to manage a satellite data processing workflow with hundreds of thousands of tasks with variable compute costs. The server uses flask to hand out tasks via a REST API and a database to track the progress of tasks. The client is also implemented in python. The presentation will focus on the software design process, the pitfalls and dead ends encountered when dealing with big data and how they were resolved.
Primary author
Magnus Karl Moritz Hagdorn
(Charité Universitätsmedizin Berlin)
