A Big Data Software Paradigm for Heterogeneous Cloud Deployments

Abstract

E2Data is an EU-funded research project aiming to provide a new Big Data software paradigm for achieving maximum resource utilization for heterogeneous cloud deployments without requiring developers to change their code. It essentially addresses the fundamental question of how modern Big Data frameworks can dynamically and transparently exploit heterogeneous hardware accelerators. The E2Data solution takes a cross-layer approach by allowing vertical communication between the four key layers of Big Data deployments: application, Big Data software, scheduler/cloud provider, and execution run time. For this purpose, four dierent use-case scenarios have been selected to prove the applicability of the project's approach: health analytics, natural language processing, green buildings and biometric security. To achieve those goals, Apache Flink and the Tornado VM are used in the project as the building blocks for enabling heterogeneous execution for Big Data frameworks.