Efficient Coflow Transmission for Distributed Stream Processing

摘要: Distributed streaming applications require the underlying network flows to transmit packets continuously keep their output results fresh. These will become stale if no updates come, and staleness is determined by slowest flow. At this point, coflows can be semantically comprised. Hence, efficient coflow transmission critical for applications. However, prior coflow-based solutions have significant limitations. They use a one-shot performance metric—CCT (coflow completion time), which cannot reflect of application.To end, we propose new metric—coflow age (CA), generated distributed The CA tracks longest time-since-last-service among all in coflow. In such context, consider data center with multiple that between source-destination pairs address problem minimizing average long-term while simultaneously satisfying throughput constraints from coflows. To solve efficiently, design randomized algorithm drift-plus-age algorithm, show they make achieve nearly two times arbitrarily close optimal value, respectively. Through extensive simulations, further demonstrate both proposed algorithms significantly reduce coflows, without violating requirement any coflow, when compared state-of-the-art solution.