Design and implementation of an efficient data stream processing system

摘要: In standard database scenarios, an end-user assumes that all data (e.g., sensor readings) is stored in a database. Therefore, one can simply submit any arbitrary complex processing the form of SQL queries or procedures to server. Data stream oriented applications are typically dealing with huge volumes data. Storing and performing off-line on this dataset be costly, time consuming impractical. This work describes our research results while designing implementing efficient management system for online streams field environmental monitoring. Our target sources wireless networks. Although focus specific application domain, thesis designed generic way, so they applied wide variety applications. starts by first presenting state-of-the-art specifically window concepts, continuous queries, filtering query languages in-network (particular TinyOS-based approaches). We present key existing engines, their internal architecture how compared platform, namely Global Sensor Network (GSN) middleware. GSN middleware enables fast flexible deployment interconnection It provides simple uniform access comprehensive set heterogeneous technologies. Additionally, offers zero-programming data-oriented integration networks supports dynamic re-configuration adaptation at runtime. virtual concept, which high-level view sources, its powerful declarative specification tools. Furthermore, we describe design, conceptual, architectural optimization decisions platform detail. order achieve high efficiency large streaming using window-based algorithms techniques intelligently group process different types queries. While adapting scale network deployments, have encountered several performance bottlenecks. One challenges faced was related scalable delivery rate streams. found out could dramatically improve processor grouping user hence sharing both memory costs among similar Moreover, issue scheduling Problem efficiently execution sliding parameters not addressed depth literature. problem becomes severe when considers these cases, scheduler only increases least magnitude but also, decreases response requirements. Finally, get integrated external visualization framework Microsoft's SenseWeb platform. gathering infrastructure globally accessible end users. (which initiated Swiss Experiment project demanded users) shows scalability combined optimized algorithms, also demonstrates flexibility.