AN ETL FRAMEWORK: FOR IMPLEMENTATION OF A MODERN DATA WAREHOUSE FOR REAL TIME ANALYTICS

Abstract

Data warehousing is the concept of storing historical data to get the insight of past and guidance for future. Previously data usually generated in relational formats, hence it was easy to manage them for analytics purpose. Nowadays data comes in verity of forms such as structured, semistructured and unstructured forms, and that makes it difficult to store them into data warehouse and analyze them. Recently, NoSQL databases have become popular because of semi-structured and unstructured handling capacity. Many data warehouses are available to manage structured data but there are few or very less research done in the field of NoSQL based data warehousing. The proposed solution provides an ETL framework with automated pipeline for modern data warehouse which can store and manage semi-structured data. To implement the proposed research authors have used MongoDB-document oriented database to store semi-structured documents. MongoDB also adds benefits of scalability and schema free structure for storing and accessing data. Moreover in the proposed work machine leaning techniques are used for accurate prediction. This analytics ready data at data warehouse reduces the execution time and provides real time analytics. The work is validated through the experiment and it shows the improvement in execution time with compared to manual process.