Espire Optimizes Data Warehouse for Resolution of Data Overload Issues of a Global Engineering Conglomerate
Business Requirement
The existing BI Data warehouse (DWH) of the client involved multiple scheduled ETL data loads from source systems such as PRS and Workbooks. This data was then converted and calculated into multiple relational databases and the cube for retrieval within the DWH. The scheduled loads were usually run overnight and during some specific times in the day. The Insight reporting application is the reporting layer which sources the data from the DWH for end user reporting.
Business Solution
Parallel Processing of tasks on multiple queues : The existing solution offered only one processing queue that received requests sequentially. A total of four queues were implemented, each for a dedicated set of business processes: PRS and Workbook loads, Operating Model Loads, GL and Budget Uploads, and a fourth queue that can be used for all other purposes (static refresh, cube process, CRM upload and any other new processes that may be implemented)
Business Benefits
- Better data refresh rate
- Ability to prioritize on demand and period end processes on data processing queues
- Synchronized data sets within the Data Warehouse
- Separate thread of execution of different queues for different purpose of data