An ETL Framework for Operational Metadata Logging
Posted by: DI Soln
An ETL framework to capture all the operational meta data you need without adding too much time to your development cycle. It captures operational metadata such as record count, error count, job run time to compute performance statistics, reconciliation etc.
Overview
Quite often there is requirement to track runtime information about your ETL jobs such as record count, error count, job run time etc... In general it is a non-functional requirement, required by the IT team to have such information for reconciliation purposes, calculate performance statistics etc... This framework, can capture all the operational meta data you need without adding too much time to your development cycle. High level Design
Features
Reusable :
highly reusable design for effortless framework implementation.
Productive :
Adds minimal development effort for framework implementation.
1. High level design of a workflow with ETL framework implementation
2. Framework Components
2.1. Relational TableA relation table will be used to store the operational metadata with the structure as below. Data in this table will be retained for historical analysis.
2.3. Reusable Mapping
Reusable mapping to load the ETL load statistics to the relational table.
Adaptable :
Adaptable to any OS environment.
2.2. Pre &Post Session Command Tasks
Post session success command task used to capture session run details.
2.4. Reusable Session
Reusable session to load the ETL load statistics to the relational table
