In my previous thread, I discussed ETL and ELT pipelines.
Today, let's see 2 variations of Loading the data:
• Full Load
• Incremental Load
🧵
⭐ Full Load
When all the data coming in from the source systems is added to a data warehouse, it is a full load.
This is generally done for the first time after the data warehouse is created (initial load) or as per the use case.
⭐ Incremental Load
In this, we update/add only the new information from the source systems to a data warehouse.
Some of the incremental load patterns are:
1️⃣ Append
To append all the new information to a data warehouse, without modifying anything.
2️⃣ In-place update
Making changes to the existing data.
3️⃣ Rolling append
Maintaining a certain duration of history.
Example: Past 3 years of the employment history of employees.
Incremental load can be performed in Streaming and Batches.
Streaming Incremental Load can be used for loading small volume of data.
Batch Incremental Load can be used for loading large volume of data at once.
Both full and incremental load approaches can be implemented by any data loading tool like Spark, SSIS, Stitch etc.
That's it for today. 👋 Thanks for reading!