In this post, I’ll explain how to add the current date and time to an existing PySpark DataFrame in a Fabric Notebook. This is particularly helpful when inserting data into a Fabric Lakehouse table, as it allows you to track when each record was added. I’ll walk through an example using a DataFrame to first load some sample data and show how to append a new column with the current timestamp to capture the insertion time.
Continue reading
Data warehousing and business intelligence are crucial for modern enterprises, enabling them to make informed decisions based on historical data. One key aspect of data warehousing is handling changing data over time, especially when tracking historical changes in dimensions. This is where Slowly Changing Dimensions (SCD) come into play. In this blog post, we’ll delve into what SCDs are, the different types of SCDs, and how to implement them effectively.
Continue reading