Data engineering on a lakehouse used to mean a lot of moving parts. A notebook for each transformation layer. A Data Factory pipeline to run them in the right order. Custom validation logic across scripts. Schedules to maintain and a Monitor Hub to watch when things break. Five different surfaces and five different things to […]
Tag: data engineering
Transfer learning in Spark for image recognition
Transfer learning in Spark demystified in less than 3 minutes reading Businesses that want to classify a huge set of images in batch per day can do this by leveraging the parallel processing power of PySpark and the accuracy of models trained on a huge set of images using transfer learning. Let’s first explain the […]
The Journey of attaining the Azure Data Engineer certificate
On February 23, 2021, Microsoft released a new beta certification exam, Exam DP-203: Data Engineering on Microsoft Azure. It is replacing the exams DP-200: Implementing an Azure Data Solution, and DP-201: Designing an Azure Data Solution. These previous exams DP-200 and DP-201 will retire on June 30, 2021. When passing the two old exams or […]