Icon

exercises

4 workflows in this folder and its subfolders

Icon
01 ETL on new customer data - extract, transform, and load S3 and web service data

October 2021 · The company has two branches: one in the United States and another one in Europe. New US customer data comes from a web service. New Europe customer data comes from Amazon S3. We access, validate, prepare, and blend the data, anonymize it, and append it to the database table "customers". TAGS: Best…

Icon
02.1 ELT on website usage data - extract and load big data

September 2021 · This workflow sets up a local big data environment for the next exercise. It creates a local big data environment and loads the usage data into it. TAGS: Education,Data engineering,Data engineer,Best practices,big data,ELT

Icon
02.2 ELT on website usage data - transform big data on cloud and Spark

September 2021 · The company tracks the usage of the website and stores the information about different actions during each session: login and logout times, opened pages, clicked buttons, as well as the session satisfaction score (optional) and wants to calculate statistics for each customer, e.g., total number of v…

Icon
03 Data pipeline orchestration

October 2021 · Once the ETL on new customer data workflow is executed successfully, i.e., customer data are extracted and anonymized, and the database table "customers" is updated, the next process - ELT on website usage data - should be triggered. If the first workflow didn't execute successfully, the second one…