Loco Data Engineering Case Study Tanuj Soni

Provided are my submissions for the assignments where each code is havinng some documentations stating the logics utilised by me solve this case study.

Part 1 : Creation of Pyspark Script to calculate Clip interests

• Logical Code for Clips
• Constants and path definition file

Part 2 : Data write to s3 or Local

• Logical Code for Writer class to write in S3/local

Part 3 : Glue Jobs

• Glue Job 1 : Clips Logic Execution
• Glue Job 2 : Interests data to Database Dump Job

Part 4 : Local Execution

• Local Execution script for provided files

Part 5 : Unit Tests

• Unit Testing Class
• Unit Test Cases definition Class

Part 6: Airflow Orchestration

• Airflow Orchestration of Glue Jobs