2021 Edition – Spark Scala coding framework , best practices and unit testing with ScalaTest

A bank wants to get insights from their Customer data.  Build a data pipeline using Hive , Spark and Postgres that will allow the bank to road data, apply all transformation and load it to a relational database from where it can be picked up by the Analytics team. Build your project following the steps mentioned below
1. Setup a Hadoop Hive environment on your local Machine
2. Install PostgreSQL Database on your local environment
3. Configure IntelliJ to have required dependency for Spark, Hive and Postgres
4. Load data into Hive table and Read Data from Hive Table
5. Apply transformation to replace Null Values in the data fetched from the Hive table
6. Store the transformed data into a PostgreSQL table

Ensure following coding best practices are followed

1. Logging – Try to implement Log4J and SLF4J
2. Error Handling – Ensure all errors are caught including System Error and Network Error
3. Managing Configuration with properties file – Capture all Transformation logic in a Config File, Capture all DB credentials in a public file 
Once you have successfully built the project please share it in the Student Gallery.  If you have implemented any other best practices related to non-functional requirement please feel free to share.