Writing Tests for SimEng Python Code conversion concepts 101

Writing Tests for SimEng Python Code conversion concepts 101, Write unit test for prevailing code in Python Pyspark SQL and configure tests in Pycharm.

Course Description

Writing Tests for SimEng – Code conversion concepts

Write unit test for prevailing code in Python Pyspark SQL and configure tests in Pycharm

What you will learn:

1. How to write unit test for prevailing code in Python Pyspark SQL
2. How to use Excel power queries
3. How to setup Pycharm, venv, unit testing, coverage for testing
4. How to write code for smaller and bigger function class task workflows
5. How to setup correct venv locally
6. Intro to code release process
7. Setting up local env variables

Topics:

1. In Excel power queries we use mostly group by and join on small data to check our results and share with seniors.
2. We can create a simpler notebook of the same logic of the code
3. A lot of type the errors are due to Local Spark sessions which can be created in many many ways, but if we want to create spark sql views which uses hive then we have to install a hadoop and spard by downloading them.
4. Debugging and creating break points and then using step into / step over to generate the output and match or save it with excel
5. Making sure we mimic spark from older tests because otherwise we can get error in Jenkins build