## Generating the 300 SPARK configurations In order to generate the 300 spark configurations, we use the OpenTuner library. The available range of values for all the parameters can be found in ```src/python/spark-config-generator.py``` file. To generate the 300 configs we execute the following: ```bash mkdir configs for i in $(seq 1 1 300) do python3 spark-config-generator.py mv out configs/$i done ``` The process may take a while. The 300 configurations will be available in the configs folder. ## Known issues ### spark-config-generator.py This script generates a random SPARK configuration based on 107 SPARK parameters using the opentuner library. When running the script, you might get the following error: ```bash TypeError: Unicode-objects must be encoded before hashing ``` To fix this, first locate the installation path of opentuner package: ```bash python3 -c "import opentuner; print(opentuner.__file__)" ``` Then edit the following line in file `search/manipulator.py` ```diff @@ 858c858 @@ - return hashlib.sha256(repr(self.get_value(config)).encode('utf-8')).hexdigest().encode('utf-8') + return hashlib.sha256(repr(self.get_value(config)).encode('utf-8')).hexdigest().encode('utf-8') ```