How to Use Configuration files in databricks
Hello Folks,
Hope you all are doing well & are safe!
Recently while working on one of the projects I was tasked to read the attributes/parameters from a config file instead of the widgets which I was doing in my previous iterations (Do let me know in responses, if you would like to learn that as well). I did some research and this is the approach I followed:
My config file is situated in a folder on my ADLS, but for experimentation I tried it in my community edition.
Check out on how to setup your own databricks community edition for experimentation.
- For this experiment I created a simple config file, with a lot of headers, & uploaded using the file > Upload data from DBFS > Drag & drop a file.
- Next thing we need is a configpaser library which would help with the reading/parsing and the whole shebang.
- Insitiate the object & use the read method to read the configuration file.
import configparser
# Instiate configParder
config = configparser.ConfigParser()The structure follows a dictionary like structure for example I have to read the host for mysql header than I would use the following syntax:
Host = config['mysql']['host']4. We need to copy the config file to local file system, in order to read the file.
# read the configuration file
# found the solution here: https://stackoverflow.com/questions/66684925/unable-to-read-configfile-using-configparser-in-databricks
# copy the file to the local file system
dbutils.fs.cp('dbfs:/FileStore/shared_uploads/dawar.rohan@gmail.com/sample_config_file-1.txt', "file:///tmp/sample_config_file-1.txt")
config.read('/tmp/sample_config_file-1.txt')5. Read the path from configuration file
path1 = config.get('My Section', 'path1')You can check the full code here
Reference:
