How To Pass The Script Path To %run Magic Command As A Variable In Databricks Notebook?

I want to run a notebook in datarbricks from another notebook using %run. Also I want to be able to send the path of the notebook that I'm running to the main notebook as a paramet

Solution 1:

You can pass arguments as documented on Databricks web site: https://docs.databricks.com/notebooks/widgets.html#use-widgets-with-run

In the top notebook you can call

`%run /path/to/notebook` $X="10" $Y="1"

And then in the sub notebook, you can reference those arguments using the widgets API as in

x_value = dbutils.widgets.get("X")
y_value = dbutils.widgets.get("Y")

To your specific question, it would look something like this where "path" is the variable to be referenced via the widgets API in the target notebook:

 %run /path/to/notebook $path="/path/to/notebook"

Solution 2:

Magic commands such as %run and %fs do not allow variables to be passed in.

The workaround is you can use dbutils as like dbutils.notebook.run(notebook, 300 ,{})

Solution 3:

Unfortunately it's impossible to pass the path in %run as variable. You can pass variable as parameter only, and it's possible only in combination with with widgets - you can see the example in this answer. In this case you can have all your definitions in one notebook, and depending on the passed variable you can redefine the dictionary.

There will be a new functionality coming in the next months (approximately, see public roadmap webinar for more details) that will allow to import notebooks as libraries using the import statement. Potentially you can emulate the same functionality by exporting the notebook into the file on disk using the Export command of Workspace API, decoding the data & importing file's content, for example, if you have notebook called module1 with content

my_cool_dict = {"key1": "abc", "key2": 123}

then you can import it as following:

import requests
import base64
import os


api_url = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiUrl().get()
host_token = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiToken().get()

path = "/Users/..../module1"

# fetch notebook
response = requests.get(f"{api_url}/api/2.0/workspace/export",
    json = {"format": "SOURCE", "path": path},
    headers={"Authorization": f"Bearer {host_token}"}
  ).json()

# decode base64 encoded content
data = base64.b64decode(response["content"].encode("ascii"))

# write the file & __init__.py, so directory will considered a module
dir = os.path.join("/tmp","my_modules")
if not os.path.exists(dir):
    os.mkdir(dir)

with open(os.path.join(dir, os.path.split(path)[-1]+".py"), "wb") as f:
  f.write(data)
with open(os.path.join(dir, "__init__.py"), "wb") as f:
  f.write("\n".encode("ascii"))

# add our directory into system path
import sys
sys.path.append(dir)

# import notebook
from module1 import my_cool_dict

and see that we got our variable: