Use case: use Translytical Task Flows to update a file in lakehouse and refresh a delta table based on that file on which power BI report is built.
I wanted to refresh the delta table after updating the file using spark, but if I include any spark code I am getting the below invocation error.
{
"functionName": "test_spark_basic",
"invocationId": "xxxxxxx-b9bb-422e-8a80-xxxxxxxxx",
"status": "Failed",
"output": "",
"errors": [
{
"errorCode": "InternalError",
"message": "An internal execution error occured during function execution",
"properties": {
"error_type": "PySparkRuntimeError",
"error_message": "Java gateway process exited before sending its port number."
}
}
]
}
code I used for spark testing:
import fabric.functions as fn
from pyspark.sql import SparkSession
udf = fn.UserDataFunctions()
@udf.function()
def test_spark_basic() -> str:
# Create Spark session
spark = SparkSession.builder.appName("TestSparkInUDF").getOrCreate()
# Create sample Spark DataFrame
data = [("Bharath", 25), ("Anita", 30)]
columns = ["Name", "Age"]
df = spark.createDataFrame(data, columns)
# Collect result and convert to string
result = df.collect()
return "\n".join([f"{row['Name']}, {row['Age']}" for row in result])
questions:
1. can we use spark inside User Data Functions? If yes, pls provide a guide. (I can see pyspark module in library section)
2. Is there any other way to refresh the delta table after modification of file from UDF itself?
