Join us for an expert-led overview of the tools and concepts you'll need to pass exam PL-300. The first session starts on June 11th. See you there!
Get registeredJoin us at FabCon Vienna from September 15-18, 2025, for the ultimate Fabric, Power BI, SQL, and AI community-led learning event. Save €200 with code FABCOMM. Get registered
Use case: use Translytical Task Flows to update a file in lakehouse and refresh a delta table based on that file on which power BI report is built.
I wanted to refresh the delta table after updating the file using spark, but if I include any spark code I am getting the below invocation error.
{
"functionName": "test_spark_basic",
"invocationId": "xxxxxxx-b9bb-422e-8a80-xxxxxxxxx",
"status": "Failed",
"output": "",
"errors": [
{
"errorCode": "InternalError",
"message": "An internal execution error occured during function execution",
"properties": {
"error_type": "PySparkRuntimeError",
"error_message": "Java gateway process exited before sending its port number."
}
}
]
}
code I used for spark testing:
questions:
1. can we use spark inside User Data Functions? If yes, pls provide a guide. (I can see pyspark module in library section)
2. Is there any other way to refresh the delta table after modification of file from UDF itself?
Solved! Go to Solution.
Hi @Bharath_Kumar_S,
Thank you for reaching out to Microsoft Fabric Community.
Spark is not supported inside User Data Functions, even though you may see pyspark in the library section. UDF’s run in a restricted python environment that does not include a Spark runtime, which is why you are getting the java gateway error.
If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it!
Thanks and regards,
Anjan Kumar Chippa
Hi @Bharath_Kumar_S,
Thank you for reaching out to Microsoft Fabric Community.
Spark is not supported inside User Data Functions, even though you may see pyspark in the library section. UDF’s run in a restricted python environment that does not include a Spark runtime, which is why you are getting the java gateway error.
If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it!
Thanks and regards,
Anjan Kumar Chippa
User | Count |
---|---|
12 | |
5 | |
4 | |
4 | |
3 |
User | Count |
---|---|
8 | |
8 | |
8 | |
7 | |
6 |