Please help, how can I use Semantic Link Labs library in a notebook inside a Fabric Pipeline?

[davidfabricdev]

I wish to use the funtions in semantic link labs in a PySpark notebook which will be an activity within a Fabric pipeline;

So, to start here is the notebook, please excuse the distracting commented-out lines where I have been experimenting.  I am running the code inside the notebook, no pipeline involved at this stage, and it works fine.  Note (highlighted) default environment and %pip install.

If I run this inside a pipeline I get this error;

Fair enough, I need to work around the %pip install command which I can't use in this context.  So after some googling I have loaded the semantic link labs library into an environment as a public library.

Now I change my notebook to use the new environment and remove the %pip install command;

When I run the pipeline again I get this error;

If I run %pip list the library is indeed loaded.

It seems that I simply don't understand how to reference it in my notebook.

Now that the library is loaded into the environment do I also need to run an import command?  If so, what is the name of the thing I need to import?  If I don't need to import in the notebook then what would the correct syntax be for the command "

labs.admin.scan_workspaces()"

This is driving me mad though I'm sure the answer is very simple.  Please help!

Thank you in advance.

[v-mdharahman]

Hi @davidfabricdev,

Thanks for reaching out to the Microsoft fabric community forum.

You're right that %pip magic commands work in interactive notebooks but are not supported when the notebook runs as part of a Fabric pipeline. You're also correct in trying to use a custom environment with the library pre-installed.

From what you've described and shown you’ve successfully added semantic-link-labs as a public library in your environment (FCMEnvironment). %pip list command confirms the library is present. But after all this you are still seeing ModuleNotFoundError during pipeline execution.

To resolve it, the key issue is likely with the import statement or the module name you're using in the notebook.

Even though the library is installed in the environment, you still need to import it manually in your code. The correct import for the semantic-link-labs package is "import semantic_link_labs as labs". Then you can call "labs.admin.scan_workspaces()".

Make sure %pip install ... is fully removed from the notebook. And set the notebook environment to your custom one (FCMEnvironment), which you've already done. Now add the import semantic_link_labs as labs line at the top of your notebook. Then call your function as labs.admin.scan_workspaces().

If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

Best Regards,
Hammad.
Community Support Team

If this post helps then please mark it as a solution, so that other members find it more quickly.

Thank you.

[davidfabricdev]

Many thanks for your reply Hammad.

Unfotunately I still can't get my notebook to work so any further help you can provide will be greatly appreciated.

Here is the error when I run the notebook directly;

And here is what happens when I run it in a pipeline;

And just to confirm, here is the environment;

Do you have any further thoughts on this?  I'm also wondering if there is a definitive way to discover the the import name of a package?

Thanks

David

[v-mdharahman]

Hi @davidfabricdev,

Even after going through all trouble-shooting steps the issue still persists, you can reach out to Microsoft Support Team by raising a ticket with Microsoft Support.

Please refer below link on how to raise a contact support or support ticket.
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

Best Regards,

Hammad.

[v-mdharahman]

Hi @davidfabricdev,

I wanted to follow up on our previous conversation. Has your issue been resolved or were you able to raise a support ticket with Microsoft Fabric Support?

Your feedback would be greatly appreciated, as it can help us better understand and address similar issues in the community.

Thank you.

[davidfabricdev]

Not as yet, I've not yet followed up with Microsoft.  Are you are able to answer my question about finding the import name of a package, any package, not just the one I am trying to use here?

For example, azure-core (at the bottom of the list below)

when I run "import azure_core as azc" I also get the module not found error

It seems that the wider problem is that I can't import modules with a hyphen in their name.  Using the hyphen causes a syntax error and replacing with underscore results in module not found.  I suspect the issue isn't specific to the semantic link labs package.  Is there a way to trouble shoot this issue?

[davidfabricdev]

So just to confirm the specific problem, from the %pip list results, I can import any module which does not contain a hyphen, but I can not import any of the modules which do contain a hyphen, even if I use functions like "__import__" or "importlib.import_module".  There must be a way to import these modules.