importing a js library fails in Jupyter Hub, works in Jupyter Lab

[mlamoureux]

I'm creating an "anywidget" to access some hardware devices known as "Phidgets."
This requires importing a library of js code in the _esm file for the anywidget.
My import statement works fine in Jupyter Lab, but not in Jupyter Hub (the classic Jupyter notebook).
My question is: what is the proper way to import a js module into the _esm file?

This statement works in Jupyter Lab, but not in Hub (i.e. not in classic notebook):
_esm = """
import "https://unpkg.com/phidget22/browser/phidget22";
....

I've also tried the following, based on the examples in the anywidget docs:
import * as phidget22 from "https://unpkg.com/phidget22/browser/phidget22"; or
import USBConnection from "https://unpkg.com/phidget22/browser/phidget22";

But they don't work at all. So, what is the right way to import?

[manzt]

My import statement works fine in Jupyter Lab, but not in Jupyter Hub (the classic Jupyter notebook).

Thank you for raising this issue. The problem you're encountering likely stems from the way Jupyter Widgets are installed and recognized within the Jupyter ecosystem, rather than an issue with your code.

Jupyter Widgets need to be detected by Jupyter upon kernel startup, meaning you have to restart the kernel to load new widgets. This is a known limitation, distinct from how regular Python libraries are installed. For regular libraries, installation in user-land suffices, but widgets need to be installed in Jupyter Hub itself.

I think I had with @hamelin at SciPy earlier this year about this very issue with classic Jupyter Widgets. Since anywidget is packaged as a classic Jupyter Widget, it must be installed in Jupyter Hub for all users. However (I believe) once anywidget is installed successfully, any custom widget developed using anywidget can be installed and managed in user-land.

To resolve this, you likely need need to install anywidget into Jupyter Hub as described for ipywidgets:

• Enabling widgets on jupyterhub for all users jupyter-widgets/ipywidgets#1949

[manzt]

There might be two issues here:

1.) The difference between JupyterLab and Jupyter Hub for importing anywidget (outlined above)
2.) "unpkg.com/phidget22/browser/phidget22" is not an ECMAScript Module (ESM)

Anywidget relies on the browser's native module system (ESM). The specifier you provided isn't a valid module specifier (in this case, not a full URL) and the code it points to isn't ESM (doesn't include import / export syntax). You'll either need to publish a version of the JS code that is ESM (to import from unpkg CDN) or use a CDN that modifies the code to be a proper module like esm.sh:

import * as phidget from "https://esm.sh/[email protected]";

export function render({ model, el }) {
}

[hamelin]

I think I had with @hamelin at SciPy earlier this year about this very issue with classic Jupyter Widgets. Since anywidget is packaged as a classic Jupyter Widget, it must be installed in Jupyter Hub for all users. However (I believe) once anywidget is installed successfully, any custom widget developed using anywidget can be installed and managed in user-land.

To resolve this, you likely need need to install anywidget into Jupyter Hub as described for ipywidgets:

* [Enabling widgets on jupyterhub for all users jupyter-widgets/ipywidgets#1949](https://github.com/jupyter-widgets/ipywidgets/issues/1949)

I would like to echo this from a slightly different perspective. Jupyter Widgets require Javascript extension code that Jupyter Lab must know where to find. In a typical standalone Jupyter Lab install — a virtual environment (Python venv, Conda environment or what have you) where package jupyterlab has been deployed — one deploys their widget package in the same environment as where Jupyter Lab lives. Jupyter Lab does find the JS extension code, loads it, everything works, the sun shines, the birds sing.

In a Jupyterhub installation, things get slightly different. Jupyterhub is deployed in some admin virtual environment, and made to run with enough power to authenticate users logging in. It includes Jupyter Lab in its environment, and when a user logs it, it spawns Jupyter Lab by running process jupyter-labhub. Now, users typically customize the computing environment to their needs, and the mechanism to do so is to set up a custom Jupyter kernel out of a virtual environment of their own, distinct from that where Jupyterhub and Jupyter Lab have been installed. Computations initiated by running the cells of a notebook are relayed to this kernel process — an instance of ipython in the most common case. Although this kernel can be made visible to their Jupyter Lab instance, if one installs packages such as Jupyter Widgets in the kernel's specific virtual environment, its JS extension code is not seen by Jupyter Lab. Changing the kernel of a notebook does not augment the set of directories Jupyter Lab scans for extension JS. But frustratingly, things look like they should work, because the packages are still importable from the notebook and thus the kernel! But their JS does not load and things fail to display in mysterious ways.

Thus, @manzt comment is right: one makes it possible to run extension widgets in Jupyterhub by installing the widget packages in the environment from which Jupyter Lab is spawned, which typically corresponds to the one where Jupyterhub is installed. Then custom user kernels also install the widget packages, so as to be able to import them. When they do, the extension JS in the Jupyterhub environment gets seen and loaded, and things work as expected. This also means that folks must do their best to install the same version of the widget package in their kernel environment as is deployed in the Jupyterhub environment. Discrepancies in conventions and expectations on either side of the Python/Javascript divide yield further weird and frustrating issues.

[manzt]

Discrepancies in conventions and expectations on either side of the Python/Javascript divide yield further weird and frustrating issues.

Couldn't have put it better myself. Thank you for such a clear description of the problem. I wonder where we could continue this discussion in the context of JupyterHub. Ideally there is a world where each user customizes their own widgets like they do their other dependencies.

[hamelin]

Discrepancies in conventions and expectations on either side of the Python/Javascript divide yield further weird and frustrating issues.

Couldn't have put it better myself. Thank you for such a clear description of the problem. I wonder where we could continue this discussion in the context of JupyterHub. Ideally there is a world where each user customizes their own widgets like they do their other dependencies.

I believe the jupyter-labhub invocation can be customized so as to be sourced out of a user-specific environment. But Jupyterhub certainly does not encourage that kind of setup when deployed.

[manzt]

From anywidget's perspective, I am committed to reducing this friction as much as possible (without being able to change how the Jupyter ecosystem and widgets work). Only anywidget's JS extension code needs to be discovered by Jupyter Lab/Jupyterhub. The frontend code for widgets derived from anywidget are served from the user's customized Jupyter kernel.

I've intensionally stripped back anywidget's frontend API (#138) to ideally ensure wider compatibility in the future. This means that users should be able to upgrade/downgrade their own custom widget that rely on anywidget, since only anywidget needs to be shared across users. I know it's not a satisfactory solution, but hopefully something that in practice will lead to less frustrations/confusion.

[mlamoureux]

Holy smokes, I think I solved the problem. This import statement works in both J Lab and the classic notebooks:

import {USBConnection, VoltageRatioInput} from "https://esm.sh/[email protected]";

Curiously, this next one does not. (suggested earlier in the comments above)

import "https://esm.sh/[email protected]";

The solution seems kind of random to me, but I am very grateful to all the suggestions you folks gave, as they were very insightful and gave me a lot of options to experiment with.

[manzt]

I'm glad that you were able to sort out your problem.

Curiously, this next one does not. (suggested earlier in the comments above)

import "esm.sh/[email protected]";

To be clear, this import does work, but you haven't actually imported anything into your code. Trying to access an exported member will throw a ReferenceError. In JavaScript modules, imports are explicit to avoid namespace pollution.

Also this exact code isn't suggested anywhere in this comment thread. The snippet I shared uses a namespace (i.e., * as phidget):

import * as phidget from "https://esm.sh/[email protected]";

which is a syntax is used to import all exported members from a module as a single object, in this case referred to as phidget.

phidget.USBConnection
phidget.VoltageRatioInput

I recommend reading about how modules work in the browser to learn more!

[manzt]

I'm closing this issue to cleanup the issues in anywidget since the OP was resolved. Thank you for chiming in @hamelin. Hopefully we can elevate this discussion elsewhere to improve widgets.

[maartenbreddels]

Yes, that has been quite a nightmare over the years, in solara we do some checking for classic notebook (i've tested this on AWS sagemaker):

If we find that the python executable != server executable: https://github.com/widgetti/solara/blob/7bf2ce53c7ede6eec3883abdc7d5f2625e22ccd1/solara/checks.py#L199

We inject the following, and give a proper error msg what to do: https://github.com/widgetti/solara/blob/master/solara/checks.html

Feedback on this is welcome, and ideas how to upstream this as well.

[paddymul]

FWIW In my own IPYwidget library buckaroo, I addressed some of this.
Users frequently pip installed the library without restarting the server. This causes a JS error with no python error. I added a check on the MTime of the python library, and the start time of the server.

https://github.com/paddymul/buckaroo/blob/main/buckaroo/widget_utils.py#L37-L55

I'm interested to hear about other approaches around this problem. It's a really poor experience for an initial user install. Maybe you could source the JS from unpackage when the user would encounter the problem.

It would be great to have a standard best practice for dealing with this for all widget libraries.