Earlier this month JupyterLab 3.0 was released, providing a slew of new features. In this blog post, I recap a few highlights and how the Elyra 2.0 extensions take advantage of them.
New to Elyra? Take a look at the overview documentation. TL;DR: Elyra provides productivity features for common data science and AI tasks.
Support for Language Servers
Many modern integrated development environments (IDEs) simplify development by providing editor features like autocomplete, "go to definition", "find references", or linting.
JupyterLab optionally supports these features by integrating with Language Servers using the jupyterlab-lsp extension. Language Servers are available for many languages, including Python and R. Note that some Language Servers might not support every convenience feature.
Elyra 2.0 includes the Python Language Server, enabling these features in the notebook editor and Python script editor, as shown below.
You can add support for other languages by installing the appropriate Language Server and its prerequisites.
Visual debugger
JupyterLab includes a visual debugger front-end that provides common debugging tasks. For example, you can set breakpoints, step through code, or inspect data. The front-end requires a kernel with support for debugging.
As of January 2021, the only Jupyter Python kernel that supports debugging is xeus-python.
Note that this kernel currently does not support all iPython features, such as cell magics.
Neither JupyterLab 3.0 nor Elyra 2.0 install xeus-python by default. You therefore have to install it separately from PyPI, conda, or build from source code.
Debugging a notebook
To debug a notebook, associate it with a kernel that supports debugging and enable the debugger. The official debugger example Python notebook provides a short introduction and is highly recommended!
Debugging a Python script
Elyra's Python editor is also enabled for debugging. To debug a Python script:
- Open the script in the Python editor.
- Right click on the canvas and select
Create Console for Editor. - Select a kernel that supports debugging.
- Enable the debugger in the console.
- Set one or more breakpoints in the source code.
- Run code to breakpoint and step through the code.
Elyra 2.0 also includes several enhancements that are not specific to JupyterLab 3.0.
Pipeline editor enhancements for Kubeflow Pipelines
Elyra's visual pipeline editor, which I introduced in this blog post, was extended to support custom resource allocation and the Tekton workflow engine.
Editor's note: Time flies! Elyra 2.1 (and later versions) also support running pipelines on Apache Airflow.
Customize resource allocation
When a pipeline node is executed on Kubeflow Pipelines, it is processed in a container. The amount of CPU, GPU, and memory (RAM) resources that are allocated for the container is governed by configuration settings in Kubernetes.
With the latest enhancement in Elyra, you can request additional CPU, GPU, or memory resources to speed up processing of your notebook or Python script during pipeline execution. You configure those resource requirements in the node's properties view:
- Select the notebook or Python script node in the pipeline editor.
- Right click, and select
Properties. - Enter the desired values.
If you leave the properties empty, the Kubernetes default settings are applied.
Support for Tekton workflow engine
Kubeflow Pipelines utilizes a container-native workflow engine to execute pipelines. As of January 2021, Argo and Tekton are the only two supported engines.
When you run (or export) a pipeline in the Elyra pipeline editor, it is compiled into a workflow engine-specific format. In previous releases, Elyra could only produce Argo-compatible outputs, but this restriction is lifted in version 2.0. Since Elyra cannot automatically determine which workflow engine your Kubeflow Pipelines installation leverages, you need to specify the type when you create a runtime configuration for your Kubeflow Pipelines deployment.
If you don't specify an engine type in your configuration, Argo is assumed as default to retain backward compatibility with older Elyra releases.
If you are unsure what engine your Kubeflow Pipelines deployment utilizes, check with the administrator. If you select the wrong engine type, your pipelines will fail to execute.
Code snippets
Elyra 2.0 makes it easier to create code snippets. The code snippet editor now allows for the creation of code snippets from highlighted code.
To create a code snippet within an editor, select the desired piece of code, right click, and choose from the context menu Save as Code Snippet.
To learn more about other enhancements or bug fixes that went into Elyra 2.0 please check the changelog.
Get started
If you are new to Elyra, it's easy to get started.
Install from PyPI or conda
You can install the Elyra JupyterLab extensions from PyPI, conda, or from source code. Refer to the installation guide for details.
JupyterLab 3.0 introduces a new extension type named prebuilt extensions. These extensions don't need a JupyterLab rebuild or a Node.js installation. Unfortunately, we are unable to distribute Elyra 2.0 as prebuilt extensions, but are considering doing it for future releases.
Run JupyterLab/Elyra in a container image
The Elyra community publishes Elyra container images on DockerHub and Quay.io. Take a look at these instructions to learn more about the images and how to run them.
The Elyra community publishes Elyra container images on DockerHub and Quay.io. Take a look at these instructions to learn more about the images and how to run them.
Get involved
During the past year we have received many suggestions and feedback from users like you. We appreciate your input and encourage you to stay involved. Open GitHub issues, reach out on gitter, post in our new forum, or join our weekly community meeting.
Use the forum to ask general questions or share things you've created, like Nick Burdakous did. He recently published a pipeline viewer for Google Chrome and Mozilla Firefox. This viewer visualizes pipeline files on GitHub, like shown in the screen shot below. The depicted pipeline was created by Nick Pentreath and presented at the Big Things conference in November 2020.
The Elyra community has lot planned for the first half of this year. Follow us here on dev.to/ibmdeveloper or on Twitter to stay in the loop.
Top comments (0)