Configure a custom env on Azure ML
Shared workspace for remote AI Teams
When your team is working remotely, they need to collaborate and work on a shared cloud-based workspace. In this way, all developers in your team members can use it to run the experiments.
My team and I in MelkRadar, have a nice experience working with Azure ML. In this platform, you are able to import a wide variety of predefined environments and delegate your tasks on the Azure computes. Fortunately, the Azure ML designers have prepared some predefined environments from the most useful and popular packages to make it more straight forward for developers. You can easily find a list of these predefined environments based on your compute type in the Azure ML platform.
Customizing packages on a predefined env
If you are a ML developer, you are familiar with Anaconda package manager. It's being used to create your local environment and install required packages. If it doesn't work, you may also know how to create a virtual env on your local machine to do so. But when it comes to remote teamwork, it's a totally different challenge!
In this case you will actually need to install your own package(s) through a customized environment on that machine. Here is my experience to handle such situations.
At the beginning of the project, it was Ok using the pre-defined env until I tried to work with some packages which was specially designed to work with a specific language. To be clear, I was working with Persian texts which has its own libraries for preprocessing tasks. I needed the Hazm library to preprocess the Persian texts. I could easily add it to the Anaconda environment and work on local machine. But working with Persian text are not as much popular as English ones. So, it won't be found in the predefined environments on Azure Machine.
The challenge was to customize the predefined environments on Azure. On the way to handle this issue, I found that Azure ML has provided curated-env for this job.
First you can prepare a must to install packages and their versions in a yml file. Then by adding some lines to your code you can say the workspace to create this environment on the Azure machine.
Here are some snippets you will get an insight about this topic:
from azureml.core import Environment
from azureml.core.runconfig import DockerConfiguration
myenv = Environment.from_conda_specification(name='azure-custom-env', file_path='./conda_dependencies.yml')
myenv.docker.base_image = 'mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04'
docker_config = DockerConfiguration(use_docker=True)
And the initialize the ScriptRunConfig with this new env:
from azureml.core import ScriptRunConfig
src = ScriptRunConfig(script='script.py',
compute_target=cluster,
docker_runtime_config=docker_config)
After starting the run, you will find a link to the installed curated environment.
Modifying packages and versions
If you define the environment once and don't change the packages or their versions, the azure compute will use the first install env. But if you add or remove packages or change their versions, the azure machine will consider it as a new env and will install a new environment.
There is also another way for environment management knows as system-managed and I will talk about my experience using this way in the future.
Our Experience at MelkRadar AI Team
I am an AI developer at MelkRadar, which is a real estate search engine in Iran. We are using Azure ML as our main platform to collaborate with AI team members. In my recent project, it was very crucial to handle the customized environment for my experiments and this feature really helped me, so shared my experience to help you as well :). You can find more information in this link:
Top comments (0)