TL;DR

ansible-isilon eases the admin tasks on Isilon / PowerScale ; watch how cool it can be on Youtube and how to use it below.OverlayFS is great but has some limitations for some use-cases ; UnionFS is not dead !

Table of Contents

• The premise
• The implementation
  • Install Ansible modules for PowerScale/Isilon
  • The files
  • List usage in Ansible
  • UnionFS
  • File system removal
• Video

The premise

In my old days at the university, I used to work Sun Ray thin client (imagine the evolution between VT100 to modern VDI). Students and teachers were all connected to the same SPARC server to work. Each of us had its own home directory accessible from the NFS server.

More than 15 years later, enterprises of any size still use home directories on NFS for their users !

In the following article, we will show how to use Ansible to manage home directories hosted on a PowerScale array in a university.The predicate is that Active Directory is the reference for the userbase. Each LDAP user can be either in the student group or the teacher group.Any student or teacher in AD must have his homedir in PowerScale and be accessible via NFS exports. Any student who is no longer enrolled and not in AD will have thier homedir removed.

The ansible playbook will :

• get the list of students and teachers from AD
• create a unix home directory in PowerScale/Isilon for each user
• set different quotas if the user is a student or a teacher
• have daily snapshots of the home directories with varying policies of retention if for the students and teachers
• mount the home directories in a list of Unix server
• cleanup the home directories of students that are not in the AD anymore

The implementation

In this chapter I will not detail all the tasks as most of them are self-explanatory, but, describe a few tips & tricks that can be reused in other playbooks.

Install Ansible modules for PowerScale/Isilon

The Product Guide documents the module installation and usage (equivalent to ansible-doc dellemc_isilon_[module]).

This example comes with a Dockerfile that has the required dependencies to run the playbook.

As the ansible-isilon is very specific about Isilon SDK version, the most important line is :

RUN pip3 install isi-sdk-8-1-1 pywinrm && \
    git clone https://github.com/dell/ansible-isilon.git

Once docker build-ed, you can execute the playbook with it with :

podman run --security-opt label=disable -e ANSIBLE_HOST_KEY_CHECKING=False \
           -v ~/.ssh/id_rsa.emc.pub:/root/.ssh/id_rsa.pub -v ~/.ssh/id_rsa.emc:/root/.ssh/id_rsa \
           -v "$(pwd)"/homedir/:/ansible-isilon \
           -ti docker.io/coulof/ansible-isilon:1.1.0 ansible-playbook \
           -i /ansible-isilon/hosts.ini /ansible-isilon/create_homedir_for_ad_users_in_isilon.yml

Note that on my Fedora 32 machine, the --security-opt label=disable is mandatory to be able to mount the volumes.

The files

To use the playbook, you will have to update a couple of files:

• hosts.ini ; which has the inventory of Unix and Domain Controller
• credentials-isi.yml ; which has the details of the PowerScale
• create_homedir_for_ad_users_in_isilon.yml ; which is the playbook with all the tasks, where you have to update the variables base_path and nfs_server_ip in several sections to point to the PowerScale path and IP.

List usage in Ansible

The first tip is in task Get userbase from Active Directory with :

    - set_fact:
        students_list: "{{members_students_group.members | list}}"
        teachers_list: "{{members_teachers_group.members | list}}"

The set_fact creates two lists of users that will be reused across the playbook.With the object list, we can loop through and execute the same task for each user as done in the FS creation :

      dellemc_isilon_filesystem:
        <<: *isi_connection_vars
        path: "{{base_path}}/students/{{item}}"
        ...
        state: 'present'
      loop: "{{ hostvars['devconad.com']['students_list'] }}"

Or make it easy to find orphan homedirs by playing with list operations when listing unix mounted dirs :

    - name: Capture files in path and register
      shell: >
        ls -1 /mnt/nfs_students
      register: students_home_dir
      run_once: True
    - set_fact:
        orphan_home_dirs: "{{students_home_dir.stdout_lines | list | difference(hostvars['devconad.com']['students_list'])}}"

UnionFS

To stick with the usual /home/<username> file system hierarchy, I wanted to mount the students and teachers sub-dirs within the same /home and keep the write in the lower dirs as follow :

/mnt/nfs_teachers/ /mnt/nfs_students/ /home
├── alice ├── carol ├── alice
└── bob └── dan ├── bob
                                                ├── carol
                                                └── dan

The capability of writing in lowerdirs live is available in AuFS and UnionFS but not in the very popular OverlayFS.

As stated by the Kernel documentation:

Changes to the underlying filesystems while part of a mounted overlay filesystem are not allowed.

There are plenty of discussions about that topic on Stackoverflow.

To achieve it I used unionfs-fuse which is available from Ubuntu repo or CentOS third-party repo. The obvious advantage of Filesystem in Userspace is that I won’t need to recompile the Linux kernel to use it. In the /etc/fstab we can use unionfs# to mount a FUSE filesystem :

        line: "unionfs#/mnt/nfs_students=RW:/mnt/nfs_teachers=RW /home/ fuse cow 0 0"

File system removal

It is possible to remove PowerScale/Isilon file system with the Ansible directive :

    - name: Remove Filesystem and Quota for missing students from AD
      dellemc_isilon_filesystem:
        path: "{{base_path}}/students/{{item}}"
        quota:
          quota_state: absent
        state: absent

Note that by design, the Ansible module will only remove the directory if empty.If you need to remove a non-empty directory, you have to issue REST call directly.

Video

For a live demo, check the video here: https://www.youtube.com/watch?v=RF5WoeRry1k&list=PLbssOJyyvHuVXyKi0c9Z7NLqBiDiwF1eA&index=2