DeiC‘s HPC team has developed a container software tool that initially aims to make it more accessible for those running AI/ML in Python to use the LUMI supercomputer.
Words: Marie Charllotte Søbye, DeiC
If you’ve ever tried to work on an HPC facility as a researcher, you know that it can be quite a task just to get started. Suddenly, you need to be familiar with a Linux setup, command lines, package managers, etc., to a degree you may not have experienced before. For many, this can entail a lot of extra work because it’s not something most people are already well-versed in, even if they are experienced in the software development process.
In DeiC, we want to address this issue because it’s important that as many people as possible can access our HPC facilities, and IT challenges shouldn’t be what determines access to supercomputer resources. That’s why, at DeiC, we have developed a piece of software called “Cotainr”. The Cotainr tool ensures that you can run directly on the HPC facility via a single file, eliminating the need to download and install software on your laptop.
“Basically, it’s about the challenges of making your software available on HPC facilities. You have to download and install software on your own computer. But there’s an alternative, which involves using these containers, where you don’t install the software, but instead the software you need is in a file called a ‘container.’ You use a special program on the HPC facility to execute the software directly from that file, without having to install all the software locally,” explains Christian Schou Oxvig, Special Consultant at DeiC.
We want to help researchers get started more easily using our HPC facilities, especially the LUMI supercomputer. In simple terms, this means we want to move away from the idea that you have to acquire a lot of detailed knowledge about Linux, supercomputers, etc., just to use our facilities. The goal is that you can do something that closely resembles what you already know as a researcher.
First use case is Python environments with AI and ML on LUMI, and more to come
“Initially, we have developed a Cotainr solution for LUMI, so you can make your Python/AI/ML software easily accessible. That’s where we start because we know there can be challenges here,” continues Christian.
“If you work in Python, for example, with AI/ML, data science, or SA analysis, and you already use the tools called Conda and Pip to manage your Python environments, you will be able to quickly transition to LUMI and run from our Cotainr.” Eske Christiansen, Chief of HPC at DeiC, adds,
“We’re focusing on the Cotainr for LUMI, and as we see the need more use cases will emerge. Our clear goal is to spread the idea to as many systems as possible to simplify access to our HPC facilities.”
More information
- Contact: Eske Christiansen, eske.christiansen@deic.dk
- Code on GitHub: https://github.com/DeiC-HPC/cotainr
- Documentation on Read the Docs: https://cotainr.readthedocs.io/en/latest/
Read or download the full magazine here
