With the growing use of computational workflows in data-driven science, reproducibility, scalability, and portability have become increasingly important. Platforms such as Galaxy have made computational analyses more accessible to non-experts by providing reusable workflows and supporting reproducible execution through software dependency and environment management. The rapid adoption of large language models (LLMs) and other computationally intensive methods has further highlighted the need for reproducible, transparent, and sustainable computational research. In this presentation, I will discuss best practices for managing software environments and developing reproducible workflows on Alliance clusters. I will also explore workflow portability and strategies for designing workflows that can be reproduced across diverse computing environments, including high-performance computing (HPC) systems, bare-cloud environments, and secure or isolated platforms.
The Amsterdam Modeling Suite (AMS) provides an integrated environment for computational chemistry and materials science, combining electronic structure methods, reactive molecular dynamics, machine learning potentials, chemical kinetics, and fluid thermodynamics.
AMS is used in a wide variety of research and engineering applications worldwide. In this seminar, we will highlight three key capabilities:
- Electronic structure calculations for chemical bonding analysis, reactivity, and spectroscopy.
- Automated reaction discovery, including the identification of reaction pathways, calculation of reaction rates, and construction of kinetic reaction networks.
- Reactive molecular dynamics, including the use, distillation, and fine-tuning of foundation machine learning interatomic potentials for specific materials and chemical systems.
Through practical examples, we will illustrate how these capabilities can be applied to research challenges in catalysis, batteries and energy materials, organic electronics, and semiconductor materials and processing.
In this course we will discuss cryptography, its relevance and importance in everyday computing. We will discuss the historical context as well as current applications of different forms of cryptography. We will cover symmetric encryption, asymmetric encryption as well as hashing. Lastly, we will put theory into practice and discuss how different cryptographic mechanisms are implemented in well known network protocols we use every day.
Git is the most widely used version control system today, underpinning commercial, private, and open source projects alike. Created in 2005 by Linus Torvalds to manage the Linux kernel, Git earned its popularity through speed, reliability, and a distributed design that gives every copy of a project its full history.
AI coding assistants have changed how software is written. Code can now be generated in minutes, but someone still has to understand it, review it, and stand behind it. As the cost of producing code falls, the value of managing it rises. Version control and sound DevOps practices are what make software traceable, reviewable, and recoverable when something goes wrong. For researchers, whose code underpins published results, that matters more than ever.
This webinar is a comprehensive introduction to Git, refreshed for current practice. We'll cover the fundamentals, including commits, branches, merging, and remotes, along with the habits that keep projects healthy in your development workflow.
Date: October 21, 12:00pm-1:00pm ET
Presenters: Ramses van Zon (SciNet)
Funding for a state-of-the-art artificial intelligence infrastructure at SciNet was announced in November 28, 2025 as part of the Canadian Sovereign AI Compute Strategy. This infrastructure is being installed as an expansion of the Trillium GPU cluster, and is one of the resources which can be requested for the 2027 resource allocation competion of the Digital Research Alliance of Canada. In this talk, we will deliver an update on this project. We will present the architecture of the AI expansion, ways in which it will be accessed, how it compares to the existing GPU resources, and what workflows are best suitable for it.
Join us for an introduction to Digital Research Infrastructure (DRI) and discover essential training pathways for new users. This colloquium outlines introductory learning opportunities and details how to access courses, resources, and support through Compute Ontario, SHARCNET, SciNet, CAC, and HPC4Health. Participants will also learn how to seamlessly navigate the broader Canadian DRI training ecosystem using Explora.
The scheduling environment on the national general purpose compute clusters is a dynamic system that adapts to various aspects of the supply of and the demand for the heterogeneous compute resources that they make available. This presentation is an update of the scheduling environment on these clusters including methods for understanding the changing states of the clusters, resources for exploring account usage of the clusters, recommendations for minimizing wait times and maximizing total usage of an account as well as ways to learn more about the scheduler’s behaviour.
While the Alliance has been using NVIDIA GPUs almost exclusively in the last few years, we do have a small test installation of AMD GPU and APU accelerators and may acquire more in the future. This seminar will focus on the AMD MI300As available on the Nibi cluster, which are APUs combining GPU and CPU components with unified memory. The seminar will also discuss the software packages for AMD accelerators (in particular AMD's ROCm/HIP stack as the analog to NVIDIA's CUDA) and how these can be installed on Alliance clusters.
Date: May 20, 2026 12:00pm-1:00pm
Instructor: Ann Allan
This colloquium will help new users navigate Compute Ontario Summer School Registration and Schedule. The session will cover creating a Compute Ontario Training Account, how to log-in, how to enrol in courses and some helpful tips about this year’s schedule. A brief overview of the summer school content will be reviewed and there will be the opportunity for new users to ask questions.
This year the school is taking place from June 1 to June 19 and offers a comprehensive curriculum packed with over 40 courses. Delivered by experts in the field, these sessions cover a wide range of topics including Advanced Research Computing (ARC), High Performance Computing (HPC), Research Data Management (RDM), and Research Software (RS). With presentations and workshops available at introductory to intermediate levels, there is something for everyone.
Many researchers approach the national compute clusters available through the Digital Research Alliance of Canada as if they were interchangeable, expecting the same workflows, software environments, and usage models everywhere. Yet the Trillium supercomputer at SciNet was designed with a different purpose and architecture than the other clusters. Understanding these differences can significantly improve productivity and research outcomes.
Trillium was built to support the largest scientific computations. This influences not only its hardware, but also how users access and use the system, despite the many features it shares with other clusters. In this colloquium, we will discuss the why and how of Trillium’s login requirements, job scheduling, and storage systems.
We will also explore which research computations and workflows are best suited to Trillium, and which may be better served by the general-purpose clusters. Finally, we will show how interactive work remains possible through Trillium’s debug partition and OnDemand service.
Simple Features is a standardized (ISO 19125-1:2004) architecture for geographic information (points, lines, and polygons). It is widely used is GIS systems (ArcGIS, QGIS, etc.), spatial databases (PostGIS, MySQL Spatial Extensions, etc.), and various libraries (GDAL, GEOS, etc.). This talk is an introduction to Simple Features for R (the sf package) in the context of the Tidyverse framework.
This workshop explores the use of artificial intelligence, especially large language models, for processing healthcare data such as electronic health records. Learners will examine how LLMs can enhance clinical decision support, automate documentation, and uncover patterns in complex datasets. The presentation highlights benefits such as improved efficiency and richer insights, while addressing risks including hallucinations, bias amplification, privacy concerns, and over‑reliance on automated summaries. Students will study pitfalls unique to healthcare data and gain an understanding of key technical challenges: data standardization, model validation, safety monitoring, and ensuring transparency in clinically sensitive environments.
Reproducibility and experiment tracking are essential in machine learning workflows. MLflow is an open-source platform for experiment tracking and model management in machine learning and AI development. This webinar introduces MLflow with quickstart examples running on the clusters, focusing on a lightweight setup with local storage. The examples will be demonstrated in Jupyter notebooks and in batch jobs.
Similar to disk space, inodes (the number of files on a filesystem) are a limited resource. Therefore, each user and group are allocated a fixed number of inodes, by default. In this webinar, filesystem quotas on the Alliance clusters and best practices for managing file quotas will be presented, including the use of archival storage. Inconsistencies in file ownership leading to “disk quota exceeded” errors will be discussed. Finally, file formats such as NetCDF and SQL BLOBs for effectively storing large sets of small files will be presented.
Learn how to create and manage virtual machines on SHARCNet's cloud infrastructure using OpenStack. This session covers the essentials of working with the OpenStack dashboard to launch VMs, configure security groups, manage storage volumes, and control your cloud resources. Whether you're setting up a web server, running custom software environments, or building virtual clusters, you'll discover how OpenStack gives you complete control over your computing environment.
The Canadian Bioinformatics Hub (CBH) supports the growth of bioinformatics and computational biology in Canada through training, mentorship, and community-building initiatives. This presentation introduces CBH and the resources it offers to students, early-career researchers, and industry professionals. We will highlight key programs within the training and community pillars and provide an overview of bioinformatics user groups across Canada, with a focus on Ontario. Join us to learn how CBH connects people, builds skills, and strengthens bioinformatics communities across the province and beyond.
In our last talk on Fully Sharded Data Parallel (FSDP), we offered insight into training large models using FSDP and strategies for customizing model training with FSDP for performance benefits.
PyTorch has an updated interface for Fully Sharded Data Parallel called FSDP2, here we will present how to implement FSDP2 in your training code, compare FSDP2 with FSDP, and examine training performance using FSDP2 on the new systems. Intermediate experience with Python, PyTorch and deep learning is expected.
This simulation will contain several advances relative to the widely used 1/48° MITgcm simulation (also known as LLC4320), including increased vertical and horizontal resolution, an updated global bathymetry, the use of a more accurate surface pressure solver, the addition of ice-shelf cavities around Greenland and Antarctica, hourly atmospheric forcing, realistic river discharge, and more accurate astronomical tides. These improvements directly address long-standing issues in earlier high-resolution MITgcm simulations, for example, a misplaced Gulf Stream, a crude representation of Antarctic shelf currents, and anemic tropical instability waves.
The resulting model output will offer an unprecedented benchmark for studies of internal tides and internal waves, turbulence parameterization, and sea-surface height variability. All configurations, tools, and outputs will be openly released, positioning this Canada-led effort as a major global resource for oceanography and climate modelling.
Date: October 22nd 12:00pm-1:00pm
Contributors: Sahar Naseer (Privacy Specialist, The Hospital for Sick Children), Roohie Sharma (Legal Privacy Counsel, The Hospital for Sick Children), Melissa Lanuza (Privacy and FOI, The Hospital for Sick Children)
This presentation provides a foundational overview of health privacy principles and legislation relevant to Ontario hospitals. It introduces the ten privacy principles that underpin Canadian privacy laws, with a focus on the Personal Health Information Protection Act (PHIPA), the Personal Information Protection and Electronic Documents Act (PIPEDA), and the Freedom of Information and Protection of Privacy Act (FIPPA). The session covers the rights and responsibilities associated with personal health information (PHI) and personal information (PI), the process for handling freedom of information requests, and the importance of cybersecurity awareness.
Date: October 15th 12:00pm-1:00pm
Presenters: Kristi Thompson (Research Data Management Librarian, Western University) and Alexandra Cooper (Data Services Coordinator, Queen’s University)
In early 2025 data and documents began to disappear from U.S. government web sites in response to a series of executive orders. This led to a scramble as various individuals and groups mobilized to save as much disappearing data as they could. These events served as a wake-up call and led to the founding of the Canadian Public Data Rescue Initiative (CPDRI).
Initially formed, in part, to support rescue efforts in the U.S., the CPDRI also set out to build infrastructure to support public data in Canada. Drawing on projects including the Canadian Government Information Digital Preservation Network and the OCUL Ontario Data Rescue Group, the CPDRI is working to establish a sustainable strategy for preservation of vital Canadian public datasets.
This colloquium will help new users navigate Compute Ontario Summer School Registration and Schedule. The session will cover creating a Compute Ontario Training Account, how to log-in, how to enrol in courses and some helpful tips about this year’s schedule. A brief overview of the summer school content will be reviewed and there will be the opportunity for new users to ask questions.
This year the school is taking place from June 2 to June 20 and offers a comprehensive curriculum packed with over 40 courses. Delivered by experts in the field, these sessions cover a wide range of topics including Advanced Research Computing (ARC), High Performance Computing (HPC), Research Data Management (RDM), and Research Software (RS). With presentations and workshops available at introductory to intermediate levels, there is something for everyone.
Checkpointing is a technique that enables programs to save their current state and resume execution from a saved state in the future. This mechanism is useful in running long jobs, which may be interrupted for various unpredictable causes, such as system failures (either hardware or software), bugs in the running program, timeout, etc.
We have a wiki page about checkpoints that only gives general guidelines. In this webinar, we will introduce checkpointing through a few concrete examples to illustrate what is the state of a program and how its states at different points of execution are saved and restored. We will discuss various topics related to checkpoints, such as saving frequency, checkpoint file types, and how to implement the checkpointing mechanism in different computational job categories: serial, threaded, and MPI.
MPI-IO is a set of extensions to the MPI library that enable parallel high-performance I/O operations. It provides a parallel file access interface that allows multiple processes to write to and read from the same file simultaneously. MPI-IO allows for efficient data transfer between processes and enables high-performance I/O operations on large datasets. In this seminar, we will talk about some MPI-IO advanced features such as file view, contiguous vs non-contiguous IO, hints info, independent vs collective IO, blocking vs non-blocking IO as well as MPI derived filetype and datatype.
With the popularity of Large Language Models and the general trend of scaling up model and dataset sizes comes challenges in training. Despite hardware improvements, many models are too large to fit onto a single GPU or large enough that small batch sizes lead to long training times.
One strategy for parallelizing training is Fully Sharded Data Parallel (FSDP), provided by PyTorch. This strategy splits models into shards and distributes shards across parallel GPUs. This strategy can be used to train very large models and to scale up training. In this talk, we'll discuss implementing FSDP in your training code, examine training performance from an efficiency perspective and compare with another parallelization strategy, data parallelism. Some experience with Python, PyTorch and deep learning is expected.
Python is often praised for its speed of development but criticized for its execution speed. However, this has changed significantly in recent years due to major improvements in both Python itself and its most popular libraries. Libraries such as TensorFlow, OpenCV, NumPy, and Pandas all leverage a tool called Cython. Cython is an extension of Python that allows functions to be compiled into C (or C++), mitigating Python's performance limitations.
Since the last webinar on this package in 2020, several major versions have been released. This talk will explore what has changed, whether performance has improved, and if development has become easier. During the webinar, a few demo problems will be explored live. Experience with Python is expected, while familiarity with C/C++ and Jupyter notebooks will be helpful.
The growing library of self-paced courses offered by SHARCNet provide materials that directly relate to research productivity on Alliance compute clusters. This colloquium presentation is a preview to a soon to be release self-paced course on job scheduling and monitoring resources on Alliance clusters. The material in this course includes introductory bash prerequisite to working with the scheduler, introduction to the role of the scheduler on a compute cluster, basics of using the SLURM scheduler, scheduling properties that are specific to the Alliance environment, methods for monitoring jobs on the clusters both from the terminal on the cluster and from portals being developed for users. This colloquium is not only intended to be an opportunity to discuss the content of the self-paced course, but also an opportunity to gather input from users about important issues to add to the course in its final revisions.
In this talk, we address the impact of rounding errors encountered but often ignored in scientific and high performance computing. We begin with examples in day to day research computing and then advance to an example of summation in parallel computing with MPI and OpenMP reduction. We wish to share our explorations and hope this talk may lead to future discussions.
This is a series of talks about using modern C++ for high-performance data science. In the first talk of the series, we talk a little bit about the pros and cons of using C++ for data science projects and then we cover Xeus-Cling (https://github.com/jupyter-xeus/xeus-cling ) and G3P (https://github.com/arminms/g3p) for rapid-prototyping of C++ codes and embedding plots and charts in a Jupyter notebook, respectively. The whole series will be available as an executable book (https://executablebooks.org/) at https://arminms.github.io/high-performance-data-science-with-modern-cpp.
Python's NumPy library is one of the standard ways for researchers to perform mathematical computations. With the wider availability and power of GPU resources, the need arises to convert NumPy programs to run on the GPU to improve their performance, ideally with as little modification as possible. This seminar will discuss CuPy, a library highly compatible with NumPy, which offers drop-in replacement for most NumPy (and SciPy) functions. The seminar will discuss the basic techniques used to convert a NumPy program to CuPy, emphasizing the good practices required to obtain an efficient code. Another more recently developed approach is the cuPyNumeric library from NVIDIA, which allows running NumPy code on the GPU with no code changes. The seminar will discuss how to install and run it on HPC clusters.
Computational research requires specialized skills not always taught as part of academic curricula. The training done by the Advanced Research Computing centers and organization in Canada and around the world fill this void. In this training landscape, the International High Performance Computing Summer School (IHPCSS) holds a special place. It is an expenses paid, in-person training event that brings together leading experts on High Performance Computing and Big Data Analytics and the students that need this knowledge. Uniquely, it has a strong mentoring component to give advice to students not only about technical issues but also on many other aspects of their studies and career. This year, the Digital Research Alliance of Canada will sponsor about 10 students to go to the IHPCSS in Lisdon, Portugal, July 6-11, 2025. In this talk, we will explain more about the school and how you might apply.
Comet is an easy-to-use platform for tracking and optimizing machine learning experiments. It integrates with popular frameworks like TensorFlow and PyTorch, allowing users to log metrics, hyperparameters, and model results. Comet helps visualize experiment progress in real-time, tune hyperparameters, and ensure reproducibility. A live demo will show how Comet improves collaboration, model performance, and tracks the environmental impact of machine learning projects.
In this talk, we will introduce Open OnDemand, a web-based interface designed to provide easy access to High-Performance Computing (HPC) resources. Terminal-based interfaces can be daunting for new users with little to no experience, creating a steep learning curve.
Open OnDemand aims to make HPC more accessible by offering an intuitive graphical interface that simplifies the process of submitting, monitoring, and managing jobs.
We will explore the key features of Open OnDemand, including web-based access, job management, file management and support for interactive applications like Jupyter Notebooks, RStudio, and VS Code. Additionally, we will demonstrate the SciNet Open OnDemand portal and discuss its deployment and use cases.
Tidyverse is an cohesive set of packages for doing data science in R. In an earlier talk, we began reviewing the data munging portions of tidyvese (dplyr, forcats, tibble, readr, stringr, tidyr, and purr) by using it to reconstruct the data hierarchy in a 500 pages reference PDF given only the words on each page and their bounding boxes. This talk will complete this.
In this presentation, I introduce a novel causal analysis methodology called
Probabilistic Variational Causal Effect (PACE) designed to evaluate the impact of both
rare and common events in observational studies. PACE quantifies the direct causal
effects by integrating total variation, which captures the purely causal component,
with interventions on varying treatment levels. This integration also incorporates the
likelihood of transitions between different treatment states. A key feature of PACE is
the parameter d, which allows the metric to emphasize less frequent treatment
scenarios when d is low, and more common treatments when d is high, providing a causal
effect function dependent on d.
In the coming months, national systems will be undergoing significant upgrades. In particular, older GPUs (P100, V100) will be replaced with newest H100 GPUs from NVIDIA. The total GPU computing power of the upgraded systems will grow by a factor of 3.5, but the number of GPUs will go down significantly (from 3200 to 2100). This will present a significant challenge for our users, as "the business as usual" (using a whole GPU for each process or MPI rank) will no longer be feasible in most cases. Fortunately, NVIDIA provides two powerful technologies which can be used to mitigate this situation: MPS (Multi-Process Service) and MIG (Multi-Instance GPU). This presentation will walk you through both technologies, and discuss the ways they can be used on our clusters. We will discuss how to figure out which of the approaches will work the best for your code. At the end a live demonstration will be given.
We introduce a secure computing enclave at SciNet High-Performance Computing Consortium. Codenamed S4H, this environment is already available to groups at the University of Toronto as a pilot project. S4H aims to meet researchers’ needs for hosting and working with sensitive data, which SciNet’s main cluster, Niagara, does not accommodate.
In the first part (Yohai), we’ll delve into the technical details. We’ll explain how S4H is different from Niagara in that the data are encrypted at rest and access is hardened, and what that means in practice. We will talk about the difficulties of providing isolation for different research groups on a shared system, and explore the different components that make it possible, such as key management and containerization mechanisms.
The second part (Shawn) will focus on adopting the Cybersecurity Maturity Model Certification (CMMC) framework. We’ll describe our journey deciphering the control set’s complexities, developing metadata for organizing remediation efforts, and crafting Plans of Action and Milestones for compliance gaps. Future steps include internal and potentially external assessments to verify compliance, with initiatives like Privacy Impact Assessment and penetration testing, with the eventual goal of being certified for Level 4 data.
This session explores strategies and tools that enhance collaboration, improve workflow efficiency, and streamline code management. Building on foundational Git concepts presented in parts 1 & 2, we'll explore branching strategies, focusing on how they shape team collaboration and project stability. We'll cover essential commands for rebasing, merging, and resolving conflicts, providing insights into when and why each method works best. Additionally, we'll look at interactive rebase for rewriting commit history, leveraging Git tags for versioning, and working with stashes to manage changes across different contexts. This deeper dive will enable users to maintain cleaner commit histories, manage complex workflows, and handle more advanced project requirements.
There is no shortage of tools available for producing presentations in various formats, each with its own strengths and limitations. However, technical presentations containing substantial amounts of code present additional challenges and demand specific features. Drawing from extensive experience in delivering technical presentations, this session will share key lessons learned in creating polished, clear, and maintainable slide decks. Markdown, a simple but versatile text-based format, will be demonstrated to be an effective input format to achieve these goals. This presentation will also provide a curated overview of markdown-based tools that can turn markdown input files into PDF or HTML presentations.
MPI-IO is a set of extensions to the MPI library that enable parallel high-performance I/O operations. It provides a parallel file access interface that allows multiple processes to write and read to the same file simultaneously. MPI-IO allows for efficient data transfer between processes and enables high-performance I/O operations on large datasets. It also provides additional features such as collective I/O, non-contiguous access, and file locking. This seminar will talk about some basic features of MPI I/O.
This presentation provides you with essential knowledge to effectively prepare data for analysis. Starting with an overview of the Data Analytics pipeline and processes, explore various statistical and visualization techniques used in Exploratory and Descriptive Analytics to understand historical data. You will then delve into the art of Data Preparation, gaining expertise in data cleaning, handling missing values, detecting, and handling outliers, as well as transforming and engineering features. By the end of the presentation, you will have a general understanding of the necessary tools to ensure data quality and integrity, enabling you to make informed decisions and derive valuable insights from their data.
Several types of data are collected about performance while a job is running; ultimately, much of this winds up in portals and can be examined. This data is also available to the job, while it runs, and can provide additional insight. Here, we'll demonstrate scripts that can execute within the job context to, for instance, provide summaries of performance at the job's end, or within sections of the job.
The C++ 2023 standard has std::mdspan which provides a light-weight non-owning multidimensional view of a contiguous single-dimensioned array. This enables the reinterpretation of an underlying contiguous array as a multi-dimensional array with support for different memory layouts (e.g., C and Fortran) and accessing elements (e.g., directly, using atomics, etc.). As found in other programming languages, subset views / "slices" are also possible. One does not need to use the latest compiler tools or C++ standard: one can use mdspan using a reference implementation that has been backported to C++17 and C++14 as well with NVIDIA's CUDA and NVHPC tools. This talk will discuss how to use mdspan in C++ programs.
Deep learning models are often viewed as uninterpretable "black boxes". As researchers, we often extend this thinking to the memory and compute utilization of such models. Using PyTorch Profiler, we can identify model bugs and bottlenecks to understand how to improve model performance from an efficiency perspective. This will improve training scaling and allow completion of large hyperparameter optimizations more efficiently. Here we will dicuss the usage of PyTorch Profiler, including some case studies of real training examples, and discuss possible optimizations based on profiler results.
In some binary classification problems, the underlying distribution of positive and negative samples are highly unbalanced. For example, fraudulent credit card transactions are rare compared to the volume of legitimate transactions. Training a classification model in such a case needs to take into account the nature of skewed distribution. In this seminar, we will develop a fraud detector which can be used to screen credit card transactions. We will describe the methods used to handle unbalanced data training.
Researchers often leave resources on the table when specifying their job requirements on the national systems. This talk builds on previous sessions and uses the Digital Research Alliance of Canada's User Facing Portals to explore what different types of jobs look like when they waste resources. Demonstrations will include interactive jobs, parallel jobs, GPU workflows, and more. With more accurate job specifications, researchers can expect less wait times, and more throughput on any general purpose system.
Developed collaboratively by major browser vendors, including Mozilla, Google, Microsoft, and Apple, WebAssembly (Wasm) addresses the limitations of traditional web programming languages like JavaScript. But what makes it so compelling for scientists? First, Wasm allows code written in languages like C/C++, Fortran or Rust to be compiled into its instruction format and run directly in the browser, making it accessible to anyone without installation hassles and eliminating the need for external servers. Second, with Wasm, developers can recycle existing code with near-native performance but without the hassle of rewriting it in JavaScript. Join us as we explore how Wasm is reshaping scientific workflows and empowering researchers worldwide.
Previous seminars in this series have described using Python tools to explore job properties and usage characteristics on the Digital Research Alliance of Canada general purpose compute clusters. The end goal of exploring job properties and usage characteristics is to get the most out of the resources available to research accounts and to minimize wait times in the job queue. This seminar reviews important properties of the scheduling configuration that may impact research throughput, then demonstrates how portals can help explore the relevant properties of research account compute usage on the clusters.
Advancements in sequencing technologies have revolutionized biological sciences.
Next Generation Sequencing (NGS) approaches are now a routine part of biological
research generating staggering amounts of data. This rapid growth poses
significant challenges in data acquisition, storage, and distribution. In addition,
large-scale data analysis among national and international collaborations requires
portable, scalable, and reproducible computational analysis. This webinar will also
cover some best practices including workflow management for pipeline
development, handling software installation, databases to run on different compute
platforms and enable workflow portability and sharing.
In this colloquium, we will present the curriculum of the 2024 Compute Ontario Summer School, to be held from the 3rd to the 21st of June. Jointly organized by the Centre for Advanced Computing, SciNet, SHARCNET, and in collaboration with the Research Data Management Network of Experts, this school is packed with around 40 free courses available at introductory to advanced levels and delivered by experts in the field and covers a wide range of topics.
Modules are source code components with a well-defined interface such that they can be reused in other code without requiring knowledge of the implementation or exposing its internals. In this talk, we will discuss how modules can be supported in C++. We will see that for most of its existance, C++ has had to use the "header file plus object files" paradigm inherited from C to support modules, until proper C++-based modules were introduced in the C++20 standard. Unfortunately, few C++ compilers fully the C++20 modules, and their implementations and usage varies a lot. We will give an overview not only of the issues but also of what is possible with some current compilers.
Tidyverse is an cohesive set of packages for doing data science in R. In an earlier talk, we began reviewing the data munging portions of tidyvese (dplyr, forcats, tibble, readr, stringr, tidyr, and purr) by using it to reconstruct the data hierarchy in a 500 pages reference PDF given only the words on each page and their bounding boxes. This talk will complete this.
Buying a new supercomputer that both maximises total performance, given our budget, and whose architecture suits our users' workloads is a very difficult balancing act. There are a wide range of decisions to be made, such as: CPU architecture; node count; memory size/bandwidth; GPU count; interconnect type; storage size; filesystem type/bandwidth; cooling type and power budget to name but a few. In order to balance all of these constraints we need to come up with a scoring system to compare potential candidate supercomputers. In this talk we describe the Scalable System Improvement (SSI) metric and apply it to the system refresh of Niagara & Mist.
Pandas, renowned as the go-to library for data manipulation and analysis in Python and widely adopted in machine learning. However, Pandas is slow. With the introduction of NVIDIA cuDF.pandas, the accelerated power of GPUs is integrated into Pandas, enabling faster processing without the need for any code changes. A live demo will showcase this enhancement on clusters.
Graph analysis plays a critical role in many applications across various domains, ranging from social network analysis to bioinformatics, to fraud detection, to cybersecurity, to recommendation systems, etc. NetworkX is the go-to library for graph analysis in Python. However, when dataset and graph sizes grow, the performance of using NetworkX becomes a significant concern. This webinar introduces NVIDIA cuGraph for accelerating graph analysis on GPUs. Moreover, a recent integration of NetworkX with cuGraph, named nx-cugraph, allows accelerating workflows in NetworkX on GPUs with zero code changes. A live demo will be done on the clusters.
Make is a classic Unix development tool, which may seem archaic and narrow-purpose. But if you think of it as a declarative, parallelized workflow automation tool, it sounds more relevant. We'll consider stereotypical use of make, then its general properties, and show some interesting examples of make applied to unusual uses.
One of the important steps of developing or maintaining a code is debugging: checking the code for errors. Simple toy codes can be debugged using print statements, but realistic codes need specialized debugging tools. We have a powerful debugger "DDT" installed on Graham and Niagara clusters. This presentation will walk you through the steps required to start debugging your codes using DDT, and will present the main features of the software. It will cover a wide range of situations: from debugging serial codes (Python, C/C++, Fortran) to debugging parallel CPU codes (MPI, OpenMP) to debugging GPU codes (CUDA, ROCm/HIP) to debugging hybrid codes (combining MPI, CUDA etc.). No familiarity with DDT or debugging in general is required.
Multi-factor authentication, MFA, two-factor authentication, or 2FA, along with similar terms, is an electronic authentication method in which a user is granted access to a device, website or application only after successfully presenting two or more pieces of evidence (or factors) to an authentication mechanism:MFA protects user data—which may include personal identification or financial assets from being accessed by an unauthorized third party that may have been able to discover, for example, a single password. We will explain the most common uses of MFA today, and how MFA in being implement in our environment.
In MySQL, constraints and joins are fundamental concepts used to ensure data integrity in a database and query data from multiple tables. Constraints are rules enforced on the data columns of a table. Constraints provide the accuracy and reliability of the data within a database. Joins in MySQL combine rows from two or more tables based on a related column. Previous parts in the series: Part 1, Part 2.
This presentation will take a high-level look at what we mean by reproducible research and how research data management (RDM) plays a key role in supporting reproducibility. We will draw from the literature to define and discuss reproducibility across disciplines, and various approaches to ensuring research data, metadata, and code are managed in ways that lend themselves to this key aspect of scientific advancement and integrity.
OpenMP is a popular, portable and widely supported shared-memory parallel programming model in HPC. The OpenMP API supports multi-platform parallel programming in C/C++ and Fortran. As computer hardware has grown to include GPU and other specialized accelerators, OpenMP has grown as well to add device support for parallel programming on GPU and accelerators.
This seminar will give an introduction to GPU programming with OpenMP, the OpenMP device and execution model to offload tasks (map loops and data) on GPUs.
In addition to commercial software for computational mechanics, there are several open-source C++ libraries available for developing custom solver, e.g., deal.ii for FEM libraries and OpenFOAM for CFD (FVM-based) libraries. Although one doesn’t require advanced level knowledge in C++, some fundamentals of object-oriented programming (OOP) and compilation of C++ code are required in order to use these libraries. This seminar will introduce some of the concepts of OOP, required to start using deal.ii libraries for writing FEM solvers. A Laplacian solver for transient heat equation will be developed using deal.ii.
In addition to commercial software for computational mechanics, there are several open-source C++ libraries available for developing custom solver, e.g., deal.ii for FEM libraries and OpenFOAM for CFD (FVM-based) libraries. Although one doesn’t require advanced level knowledge in C++, some fundamentals of object-oriented programming (OOP) and compilation of C++ code are required in order to use these libraries. This seminar will introduce some of the concepts of OOP, required to start using deal.ii libraries for writing FEM solvers. A Laplacian solver for transient heat equation will be developed using deal.ii.
Pi-Hole, a lightweight network-wide ad blocker, stands as a powerful solution for enhancing online privacy, security, and the overall user experience. We will explore the dynamic world of Pi-Hole using a Docker container implementation, which can be easily deployed on older hardware using Docker.
We'll guide you through the process of setting up Docker networking to allow Pi-Hole to have its own IP address on your network. Additionally, we'll demonstrate how to use Pi-Hole as a local DNS server. Furthermore, we'll examine other resources that complement Pi-Hole, enhancing its capabilities in blocking ads and malware domains.
PyTorch is an enormously popular framework for developing deep learning models in Python. However, scikit-learn is one of the most popular libraries for general machine learning. Skorch is a wrapper for PyTorch that allows one to use models written with PyTorch with the scikit-learn library. In this talk, we will explore how skorch allows for PyTorch models to be easily incorporated into scikit-learn data/training pipelines, cross-validation and hyperparameter search schemes and eliminates the need for boilerplate training code.
Web scraping is a method used to extract data from websites. It involves programmatically downloading web pages and parsing their HTML to extract the necessary information. It can be used to harvest data for the purpose of statistical analysis, training machine learning models, and creating alerts. In this talk, we'll discuss how to use basic programming skills in Python to scrape the web. We'll examine the technical and ethical aspects of the method, as well as practical applications.
It’s well known that GPUs can significantly accelerate neural network training. However, not everyone knows that a single GPU is sufficient to train most neural networks except for a few large ones (like LLMs). In fact, a GPU is under-utilized in most cases. In this talk, we are addressing the under-utilization issue and proposing a way to make full use of the GPU capacity. The goal is to increase the throughput with a single GPU. We will use a small NN training as an example to illustrate how to achieve the goal by splitting a physical GPU into multiple logical GPUs and then running a particular training process per logical GPU.
Compared to languages like Fortran and Python, the support for large multidimensional arrays in C++ is quite poor. There are many libraries trying to fill this deficiency, and there is hope at the horizon in the form of the planned C++23 and C++26 standards. But we would rather not wait for these, nor require C++ programmers to learn large frameworks, or worry about performance, when all they need is a multidimensional array. We will look at "rarray", a header-only library requiring only a C++11 compliant compiler. This library provides reference-counted multidimensional arrays that are easy to use, work now, are efficient, and can interface with many scientific libraries.
Often, researchers are given the seemingly impossible task of taking a piece of code that they did not develop and generalizing it to run in an HPC environment. This can provide a significant roadblock for researchers whose fields do not often teach skills such as parallel programming, or even terminal use. Through the lens of Python and Neuroscience this talk will provide a step by step guide on how to build the skills necessary to both understand and create workflows. While Neuroscience will be used as an example, many of the concepts will generalize to other fields due to the widespread nature of Python. Particular attention will be paid to what training materials exist, what technologies can be leveraged, and who should be contacted when roadblocks arise. Beginner level skills with Python and Git will be assumed.
Numerical simulations have become one of the key tools used by theorists in all the fields of astrophysics and cosmology. The development of modern tools that target the largest existing computing systems and exploit state-of-the-art numerical methods and algorithms is thus crucial. In this talk, we introduce the fully open-source highly-parallel, versatile, and modular coupled hydrodynamics, gravity, cosmology, and galaxy-formation code Swift. The software package exploits hybrid task-based parallelism, asynchronous communications, and domain-decomposition algorithms based on balancing the workload, rather than the data, to efficiently exploit modern high-performance computing cluster architectures. Gravity is solved for using a fast-multipole-method, optionally coupled to a particle mesh solver in Fourier space to handle periodic volumes. For gas evolution, multiple modern flavours of Smoothed Particle Hydrodynamics are implemented. Swift also evolves neutrinos using a state-of-the-art particle-based method. Two complementary networks of sub-grid models for galaxy formation as well as extensions to simulate planetary physics are also released as part of the code. An extensive set of output options, including snapshots, light-cones, power spectra, and a coupling to structure finders are also included. We describe the overall code architecture, summarize the consistency and accuracy tests that were performed, and demonstrate the excellent weak-scaling performance of the code using a representative cosmological hydrodynamical problem with ≈300 billion particles. The code is released to the community alongside extensive documentation for both users and developers, a large selection of example test problems, and a suite of tools to aid in the analysis of large simulations run with Swift.
p2rng (https://github.com/arminms/p2rng) is a modern header-only C++ library for parallel algorithmic (pseudo) random number generation supporting OpenMP, CUDA, ROCm and oneAPI. Playing fair, mostly required for debugging and unit testing, is one of the unique features of p2rng. That means using the same seed and distribution you always get the same sequence of random numbers on all supported platforms. p2rng provides parallel versions of STL’s std::generate() and std::generate_n() algorithms with the same interface. In this seminar we first start with a quick review of preliminary concepts about algorithmic random number generators in general and parallelization techniques in particular. Then we continue with the standard way of generating random numbers with STL algorithms and how we can turn them into parallel version using p2rng.
In the world where data has become extremely important, scientists require tools to process large volumes of data efficiently. R became increasingly popular in recent years for data processing, statistical analysis, and data science. In this session we will discuss tools that measure the performance of an R code, so that we can understand the nature of performance issues. We will also describe techniques that will improve the computational speed of R code. Basic knowledge of programming in R will be assumed.
Job wait times on the Alliance clusters are impacted by several factors. The target share of an account in relation to its recent usage, availability of resources being requested and overall load on the system from all users are major contributors to wait time variances. This presentation builds on previous seminars regarding wait time assessments, this time with a focus on analytics that describe the dynamic states of the clusters as a whole.