[wip] tutorial content for hpsf 2026

2026/HPSF/README.md

@@ -0,0 +1,71 @@
+# Flux Jupyter Tutorial
+
+This set of tutorials provides:
+
+ - [Building Tutorial Images](#build-images)
+ - [Introduction to Flux](#introduction-to-flux)
+ - [The Flux Operator with Agents](#the-flux-operator-with-agents)
+
+Pre-requisites:
+
+ - Docker client installed locally
+ - Excitement to learn about Flux!
+
+## Build Images
+
+Build the tutorial images.
+
+```bash
+docker build -f ./docker/Dockerfile -t hpsf-flux .
+docker build -f ./docker/Dockerfile.ml -t hpsf-flux-ml .
+docker build -f ./docker/Dockerfile.flux-operator -t ghcr.io/flux-framework/tutorials:flux-operator-pytorch .
+```
+
+## 1. Introduction to Flux 
+
+The introduction to Flux section is a notebook. You can simply run the Image.
+
+```bash
+docker run --rm -it  -v /var/run/docker.sock:/var/run/docker.sock --name jupyterhub  -p 8888:8888 hpsf-flux
+```
+
+## 2. The Flux Operator with Agents
+
+The Flux Operator and MCP example is a Flux Framework MiniCluster. Either create a cloud (or have) a Kubernetes cluster, or create one with kind:
+
+```bash
+kind create cluster --config ./flux-operator/kind-config.yaml
+kind load docker-image ghcr.io/flux-framework/tutorials:flux-operator-pytorch
+```
+Install the Flux Operator:
+
+```bash
+kubectl apply -f https://raw.githubusercontent.com/flux-framework/flux-operator/refs/heads/main/examples/dist/flux-operator.yaml
+```
+
+Wait until both ranks are running, then shell inside and connect to the lead broker.
+
+```bash
+kubectl get pods --watch
+kubectl exec -it pytorch-0-xxxx -- bash
+. /mnt/flux/flux-view.sh
+flux proxy $fluxsocket bash
+flux resource list
+```
+
+Now let's run Pytorch, without agents. There is a simple "hello world" and training example with cifar.
+
+```bash
+flux run -N 2 --exclusive python distributed_flux_hello_world.py
+flux run -N 2 --exclusive python distributed_flux.py 
+```
+
+We could also put `-n` for the number of tasks (processes) per node, but exclusive will ask for all of them.
+
+
+## 3. Kubeflow and Flux
+
+The Kubeflow and Flux example can use the same cluster, but we add Kubeflow.
+
+**TODO**
+

2026/HPSF/docker/Dockerfile

@@ -0,0 +1,104 @@
+FROM fluxrm/flux-sched:noble
+
+# Based off of https://github.com/jupyterhub/zero-to-jupyterhub-k8s/tree/main/images/singleuser-sample
+# Local usage
+# docker run -p 8888:8888 -v $(pwd):/home/jovyan/work test
+
+USER root
+
+ENV NB_USER=jovyan \
+    NB_UID=1000 \
+    HOME=/home/jovyan
+
+RUN adduser \
+    --disabled-password \
+    --gecos "Default user" \
+    --uid ${NB_UID} \
+    --home ${HOME} \
+    --force-badname \
+    ${NB_USER}
+
+RUN apt-get update \
+    # && apt-get upgrade -y \
+    && apt-get install -y --no-install-recommends \
+    gcc-10 \
+    g++-10 \
+    ca-certificates \
+    dnsutils \
+    iputils-ping \
+    python3 \
+    python3-dev \
+    python3-pip \
+    python3-venv \
+    openmpi-bin \
+    openmpi-common \
+    libopenmpi-dev \
+    liblz4-dev \
+    tini \
+    nodejs \
+    python3-greenlet \
+    # requirement for nbgitpuller
+    git \
+    && rm -rf /var/lib/apt/lists/*
+
+ENV VIRTUAL_ENV=/opt/venv
+ENV PATH="$VIRTUAL_ENV/bin:$PATH"
+
+COPY ./docker/requirements_venv.txt ./requirements_venv.txt
+RUN python3 -m venv $VIRTUAL_ENV && pip install --no-cache-dir -r requirements_venv.txt
+COPY ./docker/requirements.txt ./requirements.txt
+
+RUN pip install ruamel.yaml.clib && \
+    pip install -r requirements.txt && \
+    pip install ipykernel pycurl IPython && \
+    python -m IPython kernel install 
+
+RUN wget https://nodejs.org/dist/v20.15.0/node-v20.15.0-linux-x64.tar.xz && \
+    apt-get update && apt-get install -y xz-utils && rm -rf /var/lib/apt/lists/* && \
+    xz -d -v node-v20.15.0-linux-x64.tar.xz && \
+    tar -C /usr/local --strip-components=1 -xvf node-v20.15.0-linux-x64.tar
+
+# This customizes the launcher UI
+# https://jupyter-app-launcher.readthedocs.io/en/latest/usage.html
+RUN pip install jupyter_app_launcher && \
+    pip install --upgrade jupyter-server && \
+    pip install jupyter-launcher-shortcuts && \ 
+    pip install jupyterhub_idle_culler && \
+    mkdir -p /usr/local/share/jupyter/lab/jupyter_app_launcher && \
+    pip install jupyterhub && \
+    pip install ipywidgets
+
+# This customizes the launcher UI
+# https://jupyter-app-launcher.readthedocs.io/en/latest/usage.html
+RUN pip install jupyter_app_launcher && \
+    pip install --upgrade jupyter-server && \
+    pip install jupyter-launcher-shortcuts && \
+    mkdir -p /usr/local/share/jupyter/lab/jupyter_app_launcher
+
+COPY ./tutorial /home/jovyan/
+COPY ./docker/jupyter-launcher.yaml /usr/local/share/jupyter/lab/jupyter_app_launcher/jp_app_launcher.yaml
+ENV JUPYTER_APP_LAUNCHER_PATH=/usr/local/share/jupyter/lab/jupyter_app_launcher/
+
+# Give jovyan user permissions to tutorial materials
+RUN chmod -R 777 ~/ /home/jovyan
+
+WORKDIR $HOME
+# Flux assets in the tutorial/ directory were moved to tutorial/img/
+# this shouldn't be a problem for the next command but making a note in case
+COPY ./docker/flux-icon.png $HOME/flux-icon.png
+
+# note that previous examples are added via git volume in config.yaml
+ENV SHELL=/usr/bin/bash
+ENV FLUX_URI_RESOLVE_LOCAL=t
+
+EXPOSE 8888
+ENTRYPOINT ["tini", "--"]
+
+# This is for JupyterHub
+COPY ./docker/entrypoint.sh /entrypoint.sh
+RUN mkdir -p $HOME/.local/share && \
+    chmod 777 $HOME/.local/share
+
+USER ${NB_USER}
+
+CMD ["flux", "start", "--test-size=4", "jupyter", "lab", "--ip=0.0.0.0"]

2026/HPSF/docker/Dockerfile.flux-operator

@@ -0,0 +1,14 @@
+FROM nvcr.io/nvidia/pytorch:24.02-py3
+RUN apt-get update && apt-get install -y \
+    build-essential \
+    git \
+    wget \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /code
+COPY ./flux-operator/requirements.txt /code/requirements.txt
+RUN pip install flux-mcp
+COPY ./flux-operator/distributed_flux.py /code/distributed_flux.py
+ENV LD_LIBRARY_PATH=/opt/hpcx/ompi/lib:/opt/hpcx/mxm/lib:/opt/hpcx/ucx/lib:$LD_LIBRARY_PATH
+ENV PATH=/opt/hpcx/ompi/bin:$PATH
+ENV SHELL=/usr/bin/bash

2026/HPSF/docker/entrypoint.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+/usr/bin/flux start --test-size=4 /usr/local/bin/jupyterhub-singleuser

2026/HPSF/docker/jupyter-launcher.yaml

@@ -0,0 +1,100 @@
+- title: Flux Tutorial Notebook
+  description: This is the main Flux Framework Tutorial
+  type: jupyterlab-commands
+  icon: ./static/flux-icon.png
+  source:
+    - label: Flux Tutorial
+      id: 'filebrowser:open-path'
+      args:
+        path: ./tutorial/ch1/01_flux_tutorial.ipynb
+        icon: ./static/flux-icon.png
+  catalog: Notebook
+
+- title: Flux Python SDK Tutorial
+  description: This notebook will teach you about the Flux Python SDK
+  type: jupyterlab-commands
+  icon: ./static/flux-icon.png
+  source:
+    - label: Flux Tutorial
+      id: 'filebrowser:open-path'
+      args:
+        path: ./tutorial/ch2/02_flux_framework.ipynb
+        icon: ./static/flux-icon.png
+  catalog: Notebook
+
+- title: Flux Process, Monitoring, Utilities Tutorial
+  description: This will teach you about Flux utilities
+  type: jupyterlab-commands
+  icon: ./static/flux-icon.png
+  source:
+    - label: Flux Tutorial
+      id: 'filebrowser:open-path'
+      args:
+        path: ./tutorial/ch3/03_flux_tutorial.ipynb
+        icon: ./static/flux-icon.png
+  catalog: Notebook
+
+- title: Flux Framework and User-space Kubernetes
+  description: This will teach you about running AI/ML workloads with Flux and Kubernetes
+  type: jupyterlab-commands
+  icon: ./static/flux-icon.png
+  source:
+    - label: Flux Tutorial
+      id: 'filebrowser:open-path'
+      args:
+        path: ./tutorial/ch4/04_flux_framework_usernetes.ipynb
+        icon: ./static/flux-icon.png
+  catalog: Notebook
+
+- title: Flux Framework Portal
+  description: Flux Framework portal for projects, releases, and publication.
+  source: https://flux-framework.org/
+  type: url
+  catalog: Flux Resources
+  args:
+      sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']
+- title: Flux Documentation
+  source: https://flux-framework.readthedocs.io/en/latest/
+  type: url
+  catalog: Flux Resources
+  args:
+      sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']
+- title: Flux Cheat Sheet
+  source: https://flux-framework.org/cheat-sheet/
+  type: url
+  catalog: Flux Resources
+  args:
+      sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']
+- title: Flux Glossary of Terms
+  source: https://flux-framework.readthedocs.io/en/latest/glossary.html
+  type: url
+  catalog: Flux Resources
+  args:
+      sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']
+- title: Flux Comics
+  source: https://flux-framework.readthedocs.io/en/latest/comics/fluxonomicon.html
+  description: come and meet FluxBird - the pink bird who knows things!
+  type: url
+  catalog: Flux Resources
+  args:
+      sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']
+- title: Flux Learning Guide
+  source: https://flux-framework.readthedocs.io/en/latest/guides/learning_guide.html
+  description: learn about what Flux does, how it works, and real research applications
+  type: url
+  catalog: Flux Resources
+  args:
+      sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']
+- title: Getting Started with Flux and Go
+  source: https://converged-computing.github.io/flux-go
+  type: url
+  catalog: Flux Resources
+  args:
+      sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']
+- title: Getting Started with Flux in C
+  source: https://converged-computing.github.io/flux-c-examples/
+  description: ...looking for contributors!
+  type: url
+  catalog: Flux Resources
+  args:
+      sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']