Document deploying JupyterHub to Jetstream2 using Kubernetes and ClusterAPI

[jmunroe]

The most recent work on deploying JupyterHub to Jetstream2 using Kubernetes is covered by @zonca 's excellent tutorials and blog posts summarized in https://www.zonca.dev/posts/2023-09-21-gateways-2023-dask-jupyterhub

In recent discussions with the Jetstream2 team, the next iteration of a 'best practice' for deploying JuptyerHub will likely be based around ClusterAPI (https://cluster-api.sigs.k8s.io)

This issue marks a common location for iterating and discussing the steps required to support ClusterAPI on Jetstream2.

When that iterative process had progress, these recommendations for using Kubernetes should be added to the list of cloud providers documented in https://z2jh.jupyter.org/en/stable/kubernetes/setup-kubernetes.html

[cboettig]

@julianpistorius any updates on jupyterhub-based access to jetstream2?

[julianpistorius]

Hi @cboettig! Apologies for the delay. I have been traveling and got behind on my GitHub alerts. Yes, I have been evaluating options. I will meet with Jeremy & the team next week about my findings, and will update this issue then.

[cboettig]

@julianpistorius Hooray! sounds great, keep us posted!

[cboettig]

@julianpistorius any updates?

[cboettig]

also maybe an aside, but is their any connection between Jetstream2 GPU and the https://nairrpilot.org/ efforts? Or is the latter exclusively focused on the commercial cloud?

[julianpistorius]

Hi @cboettig!

I've been trying to debug an issue with Cluster-API which is preventing us from getting to the next step. I think I'm close, but it's hard to say with so many moving parts. I have a few things to try and will keep you updated.

As for NAIRR, yes Jetstream2 is involved: https://news.iu.edu/live/news/35829-ius-cloud-computing-resource-receives-49m-to

You will see Indiana Jetstream2 GPU in the resources: https://submit-nairr.xras.org/resources

And you can request it when you submit a proposal: https://submit-nairr.xras.org/opportunities/533375/requests/new

Are you interested in submitting a proposal?

[cboettig]

thanks @julianpistorius , great! Keep us posted.

Re NAIRR, excellent! Yes, actually just put in an education call proposal to try this in my classroom, though it looks like at the moment it only lists SDSC's Prototype National Research Platform (PNRP) Classroom (which I listed) or Vocareum AI as possible providers on that side. It is great to see Jetstream2 being listed on the research side (along with many other academic cloud platforms)

[julianpistorius]

@cboettig Definitely off-topic now, but I've been told that we're working on making Jetstream2 available for NAIRR education proposals.

[julianpistorius]

Update: We figured out the gotchas, and have deployed a few Kubernetes clusters on Jetstream2 using Cluster API. We also deployed a basic JupyterHub. I feel comfortable enough with this process to write documentation now.

(O/T @cboettig Jetstream2 is now available as a classroom resource for NAIRR)

[cboettig]

@julianpistorius amazing! I'm really excited to try this out both for our research program and teaching with NAIRR