Fantastic tip from Alban, particularly useful when you have a giant model and limited VRAM.

Interesting to read this breakdown of what was happening with Reader: I recall the discussions internally at Google and knew some folks floating around it (including at least one person who volunteered to take on maintenance) but this was largely new to me. Uncomfortably enough was on stage giving a talk for Google the day the shut down happened. I asked for questions, got a lot of hands, then qualified I was looking for questions about the talk and not Reader, and most of the hands went down.

https://arxiv.org/html/2311.08105v3

Inductor is PyTorch’s compiler backend designed to optimize and generate high-performance code for arbitrary models. It works over a few phases:

A wide ranging talk that includes a lot of the new work on ExecuTorch for on-device support, including the torch.export stack. The talk really goes through the whole flow from graph capture, lowering and optimizations so its a great way to get a sense for what is happening within PyTorch!

https://www.nvidia.com/content/gtc-2010/pdfs/2238_gtc2010.pdf

https://www.hpcwire.com/2023/10/05/how-amd-may-get-across-the-cuda-moat/

The PyTorch composability meeting is one of the most interesting discussions of how the various parts of PyTorch fit together, particularly in the context of the PyTorch compiler and associated technologies. The meetings themselves are public, and so are the meeting notes, which are a great reference point.

A very practical exploration of how to get a transformer model going faster for inference:

The kernel dispatch time eats a lot of performance on GPU – CUDA graphs let you chain a bunch of kernels together, and they’re now more accessible from PyTorch: