Inductor is PyTorch’s compiler backend designed to optimize and generate high-performance code for arbitrary models. It works over a few phases:

A wide ranging talk that includes a lot of the new work on ExecuTorch for on-device support, including the torch.export stack. The talk really goes through the whole flow from graph capture, lowering and optimizations so its a great way to get a sense for what is happening within PyTorch!

https://www.nvidia.com/content/gtc-2010/pdfs/2238_gtc2010.pdf

https://www.hpcwire.com/2023/10/05/how-amd-may-get-across-the-cuda-moat/

The PyTorch composability meeting is one of the most interesting discussions of how the various parts of PyTorch fit together, particularly in the context of the PyTorch compiler and associated technologies. The meetings themselves are public, and so are the meeting notes, which are a great reference point.

A very practical exploration of how to get a transformer model going faster for inference:

The kernel dispatch time eats a lot of performance on GPU – CUDA graphs let you chain a bunch of kernels together, and they’re now more accessible from PyTorch:

A blog post from PyTorch veteran and core maintainer Ed Yang based on a talk where he breaks down the fundamentals of PyTorch. From 2019 but still very useful in explaining the tensor mechanics, particularly including striding, which is one of those simple but very applicable concepts!

Thispaper from Tencent last year on the architecture of their recsys, and how it enables a high degree of freshness, via low-latency model updates to deliver fresh and relevant recommendations.

Microsoft published a paper about GPT-4, with the (literal) headline claim that it showed sparks of artificial general intelligence. The paper includes an approach to evaluate the model’s abilities, and many examples.