Things that can go wrong with priority levels
December 13, 2024
I read this post years back as we did a big migration to Bazel for Android and iOS at Lyft, but was reminded of it from a recent HN comment by jmmv, the author:
Way to use torch.compile
December 11, 2024
Engineering Culture at Meta
December 11, 2024
A question I’ve been asked recently is how Meta compares to other places I’ve worked, or what makes it different.
A very lazy mental model for parallelisms
December 6, 2024
There is a great deal of technical complexity in distributed data parallel, model parallel, tensor parallel, fully-sharded-data-parallel, ZeRO 1/2/3, context parallel etc. To fit ideas into a picture I tend to segment in-practice parallelism into three buckets.
TIL: When does PyTorch upgrade Python versions
December 5, 2024
The policy is in the RELEASE documentation:
A Meticulous Guide to Advances in Deep Learning Efficiency over the Years
November 29, 2024
Training Language Models to Self-Correct via Reinforcement Learning
November 28, 2024
Training Language Models to Self-Correct via Reinforcement Learning – Aviral Kumar, Vincent Zhuang et al at Deepmind
PyTorch while_loop
November 27, 2024
I’ve been following the development of the higher order ops in PyTorch nightlies for a little bit, and got a chance to try out while_loop. The best examples right now are in the tests, but as another, here’s a mandlebrot example:
Eureka moments
November 26, 2024
Unsupported: dynamic shape operator: aten.nonzero.default with boolean masks in torch.compile
November 25, 2024
The error message you get actually tells you the fix, but I found it non-intuitive to what I was doing enough I was hesitant to actually just try the config: