https://arxiv.org/abs/2502.03373

Who needs a tokenizer anyway!

In general, branching in GPU code is considered bad. When you write a kernel, it’s very easy to write the same kind of logic as you would on a CPU. However, GPU kernels execute on blocks of threads scheduled on streaming multiprocessors (SMs), and they are optimized for vectorized (or parallel) computation. This optimization relies on the idea that large groups of threads can execute the same instructions on different data in a lockstep fashion. Practically, these are scheduled as “warps” of 32 threads at a time (on Nvidia, the equivalent in AMD is 64 threads).

[2501.18512] Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch

One of the challenges discussed in the Deepseek v3 paper is the availability of grouped GEMM kernels, which are used to hide the performance impact of many small kernel launches on GPUs. Deepseek uses many small experts (256!) rather than a few larger ones, which exacerbates this problem.

https://eli.thegreenplace.net/2025/decorator-jits-python-as-a-dsl/

Particularly at the giant tech firms where there are many, many smart people, folks look at success and try to copy it. They set themselves up for promotions against the definition of what is expected at the next level. But those expectations describe an average, not a person. Real people are spiky.

https://drscotthawley.github.io/blog/posts/FlowModels.html

GRPO (Group Relative Policy Optimization) is an RL technique originally proposed in the DeepSeekMathpaper. Instead of using a full-blown value network like PPO does, GRPO samples a group of completions for a given prompt and then computes a relative (normalized) reward for each output. The rewards are “verifiable” because they come from checking the final answer against ground truth and confirming. E.g. does the response follow the expected format (i.e. a <think>…</think> block for reasoning and an <answer>…</answer> block for the solution) and is the answer accurate against a predetermined fact. Not every problem fits this model, but there are a bunch that do, including math reasoning with the GSM8Kdataset of grade-school math word problems. These look like this: