<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0">
    <channel>
      <title>Sonny&#039;s Blog</title>
      <link>https://lubits.ch</link>
      <description>Last 10 notes on Sonny&#039;s Blog</description>
      <generator>Quartz -- quartz.jzhao.xyz</generator>
      <item>
    <title>Flash Attention from Scratch: Appendix B - Block Size Configuration</title>
    <link>https://lubits.ch/flash/Appendix-B---Block-Size-Configuration</link>
    <guid>https://lubits.ch/flash/Appendix-B---Block-Size-Configuration</guid>
    <description><![CDATA[ This appendix dives into how block size configurations affect instruction patterns and performance in Flash Attention. ]]></description>
    <pubDate>Sat, 08 Nov 2025 00:01:00 GMT</pubDate>
  </item><item>
    <title>Flash Attention from Scratch Part 8: Instruction Reduction</title>
    <link>https://lubits.ch/flash/Part-8</link>
    <guid>https://lubits.ch/flash/Part-8</guid>
    <description><![CDATA[ Intro In Part 6, we improved our kernel to slightly outperform the reference kernel on the RTX 3090, but found that on the A100, it only reached 80.3% of the reference. ]]></description>
    <pubDate>Sat, 08 Nov 2025 00:00:00 GMT</pubDate>
  </item><item>
    <title>Appendix A - Ampere Microarchitecture</title>
    <link>https://lubits.ch/flash/Appendix-A---Ampere-Microarchitecture</link>
    <guid>https://lubits.ch/flash/Appendix-A---Ampere-Microarchitecture</guid>
    <description><![CDATA[ Ampere Microarchitecture This part dives into the SM architecture to reveal how execution units compete for resources. ]]></description>
    <pubDate>Sat, 01 Nov 2025 00:02:00 GMT</pubDate>
  </item><item>
    <title>Flash Attention from Scratch Part 7: A100 Profiling</title>
    <link>https://lubits.ch/flash/Part-7</link>
    <guid>https://lubits.ch/flash/Part-7</guid>
    <description><![CDATA[ In Part 6, we exceeded reference performance on the RTX 3090, hitting 101.5% through FP instruction fusion and auto-tuning. ]]></description>
    <pubDate>Sat, 01 Nov 2025 00:01:00 GMT</pubDate>
  </item><item>
    <title>Flash Attention from Scratch Part 6: FP Instruction Fusion and Auto-Tuning</title>
    <link>https://lubits.ch/flash/Part-6</link>
    <guid>https://lubits.ch/flash/Part-6</guid>
    <description><![CDATA[ In the previous part, we implemented three major optimizations from the CUTLASS GEMM library: eager block loading, sub-tiling with fragment interleaving, and double buffering. ]]></description>
    <pubDate>Sat, 01 Nov 2025 00:00:00 GMT</pubDate>
  </item><item>
    <title>Flash Attention from Scratch Part 5: Cutlass GEMM Optimizations</title>
    <link>https://lubits.ch/flash/Part-5</link>
    <guid>https://lubits.ch/flash/Part-5</guid>
    <description><![CDATA[ Intro In the previous part, we implemented swizzling and achieved a dramatic 2x performance improvement by eliminating bank conflicts. ]]></description>
    <pubDate>Sun, 14 Sep 2025 02:00:00 GMT</pubDate>
  </item><item>
    <title>Flash Attention from Scratch Part 4: Bank Conflicts &amp; Swizzling</title>
    <link>https://lubits.ch/flash/Part-4</link>
    <guid>https://lubits.ch/flash/Part-4</guid>
    <description><![CDATA[ Intro In the last part, we used the instructions covered in Part 2 to construct our first kernel and reached nearly half the performance of the official implementation on the RTX 3090. ]]></description>
    <pubDate>Sun, 14 Sep 2025 01:00:00 GMT</pubDate>
  </item><item>
    <title>Flash Attention from Scratch Part 3: Kernel 1</title>
    <link>https://lubits.ch/flash/Part-3</link>
    <guid>https://lubits.ch/flash/Part-3</guid>
    <description><![CDATA[ Intro In Part 2, we explored the fundamental CUDA building blocks - tensor core operations (mma) and efficient memory transfers (cp.async &amp; ldmatrix). ]]></description>
    <pubDate>Sat, 09 Aug 2025 03:00:00 GMT</pubDate>
  </item><item>
    <title>Flash Attention from Scratch Part 2: Building Blocks</title>
    <link>https://lubits.ch/flash/Part-2</link>
    <guid>https://lubits.ch/flash/Part-2</guid>
    <description><![CDATA[ Intro In this part, we&#039;ll explore the CUDA operations that form the foundation of our Flash Attention kernel. ]]></description>
    <pubDate>Sat, 09 Aug 2025 01:00:00 GMT</pubDate>
  </item><item>
    <title>Flash Attention from Scratch Part 1: Intro</title>
    <link>https://lubits.ch/flash/Part-1</link>
    <guid>https://lubits.ch/flash/Part-1</guid>
    <description><![CDATA[ Intro In this 10-part series, we&#039;re going to implement Flash Attention 2 from scratch on Ampere GPUs. ]]></description>
    <pubDate>Sat, 09 Aug 2025 00:00:00 GMT</pubDate>
  </item>
    </channel>
  </rss>