ML Systems Engineer focused on LLM inference runtime, GPU kernels, and serving infrastructure.

I build systems around large language model inference:

• custom CUDA kernels
• C++/CUDA inference runtimes
• serving stacks and scheduling systems
• compiler/codegen experiments for Transformer workloads
• distributed inference (TP / PP / KV sharding)
• quantization and compression analysis
• retrieval-augmented inference pipelines

My work sits at the boundary between ML systems, GPU performance engineering, and runtime infrastructure.

If you're visiting for the first time, these are the best entry points:

• mini-llm-inference-engine
  Transformer inference engine in C++/CUDA, without a PyTorch execution path.

• sm75-tensorcore-microkernel
  PTX-level Tensor Core GEMM programming for NVIDIA Turing (SM75).

• flash-attention-sm75
  FlashAttention v1 forward pass for SM75.

• flash-attention-sm75-register-fused
  Register-fused FlashAttention study with persistent accumulators and roofline-guided optimization.

• fused-backward-kernels-sm75
  Fused GEMM + Bias + GeLU backward CUDA kernels.

• llm-fusion-compiler
  Compiler MVP: IR → CUDA codegen → nvcc → GPU execution.

• custom-llm-serving-engine
  Production-oriented inference server with batching, SSE streaming, KV cache management, and Prometheus metrics.

• speculative-decoding-runtime
  Speculative decoding with rejection sampling, adaptive gamma, and correctness validation.

• lora-inference-runtime
  Multi-LoRA serving runtime with low-latency adapter hot-swapping.

• distributed-inference-engine
  Simulated multi-GPU runtime implementing Tensor Parallelism, Pipeline Parallelism, and KV cache sharding.

• quantization-runtime
  From-scratch AWQ and GPTQ implementation/analysis for 4-bit compression stability.

• rag-inference-stack
  Local-first RAG pipeline with ingestion, embeddings, vector retrieval, and answer generation.

• vlm-inference-runtime
  Deployment-oriented VLM serving wrapper for Qwen2-VL.

Core logic implemented directly rather than wrapped around an existing system:

• mini-llm-inference-engine
• sm75-tensorcore-microkernel
• flash-attention-sm75
• flash-attention-sm75-register-fused
• fused-backward-kernels-sm75
• llm-fusion-compiler
• custom-llm-serving-engine
• distributed-inference-engine
• quantization-runtime
• speculative-decoding-runtime

Systems tradeoff analysis, modeling, and operational benchmarking:

• llm-serving-sim
• comm-cost-modeling
• tensor-memory-allocator
• hardware-aware-llm-runtime
• capacity-planning-sim
• continuous-batching-fragmentation-sim
• paged-attention-sim

These are intentionally framed as integration/deployment systems, not custom low-level runtimes:

• rag-inference-stack
• vlm-inference-runtime

A recent 5-project series covering major components of the inference stack:

• comm-cost-modeling — communication modeling across PCIe / NVLink / InfiniBand
• llm-serving-sim — end-to-end serving simulation
• tensor-memory-allocator — allocator policy tradeoffs
• continuous-batching-scheduler — scheduling policy comparisons
• kv-cache-compaction-lab — compaction policy analysis
• cuda-graph-decode-profiler — graph vs eager decode profiling

• Contributed to LLM Inference Handbook / llm-inference-at-scale

Worked on AI infrastructure for a confidential leading Chinese LLM company under NDA, with focus on backend/runtime-adjacent systems.