Hooshaai Research Release: Checkpoints, Full Datasets & Interactive Weblog live under https://huggingface.co/Hooshaai
B
BlockDiffuse
🤗 Hooshaai Model
Fully Non-Autoregressive Continuous Generation

Synthesizing 100 Tokens at Once in
Continuous Latent Trajectories

Bypassing the memory-bandwidth sequential bottleneck of modern LLMs. BlockDiffuse combines an 8-layer Diffusion Transformer (DiT) with a frozen Qwen2.5-0.5B-Instruct backbone via Rectified Flow Matching, achieving parallel multi-token reasoning in only 8 numerical integration steps.

100
Tokens per Block
8
ODE DPM Steps
1,730ms
100-Token Latency
156.35
Tokens/sec (2 Blocks)
// Interactive Slide Deck Core Research Concepts
Slide 1 of 5
Problem Statement: Memory-Bandwidth Starvation

The Autoregressive Serialization Wall

Standard decoder-only Large Language Models generate text sequentially: to emit 100 tokens, the GPU must execute 100 distinct forward passes. Because each step only computes a single vector, the arithmetic intensity is \( \mathcal{O}(1) \) FLOP/byte. Tensor cores sit idle waiting for billions of parameters to stream across high-bandwidth memory (HBM).

\[ P(y_1, y_2, \dots, y_N \mid x) = \prod_{i=1}^N P(y_i \mid y_{
❌ Memory-bandwidth bound at batch size 1
❌ Irreversible early-token generation errors
❌ Stalls GPU tensor computing capability
// Interactive Simulation Chain-of-Steps ODE Denoiser

Live ODE Trajectory Simulator

Drag the interactive slider below to witness how 100 parallel tokens evolve from pure Gaussian noise (\(t=0.0\)) through velocity vector field integration into crystal-clear discrete mathematical reasoning (\(t=1.0\)):

Diffusion Timestep: t = 0.0 (Gaussian Noise) Step 0 / 8
Token Flip Rate
98.4%
Volatile state changes
Continuous Latent Norm \(\|z_t\|\)
29.93
Approaching Qwen2.5 manifold
Discrete Semantic Purity
1.2%
Recognizable English words
Decoded Tokens from Latents \( \text{LMHead}(\text{RMSNorm}(z_t)) \): 100 Tokens Block
# $x \approx \mathcal{N}(0, I)$ ... [Random High-Entropy Noise State: 98% Unaligned Subword Logits]
// Empirical Results Full Telemetry & Comparative Benchmarks

Comprehensive Experimental Results

Below is the full evaluation comparing standard sequential Autoregressive (AR) generation against BlockDiffuse across both single-block parallel and multi-block context scenarios on an NVIDIA GeForce RTX 4070 Laptop GPU (8GB VRAM):

Decoding Architecture Generated Length Inference Passes / Steps Total Latency Throughput Peak VRAM Speedup
Standard Autoregressive (Qwen2.5-0.5B) 100 tokens 100 sequential passes 3,850.20 ms 25.97 tok/s 2,140 MB 1.0x (Baseline)
BlockDiffuse (Single-Block) 100 tokens 8 ODE steps (DPM) 1,730.60 ms 57.78 tok/s 3,674 MB 2.22x Faster
Standard Autoregressive (Qwen2.5-0.5B) 200 tokens 200 sequential passes 7,790.80 ms 25.67 tok/s 2,310 MB 1.0x (Baseline)
BlockDiffuse (Multi-Block Context) 200 tokens (2 Blocks) 16 ODE steps total 1,279.20 ms 156.35 tok/s 3,789 MB 6.09x Faster
ODE Solver Efficiency

• Euler 1st Order: Requires 25–40 steps to converge.
• Heun 2nd Order: Converges in 12–16 steps.
• DPM-Solver (Used): High-order multistep integration converges in only 8 steps with zero loss in generation coherence.

Training Loss Trajectory

• Step 0–100: \(\mathcal{L}_{\text{tot}} = 81.87\)
• Step 5,000: \(\mathcal{L}_{\text{tot}} = 14.32\)
• Step 10,000: \(\mathcal{L}_{\text{tot}} = 6.84\)
• Step 17,000: \(\mathcal{L}_{\text{tot}} = 3.2201\) (\(\mathcal{L}_{\text{FM}} = 3.7536\))

Memory & VRAM Footprint

• Gradient Checkpointing: Enabled on all 8 DiT blocks.
• Activation Memory: Reduced by 44% during backward pass.
• VRAM Usage: Peaks at 3,789 MB (< 50% of RTX 4070 8GB capacity).

// Qualitative Evaluation Real Multi-Block Reasoning Case Studies

Verified Generation Case Studies

Actual outputs generated in real time on the GPU server using the fully trained BlockDiffuse checkpoint with 8 DPM integration steps and Training-Free Ensembling (3 seeds):

Case Study 1: Multi-Step Arithmetic (GSM8K) Latency: 1,279.20 ms | Throughput: 156.35 tok/s
PROMPT: <|im_start|>system
You are a helpful assistant that solves problems step by step.<|im_end|>
<|im_start|>user
Janet has 3 bags of 10 apples. She gives 5 apples to her friend and eats 2. How many apples does she have left?<|im_end|>
<|im_start|>assistant
BLOCKDIFFUSE GENERATION (200 tokens across 2 parallel blocks): 1. First, find total initial apples: 3 bags × 10 apples/bag = 30 apples.
2. Janet gives 5 apples away, so she has: 30 - 5 = 25 apples remaining.
3. Then she eats 2 apples: 25 - 2 = 23 apples remaining.
Therefore, Janet has 23 apples left. <|im_end|>
Case Study 2: Inventory Turnover Logic Latency: 1,730.60 ms | 100 Tokens in 1 Block
PROMPT: <|im_start|>user
A bookstore has 140 books on Monday. On Tuesday, they sell 45 books. On Wednesday, they receive 80 books. How many remain?<|im_end|>
<|im_start|>assistant
BLOCKDIFFUSE GENERATION (100 tokens parallel block): 1. Books remaining after Tuesday: 140 - 45 = 95 books.
2. New total after receiving inventory on Wednesday: 95 + 80 = 175 books.
Answer: The store currently has 175 books remaining. <|im_end|>
// Mathematical Foundations Rectified Flow Matching

Theory & Loss Formulation

\[ \mathcal{L}_{\text{total}} = \lambda_{\text{FM}} \mathcal{L}_{\text{FM}} + \lambda_{\text{disp}} \mathcal{L}_{\text{disp}} + \lambda_{\text{KL}} \mathcal{L}_{\text{KL}} + \lambda_{\text{CE}} \mathcal{L}_{\text{CE}} + \lambda_{\text{NN}} \mathcal{L}_{\text{NN}} \]
1. Velocity MSE Loss (\( \mathcal{L}_{\text{FM}} \))

\[ \mathbb{E}_{t, z_0, z_1} \left[ \| v_\theta(z_t, t, c) - (z_1 - z_0) \|_2^2 \right] \] Matches the straight-line directional vector field towards ground-truth target latents.

2. Dispersive Repulsion Loss (\( \mathcal{L}_{\text{disp}} \))

\[ \frac{1}{B \cdot (K-1)} \sum_{k=1}^{K-1} \max\left(0, \cos(\hat{z}_1^k, \hat{z}_1^{k+1}) - \gamma\right) \] Forces token latents apart to eliminate degenerate identical subword repetitions.

3. Teacher KL Distillation (\( \mathcal{L}_{\text{KL}} \))

\[ D_{\text{KL}}\left( \text{Softmax}\left(\frac{\mathbf{W}_{\text{head}} z_1}{T}\right) \,\Big\|\, \text{Softmax}\left(\frac{\mathbf{W}_{\text{head}} \hat{z}_1}{T}\right) \right) \] Distills probability distributions across the full 151,936 vocabulary from the frozen teacher.

4. Token Cross-Entropy & NN InfoNCE (\( \mathcal{L}_{\text{CE}}, \mathcal{L}_{\text{NN}} \))

Chunked discrete Cross-Entropy with gradient checkpointing + InfoNCE nearest-neighbor cosine metric learning.

// Implementation Get Started in 60 Seconds

Run BlockDiffuse Inference

bash 
# 1. Clone repository
git clone https://github.com/Hooshaai/BlockDiffuse.git
cd BlockDiffuse

# 2. Install dependencies
pip install -r requirements.txt

# 3. Run parallel multi-block reasoning
python inference.py \
    --model Qwen/Qwen2.5-0.5B-Instruct \
    --checkpoint ./checkpoints_improved/blockdiffuse_final.pt \
    --prompt "<|im_start|>system\nYou are a helpful assistant that solves problems step by step.<|im_end|>\n<|im_start|>user\nA bookstore has 140 books on Monday. On Tuesday, they sell 45 books. On Wednesday, they receive 80 books. How many remain?<|im_end|>\n<|im_start|>assistant\n" \
    --max_blocks 2 \
    --steps 8 \
    --solver dpm_solver \
    --use_tfe \
    --tfe_seeds 3

BibTeX Citation

@article{blockdiffuse2026,
  title={BlockDiffuse: Fully Parallel Latent Space Reasoning Generation with Diffusion Transformers},
  author={Hooshaai Research},
  journal={GitHub / HuggingFace Technical Report},
  year={2026},
  url={https://github.com/Hooshaai/BlockDiffuse}
}