Possible target: Diffusion Language Models

[BradKML]

One of the things I can think of that would be useful, similar to the direction of ASI-Arch but closer to NanoPoor in resource constraints https://github.com/GAIR-NLP/ASI-Arch https://github.com/VatsaDev/NanoPoor

1. Lower GPU and VRAM resources than most researchers (ideally close to consumer GPU with gaming rigs)
2. Research should be structured such that quick iteration is possible (<4 hours)
3. Benchmarking should be diverse enough to prevent overfitting and reward hacking
4. Every bit of research (both literature and experiments) should be logged in MarkDown format for easy resuming

Reference:

• https://github.com/ML-GSAI/LLaDA
• https://github.com/DreamLM/Dream and https://github.com/HKUNLP/DiffuLLaMA
• https://github.com/kuleshov-group/bd3lms and https://github.com/kuleshov-group/mdlm
• https://github.com/Hzfinfdu/Diffusion-BERT
• https://github.com/dllm-reasoning/d1
• https://github.com/HKUNLP/diffusion-of-thoughts

P.S. I might have gotten a bit talkative over at https://github.com/Jianguo99/OpenDLM/issues