geekyrakshit/indian-entrance-exams-benchmark
Viewer ā¢ Updated ā¢ 2.16k ā¢ 180 ā¢ 1
š¤ Open to Collab
Soumik Rakshit
geekyrakshit
AI & ML interests
Computer vision
Recent Activity
updated a dataset 27 days ago
geekyrakshit/indian-entrance-exams-benchmark published a dataset 27 days ago
geekyrakshit/indian-entrance-exams-benchmark updated a dataset about 1 month ago
geekyrakshit/indian-exam-questionsOrganizations
updated a
dataset 27 days ago
published a
dataset 27 days ago
updated a
dataset about 1 month ago
published a
dataset about 1 month ago
reacted to m-ric's post with š about 1 month ago
Post
2726
šš®š š š¢š§š š
ššš š«šš„ššš¬šš¬ šš¢ššØšš«šØš§, š š¦š¢šš«šØš¬ššØš©š¢š š„š¢š šš”šš š¬šØš„šÆšš¬ ššš šš«šš¢š§š¢š§š šš š©šš«šš„š„šš„š¢š³ššš¢šØš§ š„³
š°ļø Llama-3.1-405B took 39 million GPU-hours to train, i.e. about 4.5 thousand years.
š“š» If they had needed all this time, we would have GPU stories from the time of Pharaoh š: "Alas, Lord of Two Lands, the shipment of counting-stones arriving from Cathay was lost to pirates, this shall delay the building of your computing temple by many moons "
š ļø But instead, they just parallelized the training on 24k H100s, which made it take just a few months.
This required parallelizing across 4 dimensions: data, tensor, context, pipeline.
And it is infamously hard to do, making for bloated code repos that hold together only by magic.
š¤ ššš š»š¼š šš² š±š¼š»'š š»š²š²š± šµšš“š² šæš²š½š¼š š®š»ššŗš¼šæš²! Instead of building mega-training codes, Hugging Face colleagues cooked in the other direction, towards tiny 4D parallelism libs. A team has built Nanotron, already widely used in industry.
And now a team releases Picotron, a radical approach to code 4D Parallelism in just a few hundred lines of code, a real engineering prowess, making it much easier to understand what's actually happening!
ā” šš'š šš¶š»š, šš²š š½š¼šš²šæš³šš¹:
Counting in MFU (Model FLOPs Utilization, how much the model actually uses all the compute potential), this lib reaches ~50% on SmolLM-1.7B model with 8 H100 GPUs, which is really close to what huge libs would reach. (Caution: the team is leading further benchmarks to verify this)
Go take a look š https://github.com/huggingface/picotron/tree/main/picotron
š°ļø Llama-3.1-405B took 39 million GPU-hours to train, i.e. about 4.5 thousand years.
š“š» If they had needed all this time, we would have GPU stories from the time of Pharaoh š: "Alas, Lord of Two Lands, the shipment of counting-stones arriving from Cathay was lost to pirates, this shall delay the building of your computing temple by many moons "
š ļø But instead, they just parallelized the training on 24k H100s, which made it take just a few months.
This required parallelizing across 4 dimensions: data, tensor, context, pipeline.
And it is infamously hard to do, making for bloated code repos that hold together only by magic.
š¤ ššš š»š¼š šš² š±š¼š»'š š»š²š²š± šµšš“š² šæš²š½š¼š š®š»ššŗš¼šæš²! Instead of building mega-training codes, Hugging Face colleagues cooked in the other direction, towards tiny 4D parallelism libs. A team has built Nanotron, already widely used in industry.
And now a team releases Picotron, a radical approach to code 4D Parallelism in just a few hundred lines of code, a real engineering prowess, making it much easier to understand what's actually happening!
ā” šš'š šš¶š»š, šš²š š½š¼šš²šæš³šš¹:
Counting in MFU (Model FLOPs Utilization, how much the model actually uses all the compute potential), this lib reaches ~50% on SmolLM-1.7B model with 8 H100 GPUs, which is really close to what huge libs would reach. (Caution: the team is leading further benchmarks to verify this)
Go take a look š https://github.com/huggingface/picotron/tree/main/picotron
- 1 reply
liked a
dataset about 1 month ago
upvoted an article about 2 months ago
Article
Profiling in PyTorch (Part 1): A Beginner's Guide to torch.profiler
- +3
ariG23498, sayakpaul, sergiopaniego, ror, pcuenq
ā¢ ā¢ 154 updated a
dataset 2 months ago
published a
dataset 2 months ago
upvoted a paper 3 months ago
liked a
dataset 4 months ago
upvoted an article 4 months ago
Article
Running Native PyTorch on TPUs with Zero Code Changes
rishiraj
ā¢ ā¢ 6 upvoted an article 5 months ago
Article
Mixture of Experts (MoEs) in Transformers
- +5
ariG23498, pcuenq, merve, IlyasMoutawwakil, ArthurZ, sergiopaniego, Molbap
ā¢ ā¢ 173 updated a
dataset 6 months ago
published a
dataset 6 months ago
updated a
dataset 7 months ago
published a
dataset 7 months ago
updated a
dataset 7 months ago
published a
dataset 7 months ago
updated a
dataset 7 months ago