Training data language?
I noticed the public dataset for the instruction tuning seems to be Chinese only, but the description implied there is some English as well.
Does the trained base model have both?
For the base model. it was trained in this: https://huggingface.co/datasets/TriadParty/deepsword
There are also some English detective novels, but most of them were published in the past two years. I think this involves some copyright issues, so the data set has not been published.
Ah yeah I was mistaken, thats what I saw. It does seem to be mostly chinese.
I wonder how well that extrapolates to english martial arts writing...
Ah yeah I was mistaken, thats what I saw. It does seem to be mostly chinese.
I wonder how well that extrapolates to english martial arts writing...
I think there are some commonalities between them, but there may be some differences in the way of expression. In fact, I am also considering publishing a second edition recently. Some students who play games are very interested in this. If you are also interested, we can train together, hahaha
Mmm, well this is interesting to me, but I'm probably not much help? The real sticking point would be finding English martial arts literature in the first place.
I would suggest slapping unsloth into the training script if you don't already, should significantly speed things up and allow for higher contexts : https://github.com/unslothai/unsloth