How to clean up training data before fine-tuning a local language model
When we first fine-tuned an open-source model for our internal support bot the initial loss curve stopped completely by epoch two. We found out our scraper had collected thousands of forum signatures cookie banners and broken HTML tags that looked like text examples. Cleaning the dataset by hand took twelve hours. Removing that noise made our validation perplexity drop by nearly thirty percent, in a single night. You cannot put web scrapes into a training pipeline and expect clean outputs. Take time to check your raw corpus line by line before you start any training scripts. Look at your text splits for characters, repeated boilerplate and formatting problems that confuse tokenizers. What preprocessing steps do you usually use before starting a training run?
p/refs
p/runevr
p/reso
p/producthunt
p/google-cloud-platform
p/dex-by-exmergo
p/ai-context-flow
p/kodlo
p/seshka
p/duckfightclub
p/talanoa-2
p/nudgeforme
p/nexupira
p/many-displays
p/dino-kids
p/murror
p/operum
p/tabstack
p/rithub
Replies
Be the first to reply
Have a question or a thought to share? Add a comment above to start the conversation.