[W]hen a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data... Oh, and we have about 922 forks of it on git.kernel.org — but don't worry, it's actually extremely efficient on the backend, since it's mostly the same objects in every fork.
[W]hen a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data...
Oh, and we have about 922 forks of it on git.kernel.org — but don't worry, it's actually extremely efficient on the backend, since it's mostly the same objects in every fork.
Unless, of course, you're a scraper, in which case you have, oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits — which is exactly what the scrapers...
The page is ready to read now. The fuller skim-friendly version will appear here automatically.
[W]hen a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data... Oh, and we have about 922 forks of it on git.kernel.org — but don't worry, it's actually extremely efficient on the backend, since it's mostly the same objects in every fork. Unless, of course, you're a scraper, in which case you have, oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits — which is exactly what the scrapers are doing.
The Linux Foundation's director of IT infrastructure says they now spend more CPU cycles "rendering commits for scrapers than we spend on all other kinds of legitimate access." At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.... At the time of writing, linux.git is about 1.48 million commits.
Open the app view to save this story, compare related coverage, and continue from the same source.