Rate-limit the updates?
See https://huggingface.co/datasets/lunaluan/chatbox8_history/commits/main: two commits are created every minute (47K commits). This dataset is currently blocked manually (https://github.com/huggingface/datasets-server/pull/2184).

Should we create a rule to automatically limit the number of updates per hour/day? Note that doing so might imply changing how we handle the "revision". If we decide to process a specific commit, all the steps should use the same commit, even if new ones have appeared since.
Rate-limit the updates?: See https://huggingface.co/datasets/lunaluan/chatbox8_history/commits/main: two commits are created every minute (47K commits). This dataset is currently blocked manually (https://github.com/huggingface/datasets-server/pull/2184).

Should we create a rule to automatically limit the number of updates per hour/day? Note that doing so might imply changing how we handle the "revision". If we decide to process a specific commit, all the steps should use the same commit, even if new ones have appeared since.