Try to improve fts for big datasets by batches
Should fix https://github.com/huggingface/datasets-server/issues/2628 and help with search in big datasets.
When doing the 

```
FTS_BY_TABLE_COMMAND = (
    "SELECT * EXCLUDE (__hf_fts_score) FROM (SELECT *, fts_main_data.match_bm25(__hf_index_id, ?) AS __hf_fts_score"
    " FROM data) A WHERE __hf_fts_score IS NOT NULL ORDER BY __hf_fts_score DESC OFFSET {offset} LIMIT {length};"
)
```

FTS command performs the fts_main_data.match_bm25 macro over the whole "data" table in the index.duckdb file causing many OOM fails because it needs to load all the tables in memory that is for instance, for `wikimedia/wikipedia` en config, it tries to load nearly 9 GB of data (the current pod only has 6GB of memory).
I haven't found a better way to restrict and match to a better query plan than doing it by batches using __hf_index_id index. Hopefully, it will help a lot with big index files (I tried it, and worked super fast at least for the first pages), so I enabled it only for `wikimedia/wikipedia`, if it works, we can enable it for other datasets or use it for all.
**Note:**
Currently, it is like an "incremental" scanning, which can lead to a long query time if someone looks for page 1M for example, I was thinking of processing the batches in parallel, but it will be unnecessary for small datasets. Maybe in an upcoming PR, try to fit the better approach by the dataset size.
I also tried loading in two steps, first, only `SELECT __hf_index_id, fts_main_data.match_bm25(__hf_index_id, ?)` (which is super fast since it uses the __hf_index_id index) instead of `SELECT *, fts_main_data.match_bm25(__hf_index_id, ?)` and then join it with the data table but for some reason, it still gets OOM even if the first left table is reduced (Will continue investigating that approach).
Try to improve fts for big datasets by batches: Should fix https://github.com/huggingface/datasets-server/issues/2628 and help with search in big datasets.
When doing the 

```
FTS_BY_TABLE_COMMAND = (
    "SELECT * EXCLUDE (__hf_fts_score) FROM (SELECT *, fts_main_data.match_bm25(__hf_index_id, ?) AS __hf_fts_score"
    " FROM data) A WHERE __hf_fts_score IS NOT NULL ORDER BY __hf_fts_score DESC OFFSET {offset} LIMIT {length};"
)
```

FTS command performs the fts_main_data.match_bm25 macro over the whole "data" table in the index.duckdb file causing many OOM fails because it needs to load all the tables in memory that is for instance, for `wikimedia/wikipedia` en config, it tries to load nearly 9 GB of data (the current pod only has 6GB of memory).
I haven't found a better way to restrict and match to a better query plan than doing it by batches using __hf_index_id index. Hopefully, it will help a lot with big index files (I tried it, and worked super fast at least for the first pages), so I enabled it only for `wikimedia/wikipedia`, if it works, we can enable it for other datasets or use it for all.
**Note:**
Currently, it is like an "incremental" scanning, which can lead to a long query time if someone looks for page 1M for example, I was thinking of processing the batches in parallel, but it will be unnecessary for small datasets. Maybe in an upcoming PR, try to fit the better approach by the dataset size.
I also tried loading in two steps, first, only `SELECT __hf_index_id, fts_main_data.match_bm25(__hf_index_id, ?)` (which is super fast since it uses the __hf_index_id index) instead of `SELECT *, fts_main_data.match_bm25(__hf_index_id, ?)` and then join it with the data table but for some reason, it still gets OOM even if the first left table is reduced (Will continue investigating that approach).