memory: use pymongoarrow to get dataset results as dataframe
As stated in the code, the current conversion from a list of mongo entries to a dataframe is not necessarily optimal:

https://github.com/huggingface/dataset-viewer/blob/27edd1f472816f70ab6723fe7fdcb96fa8375209/libs/libcommon/src/libcommon/simple_cache.py#L813-L833

see also

https://github.com/huggingface/dataset-viewer/blob/27edd1f472816f70ab6723fe7fdcb96fa8375209/libs/libcommon/src/libcommon/queue.py#L994-L1019

We might benefit from using https://github.com/mongodb-labs/mongo-arrow/tree/main/bindings/python for that (recommended way from the mongo team)

> PyMongoArrow is the recommended way to materialize MongoDB query result sets as contiguous-in-memory, typed arrays suited for in-memory analytical processing applications.

Some comments:
- we have to implement unit tests on these methods before switching to be sure we don't break anything
- we could add memory tests (`@pytest.mark.limit_memory()`, see https://bloomberg.github.io/memray/tutorials/additional_features.html#pytest-plugin) to ensure we reduce the memory footprint
- pymongoarrow requires a specific version of pyarrow (currently: ^15, while we use ^14, and the last pyarrow version is ^16 :) )
- will it work nicely with the types?

memory: use pymongoarrow to get dataset results as dataframe: As stated in the code, the current conversion from a list of mongo entries to a dataframe is not necessarily optimal:

https://github.com/huggingface/dataset-viewer/blob/27edd1f472816f70ab6723fe7fdcb96fa8375209/libs/libcommon/src/libcommon/simple_cache.py#L813-L833

see also

https://github.com/huggingface/dataset-viewer/blob/27edd1f472816f70ab6723fe7fdcb96fa8375209/libs/libcommon/src/libcommon/queue.py#L994-L1019

We might benefit from using https://github.com/mongodb-labs/mongo-arrow/tree/main/bindings/python for that (recommended way from the mongo team)

> PyMongoArrow is the recommended way to materialize MongoDB query result sets as contiguous-in-memory, typed arrays suited for in-memory analytical processing applications.

Some comments:
- we have to implement unit tests on these methods before switching to be sure we don't break anything
- we could add memory tests (`@pytest.mark.limit_memory()`, see https://bloomberg.github.io/memray/tutorials/additional_features.html#pytest-plugin) to ensure we reduce the memory footprint
- pymongoarrow requires a specific version of pyarrow (currently: ^15, while we use ^14, and the last pyarrow version is ^16 :) )
- will it work nicely with the types?
