Api Datasets Logs List List

List dataset logs with filtering, pagination, and full-object retrieval.

Endpoints: GET /api/datasets/{dataset_id}/logs/list/ - List logs with pagination POST /api/datasets/{dataset_id}/logs/list/ - List logs with complex filtering

Query Parameters (GET): - page (integer, optional): Page number for pagination (default: 1) - page_size (integer, optional): Number of results per page - sort_by (string, optional): Field to order results by (default: unique_id) - retrieval_mode (string, optional): "async" for background full-object loading

Request Body (POST): - filters (object, optional): Complex filter criteria - page (integer, optional): Page number - page_size (integer, optional): Results per page

Response (200 OK): - count (integer): Total number of logs matching filters - next (string|null): URL for next page - previous (string|null): URL for previous page - results (array): Array of log objects - filter_options (object): Available filter options

Under-the-hood optimizations:

1. Async Full Object Preloading (retrieval_mode="async")

• When listing logs, the API returns immediately with shallow ClickHouse fields
• A background Celery task (preload_full_objects_task) is triggered to: a. Fetch full log objects from S3 storage (including input/output) b. Write them to Redis cache with key: request_log_full_object_{unique_id} c. Cache TTL: 300 seconds (5 minutes)
• Subsequent detail view requests get instant cache hits
• Callback: evaluation.utils.store_dataset_log_full_objects_to_cache

2. Overlay Precedence for Updated Logs

• When a log is updated, an overlay file is written to S3 with the new data
• ClickHouse row is updated with updated_storage_object_key pointing to overlay
• The preload task retrieves BOTH base index AND overlay keys
• Deduplication: If a unique_id is found in BOTH, overlay takes precedence (tracked via ids_to_retrieve set in batch_retrieve_full_objects)

3. ArgMax Deduplication

• Uses ClickHouse's argMax to get the latest version of each log
• Deduplicates by unique_id, sorted by updated_at field
• Ensures only the most recent version of each log is shown

4. Storage Structure

• Index files (.idx): {"unique_id1": {object1}, "unique_id2": {object2}}
• Overlay files: {object} directly (single log, not wrapped)
• Both are handled transparently in batch_retrieve_full_objects

Performance characteristics:

• List response: ~100-200ms (ClickHouse query only, no S3)
• Cache population: 1-5s background (depends on log count)
• Detail view after list: ~10ms (Redis cache hit)