---
title: "Analyze Dataset"
method: POST
path: "/felix/dataset/analyze"
tags: ["felix"]
---

# Analyze Dataset

`POST /felix/dataset/analyze`

Analyze a dataset for quality, distribution, and potential issues.

Supports NER, Classification, and Generative task types. Available
analyses: distribution, duplicates, outliers, correlation, splits,
errors, and diversity (Vendi score + embedding visualisation).

Provide data inline via ``dataset`` or reference a stored dataset
with ``dataset_name``.  See ``DatasetAnalysisRequest`` schema for
the full set of options including diversity visualisation config.

## Request body

- DatasetAnalysisRequest — Request for dataset analysis
  - `task_type` 'ner' | 'classification' | 'generative', required — Task type of the dataset
  - `task_description` string, nullable — Description of the task/domain for context. Helps the LLM quality analysis understand the intended use case.
  - `dataset` object[], nullable — List of data samples (optional if dataset_name provided)
  - `dataset_name` string, nullable — Name of stored dataset to analyze (optional if dataset provided)
  - `dataset_version` string, nullable — Dataset version (latest if omitted)
  - `analyses` string[], required — List of analyses to perform
  - `query` string, nullable — Natural language question about the dataset
  - `predictions` object[], nullable — Optional model predictions for error analysis
  - `options` object, nullable — Configuration options for analysis. Supported keys: `validation_percentage` (float) for split analysis and `diversity_visualization` (object) with optional `method` ('pca'|'tsne'|'umap'), `dimensions` (2-4), and `tsne_perplexity` (float).

## Response `200`

Successful Response

- DatasetAnalysisResponse — Response containing analysis results
  - `summary` object, required
  - `distribution` DistributionAnalysis — Result of distribution analysis
    - `text_length_stats` DistributionStats — Statistics for a numerical distribution
      - `min` number, required
      - `max` number, required
      - `mean` number, required
      - `median` number, required
      - `std` number, required
    - `token_count_stats` DistributionStats — Statistics for a numerical distribution
      - `min` number, required
      - `max` number, required
      - `mean` number, required
      - `median` number, required
      - `std` number, required
    - `token_count_histogram` HistogramBucket[], nullable
      - `range` string, required
      - `count` integer, required
    - `entity_counts` EntityCount[], nullable
      - `label` string, required
      - `count` integer, required
      - `percentage` number, required
    - `span_length_stats` DistributionStats — Statistics for a numerical distribution
      - `min` number, required
      - `max` number, required
      - `mean` number, required
      - `median` number, required
      - `std` number, required
    - `span_length_by_type` object, nullable
    - `most_frequent_spans` SpanFrequency[], nullable
      - `text` string, required
      - `count` integer, required
    - `class_balance` ClassBalance[], nullable
      - `label` string, required
      - `count` integer, required
    - `label_correlation` LabelCorrelation
      - `labels` string[], required
      - `matrix` array[], required
        - integer[]
    - `prompt_length_stats` DistributionStats — Statistics for a numerical distribution
      - `min` number, required
      - `max` number, required
      - `mean` number, required
      - `median` number, required
      - `std` number, required
    - `completion_length_stats` DistributionStats — Statistics for a numerical distribution
      - `min` number, required
      - `max` number, required
      - `mean` number, required
      - `median` number, required
      - `std` number, required
  - `duplicates` DuplicatesAnalysis
    - `duplicate_count` integer, required
    - `duplication_rate` number, required
    - `samples` DuplicateSample[], required
      - `original_index` integer, required
      - `duplicate_index` integer, required
      - `preview` string, required
  - `outliers` OutliersAnalysis
    - `outlier_count` integer, required
    - `thresholds` OutlierThresholds, required
      - `mean_length` number, nullable
      - `upper_bound_length` number, nullable
      - `lower_bound_length` number, nullable
    - `samples` OutlierSample[], required
      - `index` integer, required
      - `value` number, required
      - `z_score` number, required
      - `type` string, required
      - `sample` object, required
      - `fingerprint` string, nullable — Content-based SHA-256 fingerprint for stable identification
  - `correlations` object, nullable
  - `splits` SplitsAnalysis
    - `train_count` integer, required
    - `val_count` integer, required
    - `train_distribution` LabelDistribution[], required
      - `label` string, required
      - `count` integer, required
      - `percentage` number, required
    - `val_distribution` LabelDistribution[], required
      - `label` string, required
      - `count` integer, required
      - `percentage` number, required
    - `split_ratio_actual` SplitRatio, required
      - `train` number, required
      - `val` number, required
  - `errors` object, nullable
  - `diversity` DiversityAnalysis
    - `vendi_score` number, required
    - `sample_size` integer, required
    - `interpretation` string, nullable
    - `visualization` DiversityVisualization
      - `method` 'pca' | 'tsne', required
      - `dimensions` integer, required
      - `points` DiversityPoint[], required
        - `x` number, nullable
        - `y` number, nullable
        - `z` number, nullable
        - `w` number, nullable
        - `coordinates` number[], required
        - `text` string, nullable
        - `token_count` integer, nullable
        - `labels` string[], nullable
        - `metadata` object, nullable
        - `sample_index` integer, nullable
        - `similarity_to_centroid` number, nullable
        - `embedding` number[], nullable
      - `tsne_perplexity` number, nullable
      - `similarity_range` object, nullable
      - `token_count_range` object, nullable
    - `llm_analysis` DiversityLLMAnalysis — LLM-generated reasoning analysis of dataset diversity. Mimics how a machine learning engineer would analyze diversity.
      - `reasoning_trace` string, required — Step-by-step reasoning trace showing how the model analyzed the dataset's diversity, similar to an ML engineer's thought process
      - `summary` string, required — Concise summary of the diversity assessment
      - `diversity_rating` 'low' | 'moderate' | 'high' | 'excellent', required — Overall diversity rating based on the analysis
      - `key_observations` string[] — Key observations about the dataset's diversity
      - `recommendations` string[] — Actionable recommendations to improve diversity if needed
      - `model_used` string, required — The LLM model used for the analysis
  - `natural_language_response` string, nullable

## Other responses

- `422` — Validation Error

---

[API](https://skmtc.net/pioneer/apis/brain-api.md) · [All operations](https://skmtc.net/pioneer/apis/brain-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/pioneer/brain-api/revisions/b92f75fd3b61/schema)