VSeek-R1

Official codebase for VSeek-R1 — a tool-using video agent that learns to seek answer-critical evidence in long videos via reinforcement learning (GRPO), rather than ingesting dense frame stacks end-to-end.

Released weights, data, and index

Download these from SWARM-Lab instead of rebuilding them. Each one stands in for a later step in this README.

Download	Use it for	Replaces
VSeek-EM and VSeek-VETL	`--model-path` in Inference and Evaluation	Training (GRPO) and Merge FSDP checkpoints → Hugging Face format
VSeek-data	`data.train_files` in Training (GRPO) and the test set in Evaluation	Process Datasets step 5 (`preprocess_lvb.sh`, `preprocess_mlvu.sh`, `preprocess_videomme.sh`, `preprocess_cgbench.sh`, `preprocess_lvbench.sh`)
VSeek-Index	`retriever.index_path` / `$INDEX_PATH` in Run the Retriever Server	Process Datasets step 3 (`run_data_pipeline.py` for `lvb`, `lvbench`, `videomme`, `mlvu`, and `cgbench`)

train.parquet (4,968 rows) is LongVideoBench, MLVU, and Video-MME. test.parquet (2,153 rows) is those three plus CG-Bench and LVBench.

The index is the window-8 tree ({dataset}_window_8/{video_id}/ with embeddings.pt, frames.mp4, and metadata.json). You still start the retriever server yourself. You also still need the ViCLIP files at the end of Download Datasets: ViClip-InternVid-10M-FLT.pth and bpe_simple_vocab_16e6.txt.gz. Raw benchmark videos from Download Datasets, plus unzip and subtitle burn in Process Datasets steps 1–2, are still required for the server to register each video.

huggingface-cli download SWARM-Lab/VSeek-EM --local-dir checkpoints/VSeek-EM
huggingface-cli download SWARM-Lab/VSeek-VETL --local-dir checkpoints/VSeek-VETL
huggingface-cli download SWARM-Lab/VSeek-data --repo-type dataset --local-dir data/VSeek-data
huggingface-cli download SWARM-Lab/VSeek-Index --repo-type dataset --local-dir data/VSeek-Index

Slurm / HPC (TACC Vista): see README_SLURM.md.

Branches: main tracks v0.6.1. Both use the local lab paths (/nas/mars/..., /home/.../vseek/dataset). harsh_tacc is the branch with TACC/Slurm paths ($SCRATCH, $WORK, port 9000). See Path conventions and README_SLURM.md.

Path conventions (local vs TACC)

Configs and launch scripts hard-code filesystem roots. main tracks v0.6.1, and both assume the local shared NAS. harsh_tacc is the TACC/Slurm branch and retargets only some of those files to Vista ($SCRATCH / $WORK). Mixing branches or launching training scripts without checking paths will break the retrieval server and tool calls.

What actually differs between branches

Audited with git grep over both trees (excluding vendor/). These are the files whose dataset / index / weight roots change nas/home ↔ scratch/work:

File	Local (`v0.6.1`)	TACC (`harsh_tacc`)
`src/vseek/config/retriever/config.yaml`	`/nas/mars/dataset/...`, `/home/.../vseek/dataset`, ViCLIP under `/nas/mars/model_weights/`, port 9005	`$SCRATCH/.../datasets/...`, `$SCRATCH/.../vseek`, ViCLIP under `$SCRATCH/.../model_weights/`, port 9000
`src/vseek/config/retriever/toolconfig.yaml`	same local roots (note: MLVU is `/nas/mars/dataset/MLVU` here vs `.../MLVU/MLVU` in `config.yaml`)	`$SCRATCH` roots; MLVU uses `.../MLVU/MLVU`
`scripts/tacc/*` (harsh only)	n/a	parquet / checkpoints under `$WORK/.../vista/...`; `PROJECT_ROOT` fallback `/home1/.../VSeek-R1`; `HF_HOME` → `$WORK` or `$SCRATCH`

Resource	Local (`v0.6.1`)	TACC (`harsh_tacc` / Slurm)
Raw / burned videos	`/nas/mars/dataset/...`	`$SCRATCH/.../datasets/...`
FAISS / frame indexes	`/home/<user>/vseek/dataset`	`$SCRATCH/.../vseek`
ViCLIP weights	`/nas/mars/model_weights/viclip/`	`$SCRATCH/.../model_weights/viclip/`
RL parquet (Slurm jobs)	`/nas/mars/vseek/...` or per-dataset `window_8/`	`$WORK/.../vista/vseek/tagsummary/{train,test}.parquet`
Checkpoints (Slurm jobs)	local `checkpoints/`	`$WORK/.../vista/checkpoints` or `$SCRATCH/checkpoints`
Retriever port	`9005`	`9000`

Still local (`/nas/mars`) on both branches — override on TACC

These were not rewritten on harsh_tacc. On slurm you must override them (CLI / env / edit) or avoid them in favor of scripts/tacc/:

scripts/train/*.sh — data.train_files / data.val_files still under /nas/mars/...
scripts/data_ops/preprocess_*.sh, scripts/utils/unzip_videos.sh — default DATASET_PATH / INDEX_PATH / CHUNKS_DIR under /nas/mars (and CGBench preprocess defaults INDEX_PATH to /home/hg22723/vseek/dataset)
scripts/evals/vllm_vseek_tests_ckpt.sh — per-dataset parquet roots under /nas/mars/dataset/.../window_8/
Scratch / demo scripts with hardcoded PROJECT_ROOT=/home/hg22723/projects/VSeek-R1 (e.g. scripts/run_agent_data_vllmrollout.py, scripts/run_api_eval.py, scripts/run_video_tree.py, scripts/sftdata/run_gpt_agent_data.py)
scripts/puls/*.sh, VideoTree baseline wrappers, and some notebooks — local-only paths (many exist only on v0.6.1)

TACC/SLURM only paths `harsh_tacc`

additional_readme.md mentions indexes/data under $WORK/.../vista/{vseek,datasets}, while config.yaml / toolconfig.yaml on that branch point at $SCRATCH/.../{vseek,datasets}. Confirm which tree you actually populated.
run_vseek_job*.slurm hard-codes CHECKPOINT_DIR / parquet under /work/11123/harshgoel99/...; run_vseek_all_jobs*.slurm often uses ${SCRATCH}/checkpoints and env-provided TRAIN_DATA_PATH.
HF_HOME is $WORK/huggingface in some jobs and $SCRATCH/huggingface in others / ray_wrapper.sh.
Slurm PROJECT_ROOT fallback is /home1/11123/harshgoel99/VSeek-R1 (TACC home), not the lab /home/hg22723/projects/....

Files to touch (or override via Hydra) when moving between local and slurm:

src/vseek/config/retriever/config.yaml
src/vseek/config/retriever/toolconfig.yaml
scripts/tacc/run_vseek_job.slurm / vseek_grpo_tag_qvl34Bt_vllm_tagsummary.sh
Any scripts/train/*.sh or preprocess/eval script you still invoke on that machine
Optionally src/vseek/setting.py

On TACC, prefer Hydra / CLI overrides over committing personal $SCRATCH / $WORK paths. Full Slurm examples: README_SLURM.md.

Overview

VSeek-R1 couples a vision–language model with a video retrieval toolkit (ViCLIP + FAISS indexes over temporal windows). The agent interacts in multi-turn think → tool call → observe loops, and is trained with Group Relative Policy Optimization (GRPO) on top of verl.

Supported benchmarks (download + preprocess scripts included):

Dataset	Hugging Face repo	Config name
LongVideoBench	`lmms-lab/LongVideoBench`	`lvb`
LVBench	`lmms-lab/LVBench`	`lvbench`
Video-MME	`lmms-lab/Video-MME`	`videomme`
MLVU	`MLVU/MVLU`	`mlvu`
CGBench	(local / custom)	`cgbench`

Requirements

Linux + NVIDIA GPU (CUDA 12.4+ recommended)
Python ≥ 3.11
Conda or uv
Sufficient disk for video datasets (hundreds of GB)

Installation

1. Clone the repository

git clone https://github.com/UTAustin-SwarmLab/VSeek-R1.git
cd VSeek-R1
git submodule update --init --recursive   # pulls vendor/verl

2. Create an environment

Option A — uv (recommended on a workstation):

pip install uv
uv venv .venv
source .venv/bin/activate
uv sync

Option B — conda:

conda create -n vseek python=3.11 -y
conda activate vseek
conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia

3. Install CUDA build deps + FlashAttention

export CUDA_HOME="${CUDA_HOME:-$CONDA_PREFIX}"

# Choose one inference stack
pip install -r requirements_vllm.txt      # vLLM (default for training / eval)
# or
pip install -r requirements_sglang.txt    # SGLang

pip install flash-attn==2.8.3 --no-build-isolation

4. Install VSeek + verl

pip install -e .
pip install -e vendor/verl

5. Sanity check

python -c "import torch; import vllm; print(torch.__version__, torch.cuda.is_available())"

HPC / Slurm users: do not follow this path blindly on TACC Vista (Grace Hopper / ARM). Use README_SLURM.md instead — vLLM and FlashAttention must be built for sm_90.

Download Datasets

Set a root directory and download with huggingface-cli (faster than the Python helper):

export DATASET_ROOT=/path/to/datasets   # e.g. /nas/mars/dataset
mkdir -p "$DATASET_ROOT"

huggingface-cli download MLVU/MVLU --repo-type dataset \
  --local-dir "$DATASET_ROOT/MLVU"

huggingface-cli download lmms-lab/Video-MME --repo-type dataset \
  --local-dir "$DATASET_ROOT/Video-MME"

huggingface-cli download lmms-lab/LVBench --repo-type dataset \
  --local-dir "$DATASET_ROOT/LVBench"

huggingface-cli download lmms-lab/LongVideoBench --repo-type dataset \
  --local-dir "$DATASET_ROOT/longvideobench"

Or download everything in one loop:

for dataset in \
  "MLVU/MVLU:MLVU" \
  "lmms-lab/Video-MME:Video-MME" \
  "lmms-lab/LVBench:LVBench" \
  "lmms-lab/LongVideoBench:longvideobench"
do
  IFS=':' read -r repo_id local_dir <<< "$dataset"
  echo "→ $repo_id → $DATASET_ROOT/$local_dir"
  huggingface-cli download "$repo_id" --repo-type dataset \
    --local-dir "$DATASET_ROOT/$local_dir"
done

Also place ViCLIP weights somewhere accessible and point configs at them (defaults live under src/vseek/setting.py):

ViClip-InternVid-10M-FLT.pth
bpe_simple_vocab_16e6.txt.gz

Process Datasets

Each dataset follows the same pattern:

Unzip videos / burn subtitles (when needed)
Convert parquet → JSON (Video-MME, LVBench)
Build a retrieval index (run_data_pipeline.py)
Optionally extract PULS primitives
Create train/test parquet for RL (preprocess_*.sh)

Set shared paths once:

export DATASET_ROOT=/path/to/datasets
export INDEX_PATH=/path/to/vseek/indexes   # FAISS / frame indexes
export WINDOW_SIZE=8
export PROMPT_TYPE=tagsummary              # or: tag

LongVideoBench (`lvb`)

# Burn subtitles into frames
python3 scripts/utils/burn_subtitles.py \
  --json-file lvb_val.json \
  --output-dir "$DATASET_ROOT/longvideobench/burn-subtitles" \
  --data-folder "$DATASET_ROOT/longvideobench/LongVideoBench/"

# Index windows
python3 scripts/data_ops/run_data_pipeline.py \
  retriever.window_size=$WINDOW_SIZE \
  dataset.name=lvb \
  dataset.lvb.dataset_path="$DATASET_ROOT/longvideobench/LongVideoBench/" \
  dataset.lvb.burned_path="$DATASET_ROOT/longvideobench/" \
  retriever.index_path="$INDEX_PATH"

# Optional PULS extraction
python3 src/vseek/puls/primitives.py dataset.name=lvb

# RL parquet splits
bash scripts/data_ops/preprocess_lvb.sh \
  --local_dataset_path "$DATASET_ROOT/longvideobench/LongVideoBench" \
  --burned_path "$DATASET_ROOT/longvideobench/" \
  --train_ratio 0.8 \
  --local_save_dir "$DATASET_ROOT/longvideobench" \
  --index_path "$INDEX_PATH" \
  --window_size $WINDOW_SIZE \
  --prompt_type $PROMPT_TYPE

LVBench (`lvbench`)

CHUNKS_DIR="$DATASET_ROOT/LVBench/video_chunks" \
OUTPUT_DIR="$DATASET_ROOT/LVBench/videos" \
  bash scripts/utils/unzip_videos.sh

python3 scripts/utils/convert_parquet.py \
  --parquet_path "$DATASET_ROOT/LVBench/data/train-00000-of-00001.parquet" \
  --json_path "$DATASET_ROOT/LVBench/data/lvbench_val.json"

python3 scripts/data_ops/run_data_pipeline.py \
  retriever.window_size=$WINDOW_SIZE \
  dataset.name=lvbench \
  dataset.lvbench.dataset_path="$DATASET_ROOT/LVBench" \
  dataset.lvbench.burned_path="$DATASET_ROOT/LVBench/videos" \
  retriever.index_path="$INDEX_PATH"

python3 src/vseek/puls/primitives.py dataset.name=lvbench

bash scripts/data_ops/preprocess_lvbench.sh \
  --local_dataset_path "$DATASET_ROOT/LVBench" \
  --burned_path "$DATASET_ROOT/LVBench/videos" \
  --train_ratio 0.8 \
  --local_save_dir "$DATASET_ROOT/LVBench" \
  --index_path "$INDEX_PATH" \
  --window_size $WINDOW_SIZE \
  --prompt_type $PROMPT_TYPE

Video-MME (`videomme`)

SUBTITLE_DIR="$DATASET_ROOT/Video-MME" \
CHUNKS_DIR="$DATASET_ROOT/Video-MME" \
OUTPUT_DIR="$DATASET_ROOT/Video-MME/videos" \
  bash scripts/utils/unzip_videos.sh

python3 scripts/utils/convert_parquet.py \
  --parquet_path "$DATASET_ROOT/Video-MME/videomme/test-00000-of-00001.parquet" \
  --json_path "$DATASET_ROOT/Video-MME/videomme/videomme_val.json"

python3 scripts/utils/burn_subtitles.py \
  --json-file videomme/videomme_val.json \
  --output-dir "$DATASET_ROOT/Video-MME/burn-subtitles" \
  --data-folder "$DATASET_ROOT/Video-MME"

python3 scripts/data_ops/run_data_pipeline.py \
  dataset.name=videomme \
  retriever.window_size=$WINDOW_SIZE \
  dataset.videomme.dataset_path="$DATASET_ROOT/Video-MME/" \
  dataset.videomme.burned_path="$DATASET_ROOT/Video-MME/burn-subtitles/" \
  retriever.index_path="$INDEX_PATH" \
  retriever.gpu_number=0

python3 src/vseek/puls/primitives.py dataset.name=videomme

bash scripts/data_ops/preprocess_videomme.sh \
  --local_dataset_path "$DATASET_ROOT/Video-MME" \
  --burned_path "$DATASET_ROOT/Video-MME/burn-subtitles" \
  --train_ratio 0.8 \
  --local_save_dir "$DATASET_ROOT/Video-MME" \
  --index_path "$INDEX_PATH" \
  --window_size $WINDOW_SIZE \
  --prompt_type $PROMPT_TYPE

MLVU (`mlvu`)

Tasks: plotQA, needle, ego, count, order, anomaly_reco, topic_reasoning.

python3 scripts/data_ops/run_data_pipeline.py \
  retriever.window_size=$WINDOW_SIZE \
  dataset.name=mlvu \
  dataset.mlvu.dataset_path="$DATASET_ROOT/MLVU/MLVU" \
  dataset.mlvu.burned_path="$DATASET_ROOT/MLVU/MLVU" \
  retriever.index_path="$INDEX_PATH"

python3 src/vseek/puls/primitives.py dataset.name=mlvu

bash scripts/data_ops/preprocess_mlvu.sh \
  --local_dataset_path "$DATASET_ROOT/MLVU/MLVU" \
  --burned_path "$DATASET_ROOT/MLVU/MLVU" \
  --train_ratio 0.8 \
  --local_save_dir "$DATASET_ROOT/MLVU" \
  --index_path "$INDEX_PATH" \
  --window_size $WINDOW_SIZE \
  --prompt_type $PROMPT_TYPE

CGBench (`cgbench`)

SUBTITLE_DIR="$DATASET_ROOT/CGBench" \
CHUNKS_DIR="$DATASET_ROOT/CGBench" \
OUTPUT_DIR="$DATASET_ROOT/CGBench/videos" \
  bash scripts/utils/unzip_videos.sh

python3 scripts/utils/burn_subtitles.py \
  --json-file cgbench_mini.json \
  --output-dir "$DATASET_ROOT/CGBench/burn-subtitles" \
  --data-folder "$DATASET_ROOT/CGBench"

python3 scripts/data_ops/run_data_pipeline.py \
  dataset.name=cgbench \
  retriever.window_size=$WINDOW_SIZE \
  dataset.cgbench.dataset_path="$DATASET_ROOT/CGBench/" \
  dataset.cgbench.burned_path="$DATASET_ROOT/CGBench/burn-subtitles/" \
  retriever.index_path="$INDEX_PATH"

bash scripts/data_ops/preprocess_cgbench.sh \
  --local_dataset_path "$DATASET_ROOT/CGBench" \
  --burned_path "$DATASET_ROOT/CGBench/burn-subtitles" \
  --train_ratio 0.8 \
  --local_save_dir "$DATASET_ROOT/CGBench" \
  --index_path "$INDEX_PATH" \
  --window_size $WINDOW_SIZE \
  --prompt_type $PROMPT_TYPE

Run the Retriever Server

The agent calls a Flask retrieval server during training and inference.

On local setups (v0.6.1), defaults in config.yaml / toolconfig.yaml use /nas/mars/... datasets and /home/.../vseek/dataset indexes (port 9005). On TACC, those must be rewritten to $SCRATCH / $WORK paths (port 9000) — see Path conventions and README_SLURM.md.

CUDA_VISIBLE_DEVICES=0 python3 src/vseek/tools/server.py \
  retriever.window_size=$WINDOW_SIZE \
  dataset.name=['lvb','lvbench','videomme','mlvu'] \
  retriever.index_path="$INDEX_PATH" \
  retriever.retrieval_model_path=/path/to/ViClip-InternVid-10M-FLT.pth \
  retriever.text_encoder_model_path=/path/to/bpe_simple_vocab_16e6.txt.gz

Update src/vseek/config/retriever/toolconfig.yaml so the agent points at this server URL and the same dataset/index roots the server was started with.

Inference

Uniform frame baseline

CUDA_VISIBLE_DEVICES=0 python3 scripts/run_uniform_agent_data.py \
  inference.max_images_per_turn=64 \
  +inference.agent_prompt_type=cot \
  llm.model=Qwen/Qwen3-VL-4B-Thinking \
  inference.output_dir=results/uniform_q3_4bt_cot \
  inference.max_output_tokens=2048 \
  inference.gpu_number=0

VSeek agent (vLLM)

bash scripts/evals/run_vseek.sh
# or see scripts/run_agent_data_vllm.py / scripts/evals/*

Merge FSDP checkpoints → Hugging Face format

python3 -m verl.model_merger merge \
  --backend fsdp \
  --local_dir checkpoints/vseek/<exp>/global_step_XXX/actor \
  --target_dir checkpoints/vseek/<exp>/global_step_XXX/actor/huggingface

Training (GRPO)

Example single-node GRPO run (edit paths / GPU count to match your machine):

bash scripts/train/vseek_grpo_tag_qvl34Bt_vllm_tagsummary.sh

Key knobs live in:

src/vseek/config/lvb_grpo.yaml — trainer / rollout config
src/vseek/config/retriever/toolconfig.yaml — tool server
src/vseek/config/retriever/vllm_agent.yaml — agent loop

Multi-node Slurm jobs: README_SLURM.md.

Evaluation

Convenience wrappers under scripts/evals/:

bash scripts/evals/run_vseek.sh
bash scripts/evals/run_uniform_vllm.sh
bash scripts/evals/run_gpt_eval.sh
bash scripts/evals/vllm_vseek_tests_ckpt.sh \
  --devices 0 \
  --batch-size 16 \
  --output-base results/vseek/my-run \
  --model-path /path/to/hf/checkpoint \
  --datasets lvb,lvbench,videomme,mlvu

Repository Layout

VSeek-R1/
├── src/vseek/           # Package: agent, tools, trainer, data, PULS
├── scripts/
│   ├── data_ops/        # Indexing + parquet preprocessing
│   ├── train/           # GRPO launch scripts
│   ├── evals/           # Evaluation wrappers
│   ├── utils/           # Download, unzip, burn subtitles, …
│   └── tacc/            # Slurm / Ray helpers (see README_SLURM.md)
├── vendor/verl/         # Vendored verl (RL training)
├── requirements_vllm.txt
├── requirements_sglang.txt
├── requirements_vllm_slurm2.txt
├── README.md            # This file (workstation setup)
└── README_SLURM.md      # TACC / Slurm installation

Citation

If you use this code, please cite:

@inproceedings{goel2026incentivizing,
  title={Incentivizing Vision Language Models to Search for Long Video Question Answering},
  author={Goel, Harsh and Sharan, SP and Shah, Sahil and Choi, Minkyu and An, Joungbin and Grauman, Kristen and Chinchali, Sandeep P},
  booktitle={European Conference on Computer Vision},
  pages={539--558},
  year={2026},
  organization={Springer}
}

Acknowledgments

Built on verl, vLLM, and the public video understanding benchmarks listed above.

Name		Name	Last commit message	Last commit date
artifacts		artifacts
notebooks		notebooks
scripts		scripts
src		src
vendor		vendor
.gitignore		.gitignore
.gitmodules		.gitmodules
README.md		README.md
README_SLURM.md		README_SLURM.md
err.txt		err.txt
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt
requirements_clean.txt		requirements_clean.txt
requirements_sglang.txt		requirements_sglang.txt
requirements_vllm.txt		requirements_vllm.txt
requirements_vllm_slurm2.txt		requirements_vllm_slurm2.txt
test_server_puls.py		test_server_puls.py
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation

VSeek-R1

Released weights, data, and index

Table of Contents

Path conventions (local vs TACC)

What actually differs between branches

Still local (/nas/mars) on both branches — override on TACC

TACC/SLURM only paths harsh_tacc

Overview

Requirements

Installation

1. Clone the repository

2. Create an environment

3. Install CUDA build deps + FlashAttention

4. Install VSeek + verl

5. Sanity check

Download Datasets

Process Datasets

LongVideoBench (lvb)

LVBench (lvbench)

Video-MME (videomme)

MLVU (mlvu)

CGBench (cgbench)

Run the Retriever Server

Inference

Uniform frame baseline

VSeek agent (vLLM)

Merge FSDP checkpoints → Hugging Face format

Training (GRPO)

Evaluation

Repository Layout

Citation

Acknowledgments

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages