NeurIPS 2026, Evaluations and Datasets Track

LIBERO-PeRM

Benchmarking personalized robotic manipulation: can a policy do the task the way you want it done?

Zhixu Li^1,*,†, Keqian Tang^1,*, Litian Gong^1,*, Jingyu Yao², Wenqian Zhang¹, Tianze Xu⁵, Zehao Wang³, Yuping Wang⁴, Junge Zhang³, Jiachen Li^3,‡

¹University of California, Riverside ²Yale University ³Georgia Institute of Technology ⁴University of Michigan ⁵University of Toronto

^*Equal contribution ^†Project Lead ^‡Corresponding Author

Paper Code Dataset Cite

Tasks: 312 in five suites
Preference levels: 3 motion, skill, user profile
Preferences: 37 single, 18 paired
Demonstrations: 40.8K with and without preference

Abstract

Personalization is essential for general-purpose robots to transition into household environments and achieve true practicality. While general robotic manipulation has been extensively studied, personalized robotic manipulation (PeRM) lacks a controlled benchmark for separating task completion from preference satisfaction. We introduce LIBERO-PeRM, a benchmark for preference learning in embodied agents. LIBERO-PeRM studies five questions: 1) how policies follow preferences in zero-shot settings; 2) how they learn single preferences from demonstrations; 3) how efficiently they acquire preferences as demonstrations increase; 4) how learned preferences generalize across layouts and tasks; and 5) how policies compose preferences and behave under conflicts. We propose a controlled formulation and taxonomy of observable preferences in simulation, covering motion, skill, and user-profile choices. This formulation is restricted to language-expressible preferences that can be instantiated in LIBERO without changing the task goal, evaluated automatically, and reused across tasks. Building on LIBERO, our procedural pipeline generates paired demonstrations with and without preferences. We create five task suites (312 tasks) with high-quality demonstrations, rich metadata, and preference-satisfaction metrics. Experiments show that current VLA policies struggle with zero-shot personalization, while SFT makes preferences learnable. Mixed training achieves better no-preference retention than sequential adaptation and improves performance with contextual prompts compared with preference-only training, but can weaken motion-preference learning; preference generalization, decomposition, and conflict handling remain difficult.

LIBERO-PeRM overview: five task suites on top, five research topics below — Top: the five procedurally generated suites, with pipeline-generated and human-reviewed demonstrations. Bottom: the questions they are built to answer, from zero-shot personalization to composition and conflict.

Three preference levels, 37 preferences

A preference is not a separate goal. It is a conditioning signal that biases how the task is carried out, and it is defined by the part of execution it constrains: the motion, the skill, or the subtask-level choice. Pick any preference to watch the same task with and without it, scored by that preference's own metric.

What the policy sees

Task and preference language are concatenated into one prompt, and the policy acts from two camera views. Every suite keeps this interface, so a model trained on one can be evaluated on the others.

prompt = “Task: {task_language}. Preference: {preference_language}.”

π_θ( a_t | I_t^agent, I_t^hand, ℓ_task ∥ ℓ_pref )

A procedural pipeline that treats preferences as controllable variables

Starting from a LIBERO layout, a task and a preference, the pipeline produces a preference-ready scene, paired demonstrations with and without the preference, subtask metadata, and the metric that scores the preference.

Data generation pipeline: preference selection and layout modification, preference-conditioned generation, satisfaction metric generation — Inputs on the left, the three stages in the middle, and what each stage adds to the released data along the bottom.

Preference selection and layout modification

A VLM makes the smallest edit that lets the preference be acted on. Motion and skill preferences leave the layout and task language untouched, and the VLM only picks the target, such as the object to avoid. A user profile adds what it needs, for example a red twin of an object, and the task language is abstracted so one sentence covers both behaviours.
Preference-conditioned generation

A VLM writes a no-preference plan and a preference-conditioned plan. A library of heuristic skills, each built from one human demonstration plus an OMPL planner over randomized initial states, turns the plans into trajectories. A user profile changes the plan, a skill preference changes which skill is called, and a motion preference changes the skill's parameters. More than 72% of skills need no offset tuning on any task.
Satisfaction metric generation

Each preference gets an explicit metric. Motion preferences are scored from trajectory quantities, skill preferences at key waypoints, and user profiles from the executed subtask plan. Scores map to a four-level label d ∈ {0, ¼, ¾, 1} so that all three levels report on the same scale.

Five suites, each isolating one capability

Together the suites hold 312 tasks with more than 100 trajectories per task on average and 40.8K rollout episodes. A task is one layout, one task instruction and one preference prompt. The strip below is drawn to scale. Select a suite to open its tasks.

LIBERO-ComprePref adds no new tasks. It rewrites the prompts of tasks from the other four suites: explicit task–preference mixtures, and implicit cues that imply the preference without naming it.

Dataset statistics: tasks by suite, tasks by preference type, preference instances by type, trajectory length, number of skills — (a) Tasks by suite. (b) Tasks by preference type, including the no-preference tasks generated for comparison. (c) Preference instances by type. (d) Trajectory length. (e) Number of skills per task.

What is released

DemonstrationsPer-task HDF5 files with third-person and wrist views, full simulator states, subtask metadata and per-step preference signals.
Task definitionsBDDL scenes and mission files: instructions, no-preference and preference plans, metric parameters and calibration.
Generation and evaluation codeThe procedural pipeline, the preference metrics, policy servers and the rollout client.
Reproduction scriptsOne script per table and figure of the paper, from fine-tuning to evaluation.

Every task, with its preference-satisfaction curve

Each clip replays the demonstrations for one task and re-evaluates the task's preference metric on the trace so far, so the end of a curve is the episode score. With two rollouts, solid lines follow the top one and dashed lines the bottom one. Conflict and composition tasks plot both preferences.

Filter tasks by preference or task

Task success and preference satisfaction, reported separately

A policy can finish the task while ignoring the preference. Two metrics keep those apart, and preference satisfaction only counts on episodes where the task succeeds.

SR, task success rate

SR = (1/N) Σ_n 𝟙[ G(τ_n) ]

Whether the task goal is reached, with or without a preference in the prompt.

SA, preference satisfaction

SA = (1/N) Σ_n 𝟙[ G(τ_n) ] · d_n

Each episode gets a discrete satisfaction label d_n ∈ {0, ¼, ¾, 1}. A failed episode contributes zero.

Three generalization settings

in-domain, task or layout, preference-level

In-domain keeps tasks, preferences and layouts with small object-position changes. Task-or-layout changes one of them. Preference-level changes the task under an unseen layout.

What each level's metric looks at

Motion	The trajectory. Physical quantities of the path: time per subtask, end-effector acceleration, height during lateral moves, straightness of the carry segment, clearance from a named object, rotation while moving.
Skill	The skill's key moment. The pose at the grasp or at the release: where the gripper sits relative to the object, its approach angle, how the object is set down or let go, whether it was lifted at all.
User profile	The executed subtask. Which candidate the policy commits to: the object it picks up first, or the target it places the object at, checked against what the profile implies.

Five questions, five findings

Six VLA policies are evaluated zero-shot: OpenVLA-OFT, π₀-fast, π_0.5, GR00T-N1.6, SmolVLA and UniVLA. Fine-tuning experiments start from π_0.5, the strongest zero-shot baseline.

Q1Zero-shot personalization

Policies finish the task and ignore the preference

Minimal layout edits keep no-preference success high, with π_0.5 at 97.6%, so low satisfaction reflects preference following rather than a harder scene. On motion and skill preferences π_0.5 succeeds 97.5% and 100% of the time but satisfies the preference only 24.4% and 25.0%. A user profile changes the target itself and drops π_0.5 to 19.1% success.

Zero-shot results on LIBERO-SinglePref. SR and SA in %. No-Pref reports SR only.

	No-Pref	All-Pref		Motion		Skill		User profile
Method	SR	SR	SA	SR	SA	SR	SA	SR	SA
OpenVLA-OFT	91.2	40.00	4.53	70.00	7.30	64.10	3.20	3.92	3.92
π₀-fast	86.88	51.25	30.37	80.00	27.60	73.08	18.27	17.65	17.65
π_0.5	97.6	66.67	37.92	97.50	24.38	100.00	25.00	19.12	19.12
GR00T-N1.6	88.3	51.67	27.45	90.00	20.70	76.92	11.54	9.80	9.80
SmolVLA	60.0	4.17	2.03	16.67	4.17	0.00	0.00	0.00	0.00
UniVLA	80.8	45.83	21.52	90.00	10.17	71.79	3.59	0.00	0.00

Q2Learning ability

Fine-tuning makes preferences learnable, and mixed training is a trade-off

LoRA fine-tuning on LIBERO-SinglePref lifts all-preference SR/SA to 85.0 / 82.5. Fine-tuning on LIBERO first and preferences second costs no-preference success, which falls to 52.3%. Mixing both in one run keeps it at 91.0% and holds up under rewritten prompts: 72.8 / 70.1 on LIBERO-ComprePref against 44.0 / 42.6 for preference-only training. The price is paid on motion preferences, where success drops from 90.0% to 66.9%, so mixing is not uniformly better.

Fine-tuned results, all from the π_0.5 base checkpoint. Mean ± standard deviation over 10 evaluation runs with fixed checkpoints. * is the same task and preference under an unseen layout.

	No-Pref	All-Pref		Motion		Skill		User profile		ComprePref
Method	SR	SR	SA	SR	SA	SR	SA	SR	SA	SR	SA
π_0.5-LoRA	–	84.96±1.06	82.48±1.02	90.00±2.85	85.37±2.66	78.86±1.44	76.67±1.40	86.67±1.49	85.22±1.47	44.00±0.71	42.64±0.69
π_0.5-Mix-LoRA	91.00±0.70	84.55±1.94	84.45±2.04	66.88±3.56	66.06±4.38	89.29±2.82	89.29±2.82	91.33±3.42	91.33±3.42	72.83±0.95	70.07±0.91
π_0.5-Seq-LoRA	52.33±0.37	74.75±1.97	74.73±2.12	67.50±5.68	67.44±6.52	72.14±2.04	72.14±2.04	81.00±2.79	81.00±2.79	70.00±1.32	68.25±1.29
π_0.5-LoRA*	–	37.49±3.27	28.36±2.55	55.00±7.53	43.43±7.02	28.57±3.79	26.94±3.57	34.00±5.60	20.58±3.39	–	–

Q3, Q4Learning efficiency and generalization

Layout diversity helps more than extra demonstrations, and generalization stays hard

In-domain SR and SA track each other at every data scale: when the task succeeds, the preference is usually satisfied too. At a fixed budget of 2.25K demonstrations, spreading them over more layouts (L1 to L3) helps more than adding demonstrations per layout, and adding matched no-preference missions (L4) helps again without any new layout. Unseen layouts improve with scale. Unseen layouts with a different task do not.

Task success and preference satisfaction against training data scale, for in-domain, unseen-layout, and unseen-layout-with-different-task evaluation — Scaling on LIBERO-PrefScale. In-domain (blue), unseen layout (red), unseen layout with a different task (green). Dashed branches compare L1 (★, fewer layouts, denser), L3 (△, more layouts) and L4 (◇, matched no-preference missions) at 2.25K.

Q5Composition and conflict

Composition is partly learnable, decomposition is not

Trained on P_A and P_B separately, the policy composes P_A + P_B with 25.0% SR and SA. Asked to recover P_B after training on P_A + P_B and P_A, it reaches 37.5% SR and 0.0% SA: the task gets done, the target preference does not.

Conflicts degrade execution instead of being resolved

Under two conflicting preferences SR/SA fall from 73.0 / 63.4 to 66.0 / 47.8, and neither preference dominates (gap 11.3). When the preference argues against the task, following the task (33.3%) and following the preference (29.2%) are about as likely, and 37.5% of trials complete neither.

Cite this work

@inproceedings{li2026liberoperm,
  title     = {LIBERO-PeRM: Benchmarking Personalized Robotic Manipulation},
  author    = {Li, Zhixu and Tang, Keqian and Gong, Litian and Yao, Jingyu and Zhang, Wenqian and
               Xu, Tianze and Wang, Zehao and Wang, Yuping and Zhang, Junge and Li, Jiachen},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}