Design Book › Sim-to-real metrics

Sim-to-real metrics

The database holds no general pass threshold for any gap metric. Write your own threshold before the test, and record it. The numbers below that look like thresholds belong to one project or tool. See "Gates that are not general".

What a designer must do

  1. Set simulation kp, kd and control frequency to the values used in real data collection. A mismatch inflates the measured gap [1].
  2. Replay one motion on the robot and in simulation, then compare the logs [2].
  3. Report the gap per joint in position, velocity and torque, not as one robot-wide number [3].
  4. Compute RMSE, MAPE, correlation and cosine similarity, plus lag from cross-correlation [4], [5].
  5. Mask small real values before you compute MAPE. SAGE excludes real samples below 0.1 in absolute value [6].
  6. Estimate serial command transport delay before you fit a model [7].
  7. Judge a fit on independent validation chunks. Without them, comparing authored and accepted values is training-data replay, not a holdout [8].
  8. Report simulation and physical results separately, each with its trial count [9], [10].

Scope: the sim-to-real guidelines, the measured SO-101 and other published sim-to-real data, and the Isaac Sim guidelines on system identification. The transfer gates are in Sim-to-sim and sim-to-real. Guidelines marked (note) come from agent-written notes; see chapter 25.

Which metrics exist

No guideline gives a formula for RMSE, MAPE, correlation or cosine similarity. The guidelines name them. The code is in SAGE, linked from each guideline.

Which metrics exist
MetricMeasuresHowSource
RMSE, MAPE, correlation, cosine similarityJoint gap, per jointBetween aligned simulated and real position, velocity and torque[4], [11]
LagDelay, kept apart from amplitude errorCross-correlation peak[5]
Command transport delaySerial delayLogged delay between command and motion[7]
Command-to-response delayUnmodeled delayPer joint[12]
Loop timingReal loop against simulationTimestamp commands, compute jitter[13]
Steady-state offsetStictionReal step response. The UR10e showed about 0.25 degrees[14]
Position MSE of a fitQuality of a system identificationCompare your own fit, with your arm's values[15]
Tuned against untuned RMSEWhether a fit helpedAuthored and accepted values on identical chunks. Without independent chunks it is training-data replay[8]
Sensitivity spanParameter offset for a 10 percent cost riseReported by the Isaac Sim tool. Not a confidence interval[16]
End-effector heightCalibrationAt a known pose, against the simulation[17]
Key eventsContact matchReplay measured motion, compare real and simulated key events[18]
Link lengthsModeling gapKey lengths on the built robot against CAD[19]
MassModeling gapWeigh each removable link and compare with the model. Model masses and inertias come from CAD, not from a scale[20]
Task successWhole systemA stated trial count, with the simulation figure beside it[10]

Classify each gap as sensing, actuation, physics or modeling before you pick a fix [21]. Check input, system response and output consistency separately [22].

What SAGE defines

SAGE is a sim-to-real actuator gap estimator. The rows below come from commit 5b31cfa, and the repo has no tags. The catalog's record of this source says it defines gap metrics but states no pass or fail thresholds.

What SAGE defines
ItemSAGE doesSource
OutputPer-joint RMSE, MAPE, correlation, cosine similarity and plots, from scripts/run_analysis.py[4]
MAPE maskExcludes real samples with absolute value below 0.1[6]
LagCross-correlation for lag analysis[5]
SetupSimulation kp, kd and control frequency must match real collection[1]
Example run200 Hz physics, 50 Hz control, kp 100, kd 2, on the h1_2 humanoid. An example only[23]
LeRobot collectorRuns at 50 Hz. Record the rate with the data[24]
UnitsThe state log lists joint data in radians. Confirm on a known joint move[25]

Check versions first. SAGE's README badges name Isaac Sim 5.0.0 and Isaac Lab 2.2.0 [26]. Isaac Lab 3.0.0 does not support Isaac Sim 5.1 or older [27]. No guideline says SAGE runs on Isaac Sim 6.2.0. Test it on your version before you rely on it.

System identification in Isaac Sim

The extension is marked beta. Its run contracts, exported specifications and APIs may change [28]. So pin the version with any saved result. These rows come from the extension docs in the repo, not from a note.

System identification in Isaac Sim
StepGuideline saysSource
DataCSV, ROS 2 bag, MCAP or local LeRobot data. LeRobot data needs a recognized joint position signal such as observation.state[29]
QualityPreflight checks that timestamps strictly increase, values are finite, channel shapes are right, chunks cover the log, the motion excites the joints and commands line up with the response[30]
TorqueDeclare link_side or external. Only link-side torque seeds inertia and friction. Undeclared channels are excluded[31]
ValidationCompares authored and accepted values on identical chunks. Without independent validation chunks it is training-data replay, not a holdout[8]
PresolveFixed-base robots with base-to-tip joints, unambiguous USD bodies, a full-rank well-conditioned regressor and bounded unconstrained solutions. A failed gate keeps the original seed[32]
OptimizersLevenberg-Marquardt, CMA-ES, Bayesian, Adam gradient descent[33]

MuJoCo has its own path: mujoco.sysid, with real bounds on each parameter [34], [35].

Measured SO-101 data

Published fits

Two projects fitted the SO-101 and published the result. Run system identification on your arm and compare, as the checks on the Playground fit guideline [36] and the gtc_berlin fit guideline [37] ask. Do not mix values from the two fits in one model. No guideline states units for these values.

Playground fit, from real teleop data, MuJoCo Playground v1.0.0:

Published fits
JointArmature [36]Stiffness [38]Damping [39]Friction loss [40]
Shoulder pan0.1088251.589617.50960.0001
Shoulder lift0.1472265.095828.06200.0
Elbow flex0.1224243.984425.66690.0
Wrist flex0.1133223.280425.45430.0
Wrist roll0.1146218.406623.10580.0
Gripper0.1985-34.16330.2517

The model sets FORCERANGE = 30 * 9.81 / 100 from a stated STS3215 stall torque of 30 kg-cm at 12 V [41]. So check it against your voltage.

gtc_berlin fit, NVIDIA so101-typing-task commit 7aec2ed. The database holds two joints from this fit:

Published fits
JointArmatureStiffnessDampingDry frictionSource
Shoulder pan0.05822834261829721448.936820874351542.309705389862818-[37]
Elbow-18.5654057807300250.5346943536769580.4018746873200306[42], [43]

The fit reports best_mse 0.0001072153189590853 on position MSE [15].

The two disagree. Computed from the rows above, stiffness differs by a factor of 5.1 at shoulder pan and 13.1 at the elbow [38], [37], [42]. The Playground fit has elbow friction loss of 0.0 [40], while gtc_berlin reports dry friction of 0.4018746873200306 [43]. No guideline says whether the two fits use the same model form or the same friction term. So treat neither as the SO-101's value.

How Playground fits: 24 parameters, namely armature, friction loss, stiffness and damping on each of 6 joints, from teleoperation logs by rollout tracking error [44]. Bounds: armature 1e-4 to 0.5, stiffness 1.0 to 2500.0 [45], [46]. Excitation must move all 6 joints at varying speeds with reversals, and open and close the gripper several times [47].

Other gains in the database are set values, not fits the guidelines name. See the asset tables in RL configs that ship.

Calibration offsets

From one development setup, the September fit. These belong to that robot, so fit your own.

Calibration offsets
JointOffsetSource
Shoulder liftAbout +1.336 deg[48]
Elbow flexAbout -10.150 deg[48]
Wrist flexAbout +1.414 deg[48]
Shoulder pan, wrist rollZero[48]

Results

Read each row with its condition. None is a target.

Results
ResultValueConditionRead it asSource
Real success, Squint91.3 percentAveraged over 80 trials for 8 tasks. ManiSkill3One paper's tasks[10]
Simulation success, Squint96.1 percentOver all tasks, at the end of its training runQuote only with this condition[51]
Reset-per-key checks93 of 93Fitting recordingsFitting data, not held-out validation[52]
Six-letter simulation episodes2,048 of 2,048September run, after fittingSimulation only[9]
Physical typing trialNEWTON in 7.10 sOne supervised trial on a second SO-101One trial. Not a success rate, not an independent reproduction[53], [54]
Pick-cube policyTested on 3 robot setupsVarious lighting. A visual gap may remain with darker shadowsNo success figure in the database[55]
Training and evaluation wall time2 h 38 min 21 sRTX 5090One planning point[56]
Training rate36.72 s per 10 updates4096 environmentsMeasure your own[57]

Servo bench data

For the STS3215-C018 at 12 V, the database holds facts of kind measured from two bench reports: backlash, repeatability, no-load speed, stall torque, temperature and temperature rise. This site does not show the values: one report does not allow copying, and the terms of the other are not confirmed. One result is public through a guideline: a bench test found about 1 deg of backlash against a datasheet value of at most 0.5 deg [58]. The project README gives 30 kg.cm stall torque. Measure your own servo before you use any of these in a model.

Gates that are not general

These look like thresholds. Each belongs to one project or one tool. Do not adopt one as a pass limit for your gap.

Gates that are not general
NumberWhat it gatesWhoseWhy it is not generalSource
At least 15 mmTwo consecutive measured FK samples above the key after key-upTyping project's alignment checkA clearance for one keyboard scene[59]
90 percentStrict success, for promotion between training stagesTyping project defaultSimulation only. The source calls it an engineering gate, not a claim of hardware readiness[60]
Every recorded approachReach the intended first key in simulation, with a reset between keys and no simultaneous extra key actuation, before training for physical useTyping projectIts own task. It reports 93 of 93 on fitting recordings[61]
0.1Mask on real values before MAPESAGEA data filter, not a pass limit[6]
10 percentCost rise behind the sensitivity spanIsaac Sim system identificationA report definition, not a pass limit[16]
Tuned below untunedRMSE on validation chunksIsaac Sim system identificationA relative test. It holds no number[8]

Where the database is thin

Sources

Each guideline in this chapter with its source, the date it was read and the exact words of the source. Level: must means breaking it causes failure or harm, should is good practice, guide is a rule of thumb.

  1. Set simulation kp, kd and control frequency to the values used in real data collection; a mismatch inflates the measured gap. Level: must. raw.githubusercontent.com 2026-11-09“Mismatched parameters will inflate the measured gap.”
  2. To measure the actuation gap, replay the same motions on hardware and in simulation and compare the two logs. Level: should. docs.nvidia.com 2026-11-08
  3. Report the gap per joint in position, velocity and torque, not as one robot-wide number. Level: should. docs.nvidia.com 2026-11-08
  4. Measure the sim-to-real joint gap per joint with RMSE, MAPE, correlation and cosine similarity between aligned simulated and real signals. Level: should. raw.githubusercontent.com 2026-11-09“use `scripts/run_analysis.py` to generate per-joint metrics (RMSE, MAPE, correlation, cosine similarity) and visualization plots.”
  5. Include lag from cross-correlation among the gap metrics to separate delay from amplitude error. Level: should. raw.githubusercontent.com 2026-11-09“# 5. Cross-correlation for lag analysis”
  6. When computing MAPE, SAGE excludes real samples with absolute value below 0.1. 0.1 Level: guide. raw.githubusercontent.com 2026-11-09“valid_indices = np.abs(aligned_real["value"]) >= 0.1”
  7. Estimate the serial command transport delay before fitting, and align command times to the measured rollout. Level: should. raw.githubusercontent.com 2026-11-09“Estimate and log the serial command transport delay ($t_1 - t_0$).”
  8. Validation compares authored and accepted values on identical chunks; without independent validation chunks it is training-data replay, not a holdout. Level: should. raw.githubusercontent.com 2026-11-09
  9. The September run passed 2,048 of 2,048 six-letter simulation episodes and completed one physical NEWTON trial on a second SO-101 after fitting and training. Level: guide. raw.githubusercontent.com 2026-11-09“passed 2,048/2,048 six-letter simulation episodes and completed one physical NEWTON trial on a second SO-101 after fitting and training for that setup.”
  10. Report real success with its trial count, as Squint does with 91.3 percent over 80 trials on 8 tasks, and show the simulation figure beside it. Level: should. arxiv.org 2026-11-08
  11. Use RMSE, MAPE, correlation and cosine similarity per joint when comparing with SAGE output. Level: guide. raw.githubusercontent.com 2026-11-08
  12. Suspect unmodeled delay when a policy shakes violently on hardware. Level: should. arxiv.org 2026-11-08
  13. Measure control loop timing on hardware, because it differs from simulation. Level: should. docs.nvidia.com 2026-11-08
  14. Plot a real step response to quantify stiction, then add joint friction in simulation; the UR10e showed about 0.25 degrees of offset. 0.25 degrees Level: should. isaac-sim.github.io 2026-11-08“observed contact offsets of approximately 0.25 degrees from the commanded setpoint.”
  15. The gtc_berlin SO-101 actuator fit reports position-MSE best_mse = 0.0001072153189590853. Level: guide. raw.githubusercontent.com 2026-11-09“metric: position_mse best_mse: 0.0001072153189590853”
  16. The reported sensitivity span is the parameter offset for a 10 percent cost rise; it is not a confidence interval. Do not quote it as one. Level: must. raw.githubusercontent.com 2026-11-09
  17. Manual SO-101 middle-pose calibration can be off by 1 to 2 cm or more in end-effector height; recalibrate if the arm reaches too low or hovers too high. Level: should. raw.githubusercontent.com 2026-11-09“Manual middle-pose calibration on the physical SO101 arm can easily be off by 1–2+ cm in end-effector height.”Review note: Source only says calibration can easily be off by 1-2+ cm; it does not require recalibration, so severity must is too strong.
  18. Matching external dimensions does not validate joint-angle conventions, jaw pose or contact geometry: in this project, replaying measured real joint motion still gave different contacts in simulation. Level: must. raw.githubusercontent.com 2026-11-09“replaying measured real joint trajectories still produced different contacts in simulation.”
  19. Compare CAD against the as-built robot as part of gap measurement. Level: should. docs.nvidia.com 2026-11-08
  20. This guideline is held back: its source does not allow us to publish it yet.
  21. Classify each observed gap as sensing, actuation, physics or modeling before choosing a fix. Level: should. docs.nvidia.com 2026-11-08
  22. Check three things for sim-to-real: input consistency, system response consistency and output consistency. Level: must. raw.githubusercontent.com 2026-11-09“Successful sim-to-real transfer requires addressing three fundamental aspects:”
  23. The SAGE example run uses 200 Hz physics, 50 Hz control, kp 100 and kd 2 on the h1_2 humanoid; treat it as an example only. Level: guide. raw.githubusercontent.com 2026-11-09“--physics-freq 200 \ --render-freq 200 \ --control-freq 50 \ --kp 100 \ --kd 2 \”
  24. The SAGE LeRobot real data collector runs at 50 Hz. 50 Hz Level: guide. raw.githubusercontent.com 2026-11-09“**Frequency**: 50Hz”Review note: Source gives 50Hz as the frequency of the input TXT motion files, not as the rate of the real-robot collector.
  25. Check units in SAGE data files, because the state log format lists joint data in radians. Level: should. raw.githubusercontent.com 2026-11-08
  26. Check simulator versions before using SAGE, because its README badges and requirements name Isaac Sim 5.0.0, not 6.2.0. Level: must. raw.githubusercontent.com 2026-11-08
  27. Isaac Sim 5.1 and older are not supported by this Isaac Lab release. Level: must. raw.githubusercontent.com 2026-11-09“Isaac Sim 5.1 and older are unsupported.”
  28. Isaac Sim system identification is beta; its run contracts, exported specifications and APIs may change. Level: should. raw.githubusercontent.com 2026-11-09
  29. System identification loads CSV, ROS 2 bag, MCAP or local LeRobot data; LeRobot data needs a recognized joint position signal such as observation.state. Level: must. raw.githubusercontent.com 2026-11-09
  30. Preflight and quality reports check channel shapes, finite values, strictly increasing timestamps, chunk coverage, excitation and command alignment; give them data that meets all of these. Level: should. raw.githubusercontent.com 2026-11-09
  31. Declare torque telemetry as link_side or external; only link-side torque seeds inertia and friction, and undeclared channels are excluded. Level: must. raw.githubusercontent.com 2026-11-09
  32. For fixed-base robots, presolve requires base-to-tip joints, unambiguous USD bodies, a full-rank well-conditioned regressor and bounded unconstrained solutions; failed gates keep the original seed. Level: should. raw.githubusercontent.com 2026-11-09
  33. Available optimizer backends are Levenberg-Marquardt, CMA-ES, Bayesian and Adam gradient descent. Level: guide. raw.githubusercontent.com 2026-11-09
  34. Fit armature, damping, friction, gains, inertia and delay to hardware logs with optimization, rather than hand-tuning, when matching a specific robot in MuJoCo. Level: should. mujoco.readthedocs.io 2026-11-08
  35. Expect mujoco.sysid to use box-constrained least squares, so give each parameter real bounds. Level: must. mujoco.readthedocs.io 2026-11-08
  36. The Playground v1.0.0 SO-101 armature values identified from real teleop data are 0.1088 shoulder pan, 0.1472 shoulder lift, 0.1224 elbow, 0.1133 wrist flex, 0.1146 wrist roll and 0.1985 gripper. Level: guide. raw.githubusercontent.com 2026-11-09“# real teleop data. ARMATURE = { "shoulder_pan": 0.1088, "shoulder_lift": 0.1472, "elbow_flex": 0.1224, "wrist_flex": 0.1133, "wrist_roll": 0.1146, "gripper": 0.1985,”
  37. The gtc_berlin fit gives the SO-101 shoulder pan stiffness 48.93682087435154, damping 2.309705389862818 and armature 0.058228342618297214. Level: guide. raw.githubusercontent.com 2026-11-09“stiffness: 48.93682087435154 damping: 2.309705389862818 armature: 0.058228342618297214”
  38. The Playground v1.0.0 SO-101 position gains, identified from real teleop data, set arm stiffness from 218.4066 at wrist roll to 265.0958 at shoulder lift. Level: guide. raw.githubusercontent.com 2026-11-09“# Position servo PD gains identified from real teleop data. STIFFNESS = { "shoulder_pan": 251.5896, "shoulder_lift": 265.0958, "elbow_flex": 243.9844, "wrist_flex": 223.2804, "wrist_roll": 218.4066,”
  39. The Playground v1.0.0 SO-101 identified damping runs from 17.5096 at shoulder pan to 34.1633 at the gripper. Level: guide. raw.githubusercontent.com 2026-11-09“"shoulder_pan": 17.5096, "shoulder_lift": 28.0620, "elbow_flex": 25.6669, "wrist_flex": 25.4543, "wrist_roll": 23.1058, "gripper": 34.1633,”
  40. The Playground v1.0.0 SO-101 friction loss, identified from real teleop data, is 0.2517 at the gripper and near zero on arm joints. 0.2517 Level: guide. raw.githubusercontent.com 2026-11-09“# Per-joint frictionloss identified from real teleop data. FRICTIONLOSS = { "shoulder_pan": 0.0001, "shoulder_lift": 0.0, "elbow_flex": 0.0, "wrist_flex": 0.0, "wrist_roll": 0.0, "gripper": 0.2517,”
  41. The Playground v1.0.0 SO-101 model sets FORCERANGE = 30 * 9.81 / 100 from a stated STS3215 stall torque of 30 kg-cm at 12 V. Level: guide. raw.githubusercontent.com 2026-11-09“FORCERANGE = 30 * 9.81 / 100 # STS3215 stall torque: 30 kg-cm @ 12V.”
  42. The gtc_berlin fit gives the SO-101 elbow a stiffness of 18.565405780730025 and a damping of 0.534694353676958. Level: guide. raw.githubusercontent.com 2026-11-09“stiffness: 18.565405780730025 damping: 0.534694353676958”
  43. The gtc_berlin fit reports elbow dry friction of 0.4018746873200306. Level: guide. raw.githubusercontent.com 2026-11-09“dry_friction: 0.4018746873200306”
  44. Fit 24 parameters for the SO-101: armature, friction loss, stiffness and damping on each of the 6 joints, from teleoperation logs by rollout tracking error. 24 parameters Level: should. raw.githubusercontent.com 2026-11-09“The optimizer fits a 24-parameter model (`armature`, `frictionloss`, `stiffness`, `damping` for all 6 joints) from teleoperation MCAP logs by minimizing rollout tracking error in MuJoCo physics.”
  45. The Playground SO-101 fit bounds armature to 1e-4 through 0.5. Level: guide. raw.githubusercontent.com 2026-11-09“min_value=1e-4, max_value=0.5,”
  46. The Playground SO-101 fit bounds stiffness to 1.0 through 2500.0. Level: guide. raw.githubusercontent.com 2026-11-09“min_value=1.0, max_value=2500.0,”
  47. Excitation data must move all 6 joints at varying speeds with direction reversals and open and close the gripper several times. Level: must. raw.githubusercontent.com 2026-11-09“Sweep all 6 joints through varying speeds and directional reversals.”
  48. The September fit on one SO-101 selected about +1.336 deg shoulder lift, -10.150 deg elbow flex and +1.414 deg wrist flex; these belong to that robot, so fit your own. Level: guide. raw.githubusercontent.com 2026-11-09“The September fit selected about +1.336° shoulder lift, -10.150° elbow flex and +1.414° wrist flex, with zero pan and wrist-roll offsets.”
  49. Both the July and September fits selected about a -10 degree elbow correction, with smaller positive shoulder-lift and wrist-flex corrections. Level: guide. raw.githubusercontent.com 2026-11-09“Both the July and September fits selected about a -10° elbow correction, with smaller positive shoulder-lift and wrist-flex corrections.”
  50. Apply fitted offsets as q_model = q_LeRobot + offset on reads, and the inverse subtraction on hardware goals. Level: must. raw.githubusercontent.com 2026-11-09“Offsets use `q_model = q_LeRobot + offset`; hardware goals use the inverse subtraction.”
  51. Cite a simulation success figure only with its training-time condition: Squint's 96.1 percent is measured at the end of its training run. Level: guide. arxiv.org 2026-11-08
  52. The September fit passed 93 of 93 reset-per-key checks on its fitting recordings; this is fitting data, not held-out validation. Level: guide. raw.githubusercontent.com 2026-11-09“The September fit passed **93/93** such checks on its fitting recordings.”
  53. One supervised trial on a second SO-101 typed NEWTON in 7.10 s; this is one trial, not a success rate. Level: guide. raw.githubusercontent.com 2026-11-09“jointly fitted scene completed **NEWTON** in 7.10 seconds with matching releases and model-FK clearance,”
  54. The recorded physical result states that one NEWTON trial is not a success rate benchmark or an independent reproduction. Level: guide. raw.githubusercontent.com 2026-11-09“One operator-supervised NEWTON policy trial on the development setup; not a success-rate benchmark or independent-user reproduction.”
  55. The SO-101 pick-cube policy was tested on 3 robot setups; a visual gap can remain with darker shadows, so light the scene to avoid them. Level: should. raw.githubusercontent.com 2026-11-09“We tested on 3 different robot setups under various lighting conditions, but there may still be visual domain gap with darker shadows.”
  56. The measured training and evaluation wall time was 2 h 38 min 21 s on an RTX 5090; use it as one planning point. Level: guide. raw.githubusercontent.com 2026-11-09“Its measured training/evaluation wall time was **2 h 38 min 21 s** on an RTX 5090.”
  57. The measured rate was 36.72 s per 10 updates with 4096 environments. Level: guide. raw.githubusercontent.com 2026-11-09“At the measured 36.72 seconds per 10 updates with 4096 environments”
  58. A bench test of the STS3215 measured about 1° of encoder backlash against a ≤0.5° datasheet value. Level: guide. github.com 2026-11-08
  59. The alignment check for this project requires two consecutive measured FK samples at least 15 mm above the key after key-up. 15 mm Level: must. raw.githubusercontent.com 2026-11-09“After key-up it requires two consecutive measured FK samples at least 15 mm”
  60. The project default promotion threshold between training stages is 90 percent strict success in simulation; it is an engineering gate, not a claim of hardware readiness. 90 % strict success Level: guide. raw.githubusercontent.com 2026-11-09“default promotion threshold is 90% strict success, an engineering gate rather than a claim of hardware readiness.”
  61. Before training for physical use, require every recorded key approach to reach the intended first key in simulation, with a reset between keys and no simultaneous extra key actuation. Level: must. raw.githubusercontent.com 2026-11-09“require every recorded key approach to reach the intended first key in simulation, with a reset between keys and no simultaneous extra key actuation.”
  62. The Playground SO-101 task randomizes control delay in (0.000, 0.004) s by stochastic substeps. Level: guide. raw.githubusercontent.com 2026-11-09“delay=(0.000, 0.004),”