Distributed Neural Training

Widget ID: distributed-training. API: documented below.

Purpose

Train a neural network across the browsers of the participants. One participant, the coordinator, provides the model as a WebAssembly module and the dataset as files. Other participants accept an invitation and train on parts of the dataset; the coordinator combines their results into a new checkpoint after every round, by federated averaging (FedAvg) or, if you choose, FedAvgM or FedAdam.

A model can also generate its own data instead of using a dataset, for reinforcement learning.

How a training job runs

1. The coordinator selects the model module, a dataset manifest and the dataset files. Weei checks sizes and SHA-256 hashes and creates checkpoint 0.
2. The coordinator invites participants. Each sees the model, its hash, the data sizes and the settings before accepting; nothing is downloaded before that.
3. Each round, every active participant receives at most one dataset chunk, trains on it in a Worker and sends back updated weights.
4. The coordinator validates the updates, averages them weighted by the number of unique examples, applies the chosen aggregation method and saves the next checkpoint.
5. The job ends after the planned dataset epochs. Checkpoints can be downloaded as .weei-checkpoint files and used to continue training later.

Jobs and checkpoints are stored in the coordinator's browser (IndexedDB) and survive a page reload.

The widget's Execution diagnostics card shows, for every participant, the browser and OS, the mode it actually trains in and why it fell back, its Workers and GPU, its current assignment and stage, how long each stage took, its compute speed and how fresh these figures are. It also says whether all participants use the same settings and model. The same data is available through the API as state.diagnostics.

Model module

The model is a WebAssembly module following Weei WASM ABI 1. It exports memory, alloc, free, resultLength and these operations, each taking and returning UTF-8 JSON:

Any operation may return {error: "reason"}. The only allowed import is weei.onProgress(pointer, bytes) for progress reports. Only use modules you trust: the coordinator runs the module while preparing the job.

{
  "id": "my-model",
  "version": "1.0.0",
  "name": "My model",
  "abiVersion": 1,
  "weightFormat": "float32-le",
  "maxWorkers": 1,
  "maxChunkBytes": 8388608,
  "maxExamplesPerChunk": 10000,
  "dataFormats": [{ "id": "my-binary", "label": "My binary format", "acceptedMimeTypes": ["application/octet-stream"] }],
  "metrics": [{ "id": "loss", "label": "Loss", "direction": "minimize" }],
  "aggregation": { "methods": ["fedavgm", "fedadam"], "averageOnly": [{ "offset": 1048576, "bytes": 4096 }] }
}

aggregation is optional. It tells the coordinator how the model's weights may be merged; see What the model declares. Without it, every method is allowed and every weight is treated as trainable.

Dataset manifest

A JSON array describing the dataset files. Each file must be readable by the module on its own.

[
  {
    "id": "chunk-001",
    "fileName": "training-001.bin",
    "bytes": 4096,
    "sha256": "0000000000000000000000000000000000000000000000000000000000000000",
    "format": "my-binary",
    "exampleCount": 128
  }
]

Reinforcement learning

A model can generate its own training data instead of reading a dataset. For classic reinforcement learning the environment runs inside the model's WebAssembly: each participant plays episodes, updates the policy on them and sends back only the weights. Observations, actions and rewards never leave the participant, and nothing is sent to a server. Such a job is called an experience job.

1. The coordinator selects the model module and its package manifest; there are no dataset files. The widget then asks for Rounds and, for example, Episodes per round instead of dataset epochs and files.
2. Each round, every active participant gets one assignment: a range of units (episodes, environment steps or another unit the model chooses) and a seed.
3. The model plays those units, trains on them and returns its weights. The coordinator averages the weights, weighted by the units each participant completed, as for a dataset job; FedAvgM and FedAdam work the same way.
4. With evaluation enabled, the coordinator runs the model on fixed evaluation episodes after the round, so results are comparable across rounds.

What the model declares

The model describes the job in two optional fields of getModelInfo(). A model without workSource is a dataset model, exactly as before.

{
  "id": "cartpole-reinforce",
  "version": "1.0.0",
  "name": "CartPole policy (REINFORCE)",
  "abiVersion": 1,
  "weightFormat": "float32-le",
  "maxWorkers": 1,
  "dataFormats": [],
  "metrics": [{ "id": "episodeReturn", "label": "Episode return", "direction": "maximize" }],
  "capabilities": { "evaluation": true },
  "workSource": {
    "kind": "experience",
    "inputs": "none",
    "unit": { "id": "episode", "label": "Episodes" },
    "unitsPerAssignment": { "default": 32, "minimum": 1, "maximum": 4096 },
    "evaluationUnits": 32
  },
  "auxiliaryWeights": []
}

Package manifest

An experience model comes as a package: model.wasm and its manifest.json, in the same weei-model-package format generators write for dataset models, with workSource and auxiliaryWeights added. A plain list of chunks ([]) is not accepted for an experience model.

{
  "schema": "weei-model-package",
  "schemaVersion": 1,
  "hostRuntime": { "id": "weei-device-program-v1", "delivery": "installed-in-host" },
  "wasm": { "fileName": "model.wasm", "bytes": 123351, "sha256": "…" },
  "workSource": { "kind": "experience", "inputs": "none", "unit": { "id": "episode", "label": "Episodes" },
                  "unitsPerAssignment": { "default": 32, "minimum": 1, "maximum": 4096 }, "evaluationUnits": 32 },
  "auxiliaryWeights": [],
  "execution": {
    "schema": "weei-model-execution", "schemaVersion": 1,
    "modelId": "cartpole-reinforce", "weightFormat": "float32-le", "weightBytes": 456,
    "compatibility": { "fileName": "compatibility.json", "bytes": 640, "sha256": "…" },
    "datasetManifest": { "fileName": "dataset-manifest.json", "bytes": 3, "sha256": "…" },
    "backends": [{ "id": "cpu-wasm-simd", "kind": "cpu", "runtime": "weei-wasm-abi1",
                   "environments": ["browser", "agent"], "operations": ["train", "evaluate"],
                   "artifacts": [{ "fileName": "model.wasm", "bytes": 123351, "sha256": "…" }],
                   "requiredFeatures": ["wasm-simd128"], "minimumDeviceLimits": {},
                   "estimatedWorkingBytes": 4194304, "numericalMode": "fp32-standard" }],
    "policy": { "preference": ["cpu-wasm-simd"], "cpuFallback": "restart-assignment", "aggregationOwner": "peer",
                "assignmentExecution": "exclusive", "progressUnit": "episode", "optimizerUnit": "update" }
  },
  "inlineJson": {
    "compatibility.json": { "schemaVersion": 1, "identity": { "modelId": "cartpole-reinforce" }, "weightCount": 114, "weightBytes": 456 },
    "dataset-manifest.json": []
  }
}

Before anything is shared, Weei checks:

• the schema and host runtime; workSource and auxiliaryWeights are valid; a GPU backend comes with a shaderArchive and only for inputs: "none"; a selected WGSL archive is checked module by module;
• the selected module is the package's wasm, by name, size and SHA-256, and the CPU backend's artifact;
• the inline files match their descriptors: size and SHA-256 of JSON.stringify(value, null, 2) + " "; compatibility.json names the same model and weight size;
• for inputs: "none": the dataset manifest is empty and no validation file is selected; a selected WGSL file must be the package's own archive (shaderArchive), which enables GPU training — without it the package trains on the CPU;
• after loading the module: the model ID and initial weight size equal execution.modelId and weightBytes, and workSource and auxiliaryWeights (without fileName) equal getModelInfo().

A failed check stops job creation with an error naming the problem. Auxiliary weights files are listed in the package with their fileName and are selected together with the training files; slots filled from checkpoints (source) are listed without a file.

Job settings

await training.createJob({
  wasm: modelFile,
  manifest: packageManifestFile,               // the model's package manifest.json
  files: [],                                   // no dataset; add auxiliary weights files here, if any
  settings: {
    epochs: 30,                                // rounds
    localEpochs: 1,
    batchSize: 16,
    coordinatorParticipates: true,
    experience: { unitsPerAssignment: 32 },
    validationSchedule: { enabled: true, unit: 'round', every: 5 },
  },
});

Model interface

The ABI is the same Weei WASM ABI 1, with these differences for experience models:

• seed is an unsigned 64-bit integer as a decimal string. The same seed must give the same episodes, so a retried assignment repeats the same work.
• unitStart is the global index of the first unit; unitCount is at most unitsPerAssignment.maximum.
• Progress through weei.onProgress reports completed units in processedExamples and the assignment's units in totalExamples.
• A unit that is not finished when training stops counts as zero.

Seeds and reproducibility

When the job is created, the coordinator draws a 64-bit job seed. Each assignment gets the first 64 bits of SHA-256 of the job seed and its first unit index; a retried or reassigned assignment gets the same seed. Evaluation uses one seed derived from the job seed, the same every round. The job seed and the next unit index are saved in every checkpoint. Continuing from a checkpoint with datasetMode: 'resume' (in the widget, Restore the saved dataset progress) continues the same sequence; 'new' draws a new job seed. A continued experience job keeps counting rounds from the checkpoint: settings.epochs (Rounds) is the last round of the job, so a checkpoint from round 397 with 500 rounds trains 103 more, and a value not above 397 is rejected.

Auxiliary weights

Some methods need frozen weights besides the policy: a reference policy for a KL penalty (GRPO) or a reward model (RLHF), or earlier versions of the policy to play against (self-play). The model lists these slots in auxiliaryWeights; Weei only fills them, it does not know what a slot is for. A slot is either a file shipped with the package or a snapshot of the job's own checkpoints, chosen by a rule the model declares:

{
  "auxiliaryWeights": [
    { "id": "reward", "role": "reward-model", "bytes": 1048576, "sha256": "…" },
    { "id": "start", "role": "reference-policy", "bytes": 4096, "source": { "kind": "initial" } },
    { "id": "old", "role": "opponent", "bytes": 4096, "source": { "kind": "checkpoint", "roundsAgo": 5 } }
  ]
}

• A slot has a hash or a source, never both. Job creation fails when a slot with a source declares another size than the weights.
• The snapshot is fixed by the rule and the round, so a retried or reassigned assignment plays against the same version. Evaluation of a checkpoint uses the same rule.
• Files reach participants after the model module and before the first checkpoint, and participants accept only files the model declares. Snapshots reach them right before the checkpoint of the round that needs them, and only when they do not hold that version yet: a rule like roundsAgo: 5 costs one snapshot per participant per round.
• Slots are loaded with loadAuxiliary() after every loadModel(), on every runtime, including the coordinator's own. They are never trained, averaged or sent back.
• Checkpoints record the SHA-256 of file slots by id; continuing from a checkpoint with other files is rejected. Slots with a source are not recorded: the snapshot is a checkpoint of the job already, and all checkpoints are kept.
• At most 8 slots, each a whole number of float32 values, up to 2 GiB. A package lists file slots with their fileName; slots with a source have no file and no fileName.

Requirements and limits

• Training is on-policy: an update counts only for the round's current checkpoint. Updates computed on older weights are discarded, never averaged.
• A model without input data (inputs: "none") that declares maxWorkers above 1 has its assignment split across the participant's local Workers (the Workers per device setting and the device limit apply, up to the number of units). Each Worker plays its own range of units from the same checkpoint with its own seed, and the updates are averaged by units played. Assignments over input records and the GPU backend run on one runtime.
• GPU works for models without input data (inputs: "none"), as a hybrid: the model plays its episodes in WASM on the CPU, and the GPU runs the policy update. The package adds a WGSL archive and a GPU backend, as dataset packages do; gpuBegin receives {experience, settings} without data, each gpuNext returns one update's tensors, and the program's readback includes the weights so the model plays the next episodes with the updated policy. Without the WGSL file the same package trains on the CPU. GPU training over dataset inputs is not available yet.
• Episodes, rewards and trajectories stay on the participant. Only weights, metrics and counts are sent.
• The model module runs the environment: choose only modules you trust, as for any model.

Reference model: CartPole

The CartPole reference model balances a pole on a cart with a softmax policy (4 → 16 tanh → 2) trained by REINFORCE: each assignment plays 32 episodes, with a policy update every 16 episodes on normalized discounted returns. It is built on the TBT library's dense network and optimizer.

Getting the API

The API is available as soon as loadWeei() resolves, if the widget is enabled for your site. You do not need to open the widget window: training runs independently of it, and the API keeps working after the window is closed.

const weei = await window.weeiReady;
const training = weei.widgetsApi.get('distributed-training');
if (!training) throw new Error('Distributed training is not enabled for this site.');

get() returns undefined only when the widget is not enabled for your site (or for this user). Do not wait for the window or call enable() first; enable() only opens the built-in interface.

Rounds and dataset epochs

Example: 40 chunks, 4 participants, epochs: 3 → about 10 rounds per epoch and about 30 checkpoints in total. The job completes after 3 epochs.

interface TrainingSettings {
  epochs: number;                  // planned dataset epochs
  localEpochs: number;             // passes over a participant's chunk per round
  batchSize: number;
  learningRate?: number;
  validationSchedule?: { enabled: boolean; unit: 'round' | 'epoch'; every: number };
  coordinatorParticipates: boolean; // the coordinator also trains
  aggregation?: AggregationSettings; // how rounds are merged; omitted: FedAvg
  experience?: { unitsPerAssignment: number }; // experience jobs only; epochs then counts rounds
}

interface TrainingJobFiles {
  wasm: File;
  manifest: File;
  files: File[];      // dataset chunks named in the manifest, and any auxiliary weights files
  settings: TrainingSettings;
  validation?: File;  // optional validation data, stays with the coordinator
  executionFiles?: File[]; // optional: the model package's WGSL archive, to train on the GPU
}

The pre-release name rounds for epochs is still accepted, and checkpoints that use it still load. Do not pass both with different values.

WASM is required; WGSL is optional. The WASM module is the model: it describes it, creates the initial weights, reads the dataset and trains. A model package may add a WGSL shader archive; with it, the same WASM can run training on the GPU, without it, training runs on the CPU. A WGSL archive cannot be used without its WASM.

Validation data is optional, for a model package and for a plain list of chunks alike. Pass it to follow quality on data the model does not train on: the coordinator then evaluates checkpoints on it, after every round or epoch with settings.validationSchedule, or on demand with runValidation(). Without it the job trains normally and records training metrics only.

The validation file stays on the coordinator and is never sent to participants. Training metrics alone show less well when a model starts to overfit, so pass validation data for real training.

Aggregation

At the end of every round the coordinator first averages the participants' weights, weighted by their unique examples. That average is FedAvg, the default. FedAvgM and FedAdam then treat the change from the round's starting weights to the average as a step direction and apply it with momentum or with a per-weight step size.

type AggregationSettings =
  | { method: 'fedavg' }
  | { method: 'fedavgm'; serverLearningRate?: number; momentum?: number }
  | { method: 'fedadam'; serverLearningRate?: number; beta1?: number; beta2?: number; epsilon?: number };

Here Δ = average − weights at the start of the round, per weight. serverLearningRate is from 0 to 10, momentum, beta1 and beta2 from 0 to below 1, epsilon from 0 to 1. An option that does not belong to the method is rejected.

• Where it runs. Everything happens in the coordinator's browser, after FedAvg. Participants, the model module and the traffic between browsers do not change, and nothing is sent to Weei servers.
• What it costs. One or two extra passes over the weights per round on the coordinator. Checkpoints in IndexedDB and exported files grow to twice (FedAvgM) or three times (FedAdam) the weight size.
• State. Each checkpoint stores the state after its round. Restoring a checkpoint restores its state. Changing the method starts the new method from zero; changing its options keeps the state.
• Non-trainable values. FedAvgM and FedAdam treat every float in the weights as trained, unless the model marks it as average-only.
• Failures. If a step produces a value that is not finite, the round fails with an error and the previous checkpoint stays. Lower serverLearningRate and retry.

Trainable and non-trainable weights

The coordinator sees the weights as one array of floats and does not know what each value means. FedAvg only averages them, which is safe for any value. FedAvgM and FedAdam treat the change of every value during a round as a training step and continue it with momentum or rescale it, so they assume every value in the array is a parameter trained by gradient steps.

If the weight array of your model also holds values that are not trained this way, these methods can damage them. Examples:

• running statistics, such as the mean and variance of BatchNorm;
• counters, such as an optimizer step count or the number of examples seen;
• flags, indexes or other values stored as floats.

Momentum would push such a value further in the direction it changed last round, and FedAdam would move it by a fixed-size step whatever its scale. This is a property of the model's weight layout, not of its training code: the module is not told which method is used. The model in the comparison below showed no such damage with either method.

What the model declares

Only the model knows what its weights hold, so it describes them in the optional aggregation field of getModelInfo(). Weei then applies any method safely to any model that describes itself:

interface ModelAggregationInfo {
  // Methods the model can be trained with. FedAvg is always allowed. Omitted: all methods.
  methods?: Array<'fedavg' | 'fedavgm' | 'fedadam'>;
  // Byte ranges of the weights holding values not trained by gradient steps.
  // offset and bytes are multiples of 4; ranges must not overlap; up to 4096 ranges.
  averageOnly?: Array<{ offset: number; bytes: number }>;
}

Only the metadata changes: the module's training code stays the same. A model without aggregation works as before. If your weights hold values that are not trained by gradient steps, either list them in averageOnly, keep them outside the weight array, or train with FedAvg.

Which method to choose

Measured on a 2.3 MB byte-level transformer (synthetic-reasoning-v2, 55 chunks), 5 participants per round, 3 dataset epochs (33 rounds), the same initial weights and chunk order for every method, with validation after every round:

Here FedAvgM and FedAdam reached the same quality in about a third fewer rounds, and after the same number of rounds the validation loss was about 40% lower. FedAdam started slower: its first rounds were worse than FedAvg, and a step that is too small (0.003) did not catch up. This is one model and one run, not a guarantee: measure on your own model, keep FedAvg if your weights contain values that are not trained by gradient steps, and start with FedAvgM if you try only one method.

await training.updateSettings({
  ...training.getState().job.settings,
  aggregation: { method: 'fedavgm', momentum: 0.9 }
});

Methods

The built-in window uses exactly these methods and this state, so anything it shows or does can be built into your own interface.

Commands check the participant's role: for example, only the coordinator can invite or start. A command that is not allowed now throws an error describing why. subscribe() calls the listener immediately and after every change.

State

getState() returns a copy; changing it has no effect. Files, dataset contents and weights are not included.

interface TrainingState {
  ready: true;
  myPeerId: string;         // this browser's peer ID
  busy: boolean;            // a command is running
  preparing: boolean;       // createJob / continueFromCheckpoint in progress
  transitioning: boolean;   // the coordinator is averaging updates into a checkpoint
  evaluating: boolean;      // validation is running
  error?: string;           // the last error, for display
  peers: string[];          // peer IDs connected to you now
  invitations: Array<[jobId: string, invitation: JobInvitationState]>; // waiting for your decision
  progress: Record<string, PeerProgress>;   // live progress of each training peer, by peer ID
  liveMetrics: Record<string, Array<{ at: number; values: Record<string, number> }>>; // by peer ID
  transfers: TransferProgress[];
  transferErrors: Record<string, string>;
  estimatedRemainingMs?: number;
  pausedContribution: boolean;
  activeWorkers: number;    // training Workers running on this device now
  backendPreference: 'auto' | 'cpu' | 'gpu' | 'hybrid';
  execution: { requested: string; actual: string; phase: string; fallback: string }; // this device
  maxLocalWorkers: number;  // setMaxLocalWorkers(), saved for this browser
  gpuMemoryBudgetMiB: number; // setGpuMemoryBudgetMiB(), saved for this browser
  diagnostics: Record<string, PeerExecutionDiagnostics>; // by peer ID, see Execution diagnostics
  job?: TrainingJob;        // the job you coordinate or take part in
}

job

interface TrainingJob {
  id: string;
  role: 'coordinator' | 'participant';
  status: 'ready' | 'running' | 'paused' | 'completed';
  owner: string;                    // coordinator's peer ID
  info: ModelInfo;                  // getModelInfo() of the module
  settings: TrainingSettings;
  participants: Record<string, ParticipantState>; // by peer ID
  progress?: {                      // dataset progress reported by the coordinator
    datasetEpoch: number;           // completed passes
    plannedEpochs: number;          // settings.epochs
    remainingChunks: number;
    totalChunks: number;
    running: boolean;
    estimatedRemainingMs?: number;
  };
  datasetEpoch?: number;            // completed passes (coordinator)
  queue: string[];                  // chunk IDs left in the current pass
  checkpoints: CheckpointSummary[]; // oldest first
  slots: RoundSlot[];               // assignments of the current round
  localWorkers: {                   // this device
    requested: number;              // settings.workers
    capacity: number;               // what the model and this device allow
    effective: number;              // usable for the current or next assignment
    active: number;                 // running now
  };
  incomplete: boolean;              // finished before every chunk was trained
  shuffled: boolean;                // chunks were shuffled; shuffleChunks() shuffles again
  validationUnavailable: boolean;   // a validation file was set but is missing from browser storage
  module: { fileName: string; bytes: number; sha256: string };
  hasModuleFile: boolean;           // exportModule() can return the model module
}

interface ParticipantState {
  connected: boolean;
  approved: boolean;
  paused?: boolean;
  state: 'invited' | 'approved' | 'declined' | 'unknown' | 'disconnected' | 'modelLoaded'
       | 'assigned' | 'training' | 'updateReady' | 'resultReceived' | 'stopped' | 'modelLoadFailed';
}

interface RoundSlot {
  peerId: string;
  chunkId: string;
  status: 'loading' | 'training' | 'received' | 'stopped';
  startedAt?: number;
  error?: string;
}

Checkpoints, metrics and validation

interface CheckpointSummary {
  artifact: { round: number; sha256: string; bytes: number; fileName: string };
  metrics: TrainingMetrics;            // averaged training metrics of this round
  validationMetrics?: TrainingMetrics; // when validation ran for this checkpoint
  validationError?: string;
  datasetEpoch?: number;               // completed passes at this checkpoint
  datasetProgress?: number;            // passes done, with a fraction: 1.25 = a quarter into the 2nd pass
  remaining: string[];                 // chunk IDs left in the pass
  settings: TrainingSettings;
  aggregation?: {                      // FedAvgM / FedAdam state saved with this checkpoint
    method: 'fedavgm' | 'fedadam';
    step: number;                      // rounds aggregated with this state
    bytes: number;
  };
  createdAt: number;
}

interface TrainingMetrics {
  values: Record<string, number>; // keys are the metric IDs from ModelInfo.metrics
  examplesProcessed: number;
}

Metric names, labels and whether lower or higher is better come from job.info.metrics ({ id, label, direction: 'minimize' | 'maximize' }). To chart training, plot checkpoint.metrics.values[id] against checkpoint.artifact.round; validation results are in validationMetrics of the same checkpoint.

Live progress

interface PeerProgress {
  processedExamples: number;
  totalExamples?: number;
  metrics?: Record<string, number>; // latest values reported by the module
  recordsPerSecond?: number;
  updatedAt: number;
}

interface TransferProgress {
  jobId: string;
  peerId: string;
  artifactId: string;
  direction: 'send' | 'receive';
  bytes: number;
  totalBytes: number;
  bytesPerSecond: number;
  remainingSeconds?: number;
}

Execution diagnostics

state.diagnostics shows where, how and how fast each peer trains. Every peer has its own entry; a coordinator also receives one from every participant. Use it to compare participants, for example Chrome and Firefox running the same package.

• Participants send a snapshot to the coordinator every 2 seconds while they have a job and immediately on every stage change. After a reconnection the participant sends a complete snapshot at once.
• Fields that a peer cannot determine are the string 'unknown'. Never treat 'unknown' as a reason: show it as missing data.
• Telemetry contains no dataset contents and no weights.

type Unknown = 'unknown';

interface PeerExecutionDiagnostics {
  schemaVersion: 1;
  sequence: number;     // increases with each snapshot of this peer
  measuredAt: number;   // when the peer took it, peer's clock, ms since epoch
  receivedAt: number;   // when this browser received it, local clock, ms since epoch
  ageMs: number;        // local time since receivedAt, at getState()
  stale: boolean;       // ageMs > 10000, or the peer is disconnected
  connected: boolean;

  environment: {
    browser: string | Unknown;         // 'Chrome', 'Firefox', 'Safari', 'Edge', 'Opera'
    browserVersion: string | Unknown;
    os: string | Unknown;              // 'Windows', 'macOS', 'Linux', 'Android', 'iOS', 'ChromeOS'
    logicalCores: number | Unknown;
  };
  execution: {
    requested: 'auto' | 'cpu' | 'gpu' | 'hybrid';
    actual: 'cpu' | 'gpu' | 'hybrid' | Unknown;  // unknown until the peer has started computing
    backend: string | Unknown;         // backend ID from the model package, or 'cpu-wasm'
    fallbackReason: string | null;     // null: actual is what was requested
    activeWorkers: number;             // training Workers running now
    workerLimit: number;               // Workers this peer allows
  };
  gpu: {
    webgpu: 'available' | 'unavailable';      // navigator.gpu exists in this browser
    adapter: { vendor: string; architecture: string; device: string; description: string } | Unknown;
    memoryBudgetMiB: number;                  // the configured budget, NOT memory in use
    initializationError: string | null | Unknown; // last GPU attempt: error, null on success, unknown if none
  };
  model: {
    id: string;
    version: string;
    wasmSha256: string;
    wgsl: { present: boolean; sha256: string[] };
  } | Unknown;
  assignment: {                        // what timings and performance belong to; null between assignments
    jobId: string;
    round: number;
    assignmentId: string;
    attemptId: string;
    chunkId: string | Unknown;         // filled in on the coordinator
    bytes: number;
    exampleCount: number;
    batchSize: number;
    localEpochs: number;
    learningRate: number | Unknown;    // unknown: the job leaves it to the model's default
  } | null;
  stage: 'waiting' | 'receiving' | 'preparing' | 'compiling' | 'training'
       | 'uploading' | 'awaitingAggregation' | 'error';
  error: string | null;
  timings: {                           // ms, this assignment only; null until the stage has finished
    preparationMs: number | null;
    compilationMs: number | null;      // GPU only
    trainingMs: number | null;
    uploadMs: number | null;
  };
  performance: {
    processedExamples: number;
    totalExamples: number | Unknown;
    optimizerSteps: number | Unknown;  // reported only by models that count steps
    currentExamplesPerSecond: number | null;  // last 5 s of training
    averageExamplesPerSecond: number | null;  // processedExamples / training time
    lastProgressAt: number | null;            // peer's clock, ms since epoch
  };
}

Stages

How figures are measured

• Durations are measured on the peer with a monotonic clock and count only their own stage: trainingMs excludes transfer, preparation and waiting. A stage entered twice is summed.
• averageExamplesPerSecond divides processed examples by training time only. currentExamplesPerSecond uses progress samples of the last 5 seconds and is null until two samples exist.
• All timings and performance figures belong to assignment; they reset when the next assignment starts.
• measuredAt and lastProgressAt use the peer's clock, which may differ from yours; use ageMs and stale to judge freshness.
• memoryBudgetMiB is a limit set on the peer. Browsers do not report GPU memory use.

Disconnection and reconnection

A disconnected participant keeps its last snapshot with connected: false and stale: true. When it reconnects it sends a new complete snapshot; snapshots with a lower sequence than the one already received are ignored. Diagnostics of another job are not shown.

Example: compare participants

const weei = await window.weeiReady;
const training = weei.widgetsApi.get('distributed-training');
const show = value => (value === 'unknown' || value === null ? '—' : value);

training.subscribe(state => {
  const rows = Object.entries(state.diagnostics).map(([peerId, d]) => ({
    peer: peerId,
    browser: show(d.environment.browser) + ' ' + show(d.environment.browserVersion),
    mode: show(d.execution.actual) + (d.execution.fallbackReason ? ' (fallback: ' + d.execution.fallbackReason + ')' : ''),
    workers: d.execution.activeWorkers,
    batch: d.assignment ? d.assignment.batchSize : '—',
    examples: d.assignment ? d.assignment.exampleCount : '—',
    trainingMs: show(d.timings.trainingMs),
    speed: show(d.performance.currentExamplesPerSecond),
    fresh: d.stale ? 'stale ' + Math.round(d.ageMs / 1000) + ' s' : 'live'
  }));
  console.table(rows);
});

Invitations

A participant sees each pending invitation in state.invitations as [jobId, invitation], where invitation.invitation holds the coordinator's peer ID, modelInfo, module size and hash, and the settings. Accept or decline with decideInvitation(jobId, accepted).

Example: run a job from your own interface

const weei = await window.weeiReady;
const training = weei.widgetsApi.get('distributed-training');

const stop = training.subscribe(state => {
  const job = state.job;
  if (!job) return;
  const passes = job.progress ? job.progress.datasetEpoch + ' / ' + job.progress.plannedEpochs : '';
  const last = job.checkpoints.at(-1);
  console.log(job.status, passes, last?.metrics.values, Object.keys(job.participants));
});

await training.createJob({
  wasm: wasmInput.files[0],
  manifest: manifestInput.files[0],
  files: [...datasetInput.files],
  settings: { epochs: 3, localEpochs: 1, batchSize: 32, coordinatorParticipates: true }
});
for (const peerId of training.getState().peers) training.invite(peerId);
await training.start();

Continue from a checkpoint

const checkpoint = training.exportCheckpoint(); // the latest checkpoint, as a File
const wasm = training.exportModule();           // the job's model, if this browser keeps it
await training.leave();                          // the current job stays in Saved jobs
await training.continueFromCheckpoint({
  checkpoint,
  wasm, manifest, files, settings,
  datasetMode: 'new' // 'new': fresh dataset queue; 'resume': restore the old queue
});

An exported checkpoint of a FedAvgM or FedAdam job also contains the method's state (file format version 2). It continues when settings.aggregation uses the same method; with another method, training starts that method from zero. Version 1 files, without state, still load.

Exported files

Three methods return a File you can save, upload or pass back to the API. None of them contains the dataset.

Checkpoint and module files can be large: a checkpoint is the weight size plus one or two copies of it for FedAvgM or FedAdam. Weei accepts checkpoint and module files of up to 256 MB.

Debug log

The log keeps the last 1,000 events of this browser profile, across jobs and page reloads, in localStorage. getDebugLog() returns the events; clearDebugLog() deletes them.

interface TrainingDebugLogFile {
  schemaVersion: 2;
  exportedAt: string;               // ISO 8601
  baseline: {
    // Duration of each finished phase, for example 'fedavg' or 'indexeddb.checkpoint-save'.
    phases: Record<string, { count: number; minMs: number; avgMs: number; p50Ms: number; maxMs: number }>;
    idle: { samples: number; totalMs: number; avgMs: number }; // gaps between phases of a round
  };
  events: Array<{
    at: string;                     // ISO 8601
    event: string;                  // for example 'phase.finished', 'aggregation.started'
    details?: Record<string, unknown>;
  }>;
}

Checkpoint file format

A .weei-checkpoint is a single binary file. Numbers in the header and the weights are little-endian.

Version 1 has no aggregation state: everything after the metadata is weights. Weei writes version 2 only for checkpoints of FedAvgM and FedAdam jobs, and reads both.

interface CheckpointMetadata {
  version: 1 | 2;
  jobId: string;
  module: { artifactId: string; fileName: string; bytes: number; sha256: string }; // the model module
  modelInfo: ModelInfo;             // getModelInfo() of the module
  settings: TrainingSettings;       // settings of the round that produced the checkpoint
  round: number;                    // the round number, 0 for the initial weights
  checkpoint: {                     // the weights
    artifactId: string;
    kind: 'checkpoint';
    fileName: string;
    format: 'float32-le';
    bytes: number;                  // size of the weights
    sha256: string;                 // SHA-256 of the weights, lowercase hex
    round: number;
  };
  metrics: TrainingMetrics;         // averaged training metrics of the round
  incomplete: boolean;              // the job finished before every chunk was trained
  chunkOrder?: string[];            // shuffled order of the chunk IDs, if the chunks were shuffled
  // Dataset progress: whole chunks…
  trainedChunkIds?: string[];
  remainingChunkIds?: string[];
  // …or, for models that split chunks, the ranges still to train, by partition ID.
  remainingWorkItems?: Record<string, { partitionId: string; sourceChunkId: string; recordStart: number; recordCount: number }>;
  experience?: { jobSeed: string; nextUnit: number }; // experience jobs: job seed and next unit index
  auxiliaryWeights?: Record<string, string>;            // SHA-256 of each auxiliary weights file, by id
  aggregationState?: {              // version 2 only
    method: 'fedavgm' | 'fedadam';
    step: number;                   // rounds aggregated with this state, 1 or more
    bytes: number;                  // size of the state: weights × 1 (FedAvgM) or × 2 (FedAdam)
    sha256: string;                 // SHA-256 of the state, lowercase hex
  };
}

What Weei checks when it reads a checkpoint

• The metadata length fits the file and is at most 1 MiB; the metadata is valid JSON with version 1 or 2, a whole round of 0 or more and checkpoint.format float32-le.
• The weights are exactly checkpoint.bytes long, every value is finite, and their SHA-256 equals checkpoint.sha256.
• In version 2, the state is exactly aggregationState.bytes long, which is one (FedAvgM) or two (FedAdam) times the weight size; its values are finite and its SHA-256 equals aggregationState.sha256.
• modelInfo, settings and metrics are valid. settings may use the pre-release name rounds for epochs.
• trainedChunkIds and remainingChunkIds come together, contain no duplicates and do not overlap; every key of remainingWorkItems is in remainingChunkIds.
• The module you select must load the weights: its loadModel() decides compatibility. The module's hash may differ from module.sha256, for example after a compatible rebuild.

A file that fails a check is rejected with an error naming the problem, and no job is created.

Example: read a checkpoint outside Weei

async function readCheckpoint(file) {
  const bytes = await file.arrayBuffer();
  const metadataBytes = new DataView(bytes).getUint32(0, true);
  const metadata = JSON.parse(new TextDecoder().decode(new Uint8Array(bytes, 4, metadataBytes)));
  const start = 4 + metadataBytes;
  const weights = new Float32Array(bytes.slice(start, start + metadata.checkpoint.bytes));
  const state = metadata.version === 2
    ? new Float32Array(bytes.slice(start + metadata.checkpoint.bytes))
    : undefined;
  return { metadata, weights, state };
}