Methodology

This page describes exactly how Vidhata currently computes question probability, individual Brier scores, user averages, and leaderboard ordering in the present web-only architecture (Firestore + Auth). It reflects the current implementation, including safeguards, constraints, and manual admin operations that will later move to Cloud Functions.

Current: Firestore-driven Planned: Cloud Functions automation

1. Question Lifecycle and Timeline

Each prediction question moves through a strict lifecycle. Forecasting is only accepted while status is open. Admin can move a question to closed (forecasting stopped, awaiting official outcome), then to resolved.

Stage	Who	Data Written	Timing
Create Question	Authenticated user	`questions/{id}` with title, description, category, probability, resolutionDate, effectiveCloseAt, resolutionCriteria, createdBy, createdByName, createdAt, updatedAt, status=open	One-time at creation
Submit Forecast	Authenticated forecaster	History entry in `forecasts` + overwrite latest in `latestForecastByUser/{uid}`	Any time while open, but max once per 5 minutes per user/question
Recompute Aggregate	Admin	Updates question `probability` and `forecastCount`	Manual button now; target is automated trigger later
Resolve	Admin	Sets status=resolved, resolutionOutcome (0/1), resolvedAt, resolutionSource, effectiveCloseAt	After external event is deterministically known
Score	Admin	Writes per-user score docs, updates user aggregates + leaderboard docs, marks scoredAt on question	After resolution, repeatable safely for idempotent recalculation

2. Forecast Data Model and Anti-Spam Rules

Vidhata keeps both immutable history and one authoritative latest forecast per user. This allows auditability plus fast reads for aggregation/scoring.

questions/{id}/forecasts/{autoId}: append-only history of submissions.
questions/{id}/latestForecastByUser/{uid}: single latest forecast used for aggregation and scoring.
Security rules enforce 0 <= probability <= 1.
Security rules enforce question must be open.
Security rules block forecast writes after effectiveCloseAtTs when set.
Security rules enforce a 5-minute cooldown per user per question, based on updatedAt.

Practical effect: users can revise over time. Latest values drive consensus, while full forecast history is used for time-weighted scoring.

3. Question Probability Aggregation

Current aggregate uses latest forecast per user with skill weighting.

For each user i: p_i = latest forecast probability in [0,1] baseSkill_i = 1 - brierScoreAvg_i w_i = skill weight Question consensus: question_probability = sum(w_i * p_i) / sum(w_i) forecastCount = number of latest forecasters Fallback: If no forecasts (or no valid total weight), probability = 0.5

Source: all docs from latestForecastByUser.
Each probability is clamped to [0,1] before inclusion.
Skill weights use 1 - brierScoreAvg with operational safeguards.
For low-history users (resolvedCount below threshold), a default neutral weight is used.
Skill weights are clamped to a configured range to prevent dominance.
Stored value is rounded to 4 decimals before write.
Output fields on question doc: probability, forecastCount, updatedAt.

Current trigger is manual via admin action. Intended future state is automatic recompute in Cloud Functions on every accepted forecast write.

4. Individual Brier Score Calculation

Vidhata uses time-weighted Brier per resolved question. Lower is better.

For a resolved binary event: o = final outcome in {0,1} Question close time for scoring = effectiveCloseAt (fallback: resolvedAt) Each forecast interval k has: p_k = probability held during interval k delta_t_k = duration of interval k time_weighted_brier = sum(delta_t_k * (p_k - o)^2) / sum(delta_t_k)

Interpretation:

Forecasts that were accurate for longer durations get more credit.
Last-minute obvious updates contribute less because their time exposure is small.
Perfect prediction path yields 0.0. Worst possible path approaches 1.0.
If no interval duration can be formed (edge case), the final pre-close forecast is used as fallback for that question score.

For each resolved question, Vidhata computes one time-weighted score per user and writes it to users/{uid}/scores/{questionId}.

5. User Aggregate Score (Profile-Level)

After per-question scores are written, each user gets an aggregate average over their scored resolved questions.

Let user scores be b1, b2, ..., bK: user_brierScoreAvg = (b1 + b2 + ... + bK) / K resolvedCount = K

Stored on users/{uid} as brierScoreAvg and resolvedCount.
lastScoredAt and updatedAt are updated at run time.
Rules allow only admin to write these scoring summary fields.

6. Leaderboard Computation

Leaderboard reads sanitized public docs from leaderboard/{uid}, populated from the scoring pass.

Sort order: ascending brierScoreAvg (lower is better).
Tie-break: higher resolvedCount first.
Qualified board threshold: resolvedCount >= 10.
Developing board: resolvedCount < 10.

Writes to leaderboard docs are admin-only by rules. Reads are public.

7. Resolution Standards and Governance

Each question must include resolution criteria and a source field is required at resolution. This keeps scores auditable.

resolutionCriteria is specified up front when question is created.
effectiveCloseAt is used as the scoring cutoff when uncertainty is effectively over before nominal resolution date.
effectiveCloseAtTs (timestamp) is used for rule-level forecast cutoff enforcement.
resolutionSource must be supplied when resolving.
resolvedAt stores when resolution was recorded.
scoredAt + scoredForResolvedAt identify whether scoring is current for that resolution state.

8. Current Limitations and Planned Upgrade

Aggregation and scoring are currently admin-triggered from UI buttons.
This is operationally acceptable for early stage and Spark constraints.
On Blaze, the same logic should move to Cloud Functions to ensure automatic, tamper-resistant, low-latency updates.
When moved, admin UI buttons can be removed and rules tightened further.