Week 40 Bucket Submission Post-Mortem

Introduction

During week 40 of the Glow protocol, an issue arose in the submission process for the weekly report. It's important to note that the week 40 report contains all the data from week 39, as per the protocol's reporting structure on the weekly reports page.

A bug in the submission script led to the publication of incorrect rewards weights on-chain. This error was promptly identified by the Veto Council, who took immediate action by delaying the finalization of the affected bucket for 90 days. As a consequence, user rewards are expected to be delayed for 100 days.

The impact of this incident is significant but temporary. With the upcoming launch of GlowV2 on the horizon, there is a strong expectation that this error will be rectified as part of the relaunch process. This incident underscores the importance of rigorous checking and the value of having oversight mechanisms like the Veto Council in place to catch and address such issues promptly.

Key Details

Affected Systems/Components

The rewards for bucket 40 (containing the week 39 weekly report) are expected to be delayed for 100 days.

Incident Description

An automated script used by Glow Certification Agents (GCAs) to generate weekly on-chain reports contained an error. This error incorrectly converted a farm's carbon credit production into an Ethereum uint256, significantly overstating its output. While this didn't affect the total carbon credits reported on-chain, it skewed the reward distribution weights. As USDG rewards are allocated based on relative carbon production, this farm could have claimed a disproportionately large share of USDG compared to other farms.

Discovery

Simon from the Veto Council was reviewing the weekly report and noticed the inconsistency. He confirmed the inconsistency, found the root source of the bug in the automated script, and then delayed the bucket.

Impact

The impact is that the rewards for bucket 40 (containing the week 39 weekly report) are expected to be delayed for 100 days.

Root Cause Analysis

The code used to create the report for week 40 contained a bug in the conversion of float values to BigNumbers. Specifically, the issue occurred in the following code block:

const finalLeaves: FinalLeaf[] = merkleLeaves.map(
  ({ wallet, glowWeight, usdgWeight }) => ({
    wallet,
    glowWeight: parseUnits(
      glowWeight.toString(),
      GLOW_WEIGHT_DECIMAL_PRECISION
    ).toString(),
    usdgWeight: parseUnits(
      usdgWeight.toString(),
      USDG_WEIGHT_DECIMAL_PRECISION
    ).toString(),
  })
);

One of the devices had a very small amount of carbon credits produced that caused the USDG weight to be represented in scientific notation. The exact number was: 9.955189695275401e-7. This number is then stringified and parsed into a BigInt using viem's parseUnits function. The error came from viem not recognizing scientific notation. This caused viem to parse 9.9551... with USDG_WEIGHT_DECIMALS rather than parsing .000000995.... into a BigNumber which caused the reported weight to be 10x bigger than intended.

Resolution

The Fix

To properly reconcile the rewards for week 40, the veto council delayed the finalization of the bucket for 90 days. It is expected that GlowV2 will launch before the bucket finalizes, and the rewards will be corrected upon relaunch. In the case that GlowV2 is no longer expected to launch before the new bucket finalization timestamp, Governance is expected to slash the GCA which will invalidate the report and give new GCAs time to submit the correct report.

Preventative Measures

Patch

1. As soon as the bug was discovered, Simon reached out to the viem team to address the issue.
2. The viem team acknowledged the severity of the issue and is planning to implement a patch to throw on scientific notation strings. The planned patch can be found here

The automated script has also now been adjusted to include several more checks:

1. Convert numbers using customToFixed which ensures proper decimals as opposed to using toString
   • The function can be found here
   • The tests for the function can be found here
2. Added invariant checks to ensure that carbon credits produced and weekly payments match up with the usdg and glow weight reported respectively in this commit with tests for the function here