1. Introduction
During week 40 of the Glow protocol, an issue arose in the submission process for the weekly report. It's important to note that the week 40 report contains all the data from week 39, as per the protocol's reporting structure on the weekly reports page.
A bug in the submission script led to the publication of incorrect rewards weights on-chain. This error was promptly identified by the Veto Council, who took immediate action by delaying the finalization of the affected bucket for 90 days. As a consequence, user rewards are expected to be delayed for 100 days.
The impact of this incident is significant but temporary. With the upcoming launch of GlowV2 on the horizon, there is a strong expectation that this error will be rectified as part of the relaunch process. This incident underscores the importance of rigorous checking and the value of having oversight mechanisms like the Veto Council in place to catch and address such issues promptly.
1.1 Key Details
| Event | Date | Transaction |
|---|---|---|
| Faulty Submission | August 31, 2024 | View on Etherscan |
| Issue Detected | September 6, 2024 | N/A |
| Bucket Delay by Veto Council | September 6, 2024 | View on Etherscan |
1.2 Affected Systems/Components
The rewards for bucket 40 (containing the week 39 weekly report) are expected to be delayed for 100 days.
2. Incident Description
An automated script used by Glow Certification Agents (GCAs) to generate weekly on-chain reports contained an error. This error incorrectly converted a farm's carbon credit production into an Ethereum uint256, significantly overstating its output. While this didn't affect the total carbon credits reported on-chain, it skewed the reward distribution weights. As USDG rewards are allocated based on relative carbon production, this farm could have claimed a disproportionately large share of USDG compared to other farms.
2.1 Discovery
Simon from the Veto Council was reviewing the weekly report and noticed the inconsistency. He confirmed the inconsistency, found the root source of the bug in the automated script, and then delayed the bucket.
2.2 Impact
The impact is that the rewards for bucket 40 (containing the week 39 weekly report) are expected to be delayed for 100 days.
3. Root Cause Analysis
The code used to create the report for week 40 contained a bug in the conversion of float values to BigNumbers. Specifically, the issue occurred in the following code block:
1const finalLeaves: FinalLeaf[] = merkleLeaves.map(
2 ({ wallet, glowWeight, usdgWeight }) => ({
3 wallet,
4 glowWeight: parseUnits(
5 glowWeight.toString(),
6 GLOW_WEIGHT_DECIMAL_PRECISION
7 ).toString(),
8 usdgWeight: parseUnits(
9 usdgWeight.toString(),
10 USDG_WEIGHT_DECIMAL_PRECISION
11 ).toString(),
12 })
13);One of the devices had a very small amount of carbon credits produced that caused the USDG weight to be represented in scientific notation. The exact number was: 9.955189695275401e-7. This number is then stringified and parsed into a BigInt using viem's parseUnits function. The error came from viem not recognizing scientific notation. This caused viem to parse 9.9551... with USDGWEIGHTDECIMALS rather than parsing .000000995.... into a BigNumber which caused the reported weight to be 10x bigger than intended.
4. Resolution
4.1 The Fix
To properly reconcile the rewards for week 40, the veto council delayed the finalization of the bucket for 90 days. It is expected that GlowV2 will launch before the bucket finalizes, and the rewards will be corrected upon relaunch. In the case that GlowV2 is no longer expected to launch before the new bucket finalization timestamp, Governance is expected to slash the GCA which will invalidate the report and give new GCAs time to submit the correct report.
5. Preventative Measures
5.1 Patch
1. As soon as the bug was discovered, Simon reached out to the viem team to address the issue. 2. The viem team acknowledged the severity of the issue and is planning to implement a patch to throw on scientific notation strings. The planned patch can be found here
The automated script has also now been adjusted to include several more checks:
- Convert numbers using
customToFixedwhich ensures proper decimals as opposed to usingtoString - Added invariant checks to ensure that carbon credits produced and weekly payments match up with the usdg and glow weight reported respectively in this commit with tests for the function here