Week 40 Bucket Submission Post-Mortem
Glow Team / September 10, 2024
Introduction
During week 40 of the Glow protocol, an issue arose in the submission process for the weekly report. It's important to note that the week 40 report contains all the data from week 39, as per the protocol's reporting structure on the weekly reports page.
A bug in the submission script led to the publication of incorrect rewards weights on-chain. This error was promptly identified by the Veto Council, who took immediate action by delaying the finalization of the affected bucket for 90 days. As a consequence, user rewards are expected to be delayed for 100 days.
The impact of this incident is significant but temporary. With the upcoming launch of GlowV2 on the horizon, there is a strong expectation that this error will be rectified as part of the relaunch process. This incident underscores the importance of rigorous checking and the value of having oversight mechanisms like the Veto Council in place to catch and address such issues promptly.
Key Details
Event | Date | Transaction |
---|---|---|
Faulty Submission | August 31, 2024 | View on Etherscan |
Issue Detected | September 6, 2024 | N/A |
Bucket Delay by Veto Council | September 6, 2024 | View on Etherscan |
Affected Systems/Components
The rewards for bucket 40 (containing the week 39 weekly report) are expected to be delayed for 100 days.
Incident Description
An automated script used by Glow Certification Agents (GCAs) to generate weekly on-chain reports contained an error. This error incorrectly converted a farm's carbon credit production into an Ethereum uint256, significantly overstating its output. While this didn't affect the total carbon credits reported on-chain, it skewed the reward distribution weights. As USDG rewards are allocated based on relative carbon production, this farm could have claimed a disproportionately large share of USDG compared to other farms.
Discovery
Simon from the Veto Council was reviewing the weekly report and noticed the inconsistency. He confirmed the inconsistency, found the root source of the bug in the automated script, and then delayed the bucket.
Impact
The impact is that the rewards for bucket 40 (containing the week 39 weekly report) are expected to be delayed for 100 days.
Root Cause Analysis
The code used to create the report for week 40 contained a bug in the conversion of float values to BigNumbers. Specifically, the issue occurred in the following code block:
const finalLeaves: FinalLeaf[] = merkleLeaves.map(
({ wallet, glowWeight, usdgWeight }) => ({
wallet,
glowWeight: parseUnits(
glowWeight.toString(),
GLOW_WEIGHT_DECIMAL_PRECISION
).toString(),
usdgWeight: parseUnits(
usdgWeight.toString(),
USDG_WEIGHT_DECIMAL_PRECISION
).toString(),
})
);
One of the devices had a very small amount of carbon credits produced that caused the USDG weight to be represented in scientific notation. The exact number was: 9.955189695275401e-7
. This number is then stringified and parsed into a BigInt using viem's parseUnits
function. The error came from viem not recognizing scientific notation. This caused viem to parse 9.9551...
with USDG_WEIGHT_DECIMALS
rather than parsing .000000995....
into a BigNumber which caused the reported weight to be 10x bigger than intended.
Resolution
The Fix
To properly reconcile the rewards for week 40, the veto council delayed the finalization of the bucket for 90 days. It is expected that GlowV2 will launch before the bucket finalizes, and the rewards will be corrected upon relaunch. In the case that GlowV2 is no longer expected to launch before the new bucket finalization timestamp, Governance is expected to slash the GCA which will invalidate the report and give new GCAs time to submit the correct report.
Preventative Measures
Patch
- As soon as the bug was discovered, Simon reached out to the viem team to address the issue.
- The viem team acknowledged the severity of the issue and is planning to implement a patch to throw on scientific notation strings. The planned patch can be found here
The automated script has also now been adjusted to include several more checks:
- Convert numbers using
customToFixed
which ensures proper decimals as opposed to usingtoString
- Added invariant checks to ensure that carbon credits produced and weekly payments match up with the usdg and glow weight reported respectively in this commit with tests for the function here
Related articles
David Vorick / July 4, 2024
5 min readWhat is Glow?
An overview of the Glow protocol and its mission to revolutionize the solar energy landscape.
Simon Boccara / April 10, 2024
5 min readExploring the GCA Protocol Binary Bug
We go through the GCA Binary bug that caused weeks [0-8] to be aggregated into a single week. We explore how this affected the rewards for farms and the carbon credits submitted on-chain.
David Vorick / March 29, 2024
7 min readThe First Major Refactor of the Glow Audit Standard
Discloses the first cheating incident on the Glow protocol, and explains how the audit standard was refactored to be more effective.
David Vorick / January 31, 2024
10 min readThe Glow Impact Platform
Rebuilding climate strategy from first principles.
David Vorick / January 9, 2024
3 min readGuarded Launch: Protecting Glow Users Against Hacks
Code audits are both expensive and insufficient; using a guarded launch can improve safety while reducing costs and time-to-market.
David Vorick / December 19, 2023
5 min readWhy Glow Excites Me
How I found my way to Glow, and why Glow is the path to effective climate action.