SUI Testnet: Wave 2 Report
by: Trace and Velvet
Acknowledgments: Special thanks to Zygis Magelinskas (Skip Protocol), Jenny Shek (Figment), Saad Asif (Figment), and Silviu Gae (Bware Labs), whose insights and perspectives helped contribute to this report.
Overview
Sui Testnet Wave 2 (‘Wave 2”) ran from January 25th to February 15th, 2023. To better understand the network, our team monitored the testnet and analyzed on-chain and off-chain data. In this report, we summarize our quantitative and qualitative findings from observing Wave 2. The Mysten Labs team has also published results on Wave 2 testnet activity and network economics.
Sui’s first testnet successfully generated basic network stability. Wave 2 built upon that progress. This involved expanding the validator set from 20 to 41 and onboarding Sui developers from devnet to testnet. Although many experiments were run during the testnet, the central focus was on the network’s economics. The Mysten team used Wave 2 to better understand how Sui’s economics will function during Mainnet and to help validators gain familiarity with the chain’s unique tokenomic model. We concluded our Sui Testnet Wave 1 report with a list of open questions we hoped to investigate and answer during Wave 2. However, given Wave 2’s emphasis, our team decided to shift this report’s focus and dive deeper into the network’s economics instead.
Our key findings include:
- After an initial learning curve, most validators were successfully able to quote profitable reference gas prices (RGPs)
- Over 90% of validators achieved a greater number of profitable epochs than unprofitable ones; only 1 validator was net unprofitable over the entire testnet
- The network had a strong inverse correlation between the SUI token price and the Reference Gas Price, with top performing validators averaging a -0.69 correlation coefficient from epoch 8 onward.
- The network maintained stability with the exception of a few minor challenges; network bugs were generally identified and patched in under 6 hours
Economics
Sui’s tokenomic model attempts to maintain stable gas prices despite variance in network demand. To do this, the network sets a Reference Gas Price (RGP) at the start of every epoch. Sui epochs are 24 hours long. If a transaction has a gas price >= RGP, the user can expect their transaction will be included in a block. At each epoch boundary, validators vote on the next epoch’s RGP. The RGP is then set as the 2/3rd mean by stake of all validators’ votes.
Validators can update their RGP vote arbitrarily or choose a more sustainable approach. For operators who chose the latter, many prepared a financial model that adjusted their RGP based on several factors including the current Sui token price, their delegated stake, overhead costs (such as hardware, labor, maintenance, etc), and the price quotes set by other validators. As this process is unique to Sui, Wave 2 offered node operators their first opportunity to familiarize themselves with this novel gas mechanism and a chance to prepare a gas price model ahead of Mainnet.
Validator Contest: Setup
Since the testnet SUI token had no real-world value, the Mysten team created participation incentives for validators to dynamically quote gas prices. The team created a “validator contest” (the “Contest”) where each validator competed against one another in order to score points. The highest scoring validator by the end of testnet would be the winner. To increase participation among validators, the Mysten team informed everyone that their performance in the Contest will play a role in the Sui Foundation’s future token delegations.
The Contest awarded validators points based on the quality of their RGP votes. After each epoch, validators received 2 points if their RGP vote achieved a profitable epoch for the validator, and 0 points if they were unprofitable. Validators received one additional point if their gas quote was included in the bottom 2/3’s of RGP quotes by stake; this was done in an effort to incentivize validators to drive costs down. Validators that remained profitable while setting gas quotes below the RGP demonstrated an expert understanding of the Sui economic model. Note that this point system is only for the purposes of the testnet’s Validator Contest; it’s not an enshrined part of the protocol.
Compared to Sui’s proposed economics during Mainnet, the economics of the Contest were significantly more straightforward. During testnet, validators were informed ahead of time of the Sui token price, and the price only changed at epoch boundaries. Moreover, validators were given a hypothetical operating cost, whereas real operating costs at Mainnet will need to be measured by each validator according to their unique setup. A validator’s revenue and cost per epoch were calculated as follows:
Revenue in epoch e = (stake rewards in epoch e) * (SUI price in epoch e)
Cost in epoch e = (gas units executed in epoch e) * (cost per unit in epoch e)
The number of gas units executed is proportional to the amount of activity on Sui during that epoch, and is the same for every validator. The cost per unit is unique to each validator and is the marginal cost per gas unit for that epoch, estimated across all real-world expenses including hardware, electricity, and labor costs. Staking rewards are a function of the validator’s stake, their commission rate, the amount of network activity, and the reference gas price, among other factors. Given all variables, validators needed to vote on a RGP that made their revenue exceed their costs.
The Mysten team was helpful in explaining to validators the relevant information to produce a working model, and built a template calculator to guide validators as they configured their RGP model. Nonetheless, the responsibility remained on the node operator to build their own process.
An added complexity to the Validator Contest was Frenemies, another testnet economics game played by users, rather than validators. Frenemies provided users with the opportunity to experiment with Sui’s staking functionality for a chance to receive early SUI tokens. Each epoch, players would be assigned a validator and a role of either “friend,” “neutral,” or “enemy.” Players delegated their stake differently depending on the validator they were assigned and their role. Ten points were awarded for positively impacting a friendly validator’s ranking in the Contest, and ten points for negatively impacting an enemy validator. An additional one point would be assigned for each rank the validator moved in the desired direction. For example, if a friendly validator moved from 12th to 5th that epoch, the player would receive 10+7=17 points. A commonly employed strategy involved unstaking from enemy validators, thereby lowering the validator voting weight when setting the RGP. From Mysten Labs’ perspective, the purpose of Frenemies was to evaluate how benign or malicious staking, unstaking, and re-staking behavior could impact validator profitability.
Validator Contest: Results
We found that, across the testnet’s 32 epochs, the vast majority of validators were able to quote profitable RGPs. The graph below presents every validator’s profitability from most profitable to least profitable.
Only 4 out of 41 validators had more unprofitable epochs than profitable. Only 1 validator was net unprofitable across all 32 epochs.
We also saw validators improve over time. The majority of validators were unprofitable during epochs 1–6. After an initial learning curve, unprofitability dropped to between 0–34% of validators per epoch.
The table below shows a more granular analysis that illustrates the total count of validator’s scores per epoch.
There are some interesting findings:
- During Epochs #1–6, approximately 63% of validators were unprofitable
- Epoch #7 is the first time a majority of validators are profitable, scoring 2 points or greater
- From Epoch #8 onwards, the majority of validators were able to maintain perfect scores (note — the maximum number of validators that can achieve a perfect score is ⅔)
The graph above presents validators’ scores on an epoch-by-epoch basis and helps us visualize the time it took for validators to build their financial models, refine their assumptions, and gradually begin to improve their scores. It’s interesting to observe how quickly validators were able to avoid unprofitable scores and achieve profitable scores.
Once validators perfected their financial models, the SUI token price was the primary factor that influenced how validators voted on the RGP. All else equal, we expected to observe an inverse correlation between the SUI token price and the RGP. This is because as the SUI token price rises, validators can maintain the same revenue with a lower RGP. The SUI token price and RGPs throughout testnet are shown below:
From Epoch #8 onwards, we observed a moderate inverse correlation (-0.49) between SUI price and RGPs. The top 20 validators (50th percentile) showed a slightly stronger correlation (-0.51) while the top 10 validators (75th percentile) demonstrated a strong correlation (-0.69).
The graph below shows the inverse correlation between the SUI token price and RGP. We also show the top 10 validators, who consistently quoted lower RGPs than the rest of the validator set.
RGP quotes are on-chain objects visible to anyone, including other validators. Instead of building one’s own RGP model, validators are able to copy the RGPs set by others, and set a lower gas quote immediately before the epoch boundary. This technique lowers validator profitability and is unlikely to be executed in a similar fashion once Mainnet launches, however it was a clever tactic some validators exercised to score additional points while also bypassing the need to build one’s own model for testnet.
Although building a gas price model requires more involvement for validators, Sui lets the free market decide the RGP. Validators that do not effectively model gas prices themselves will be less able to make decisions that impact their own profitability. Those that don’t want to participate open room for a new operator that is willing to capture the incentives. Over time, Sui’s operator overhead will shrink as the protocol matures and validators build familiarity with the network. The manual gas quote update process will be automated so that it does not require daily maintenance.
Throughout the Contest, we saw strong improvements in validators’ understanding of network economics and their ability to effectively quote gas prices. A strong inverse correlation between the SUI token price and RGPs bodes well for Mainnet. The majority of validators progressed from being unprofitable to scoring the maximum number of points per epoch. Validators are more prepared for Mainnet than ever.
Tallying Rule
Another factor impacting validator rewards is their tallying score. The tallying rule is a unique characteristic of Sui — validators assign every other validator a score of 0 or 1 each epoch based on whether they believe the validator is “operating performantly.” A score of 1 represents permanent operations, and a score of 0 represents non-performant operations, according to that validator. If ⅔ of validators by stake assign a validator a score of 0 in a given epoch, that validator is slashed. This peer regulation mechanism further incentivizes validators to maintain good performance and honest behavior.
Validators were instructed to start participating in the tallying rule 2 weeks into testnet. Because of high-performance across the validator set, we were not able to analyze any significant tally rule activity. In fact, there was only a single assignment of 0 by a validator during all of Wave 2, which we suspect was done for experimentation purposes. This dynamic likely reflects Mainnet behavior where we anticipate scores of 0 to rarely be assigned. Like most other slashing mechanisms, slashing through the tallying rule is an important enforcement tool to have in theory, rather than a tool that is commonly used in practice. For Mainnet readiness, the most important thing is for validators to understand what the tallying rule is and how to use it.
Infrastructure and Stability
A network’s infrastructure complexity, stability, and real-world costs are important factors in determining its decentralization and sustainability.
Hardware Requirements
Sui is a high performance network that needs powerful hardware. For testnet, validators were instructed to use the following hardware setup:
- 10 core CPU / 20 core virtual CPU
- 128 GB RAM
- 1T SSD
A comparison between Sui’s requirements and the requirements of other major networks is shown below:
Sui’s hardware requirements are comparable to other high performance networks. Memory usage may go down over time as the network is optimized but we also expect storage usage to increase as the network’s state grows.
During Wave 2, a testnet inefficiency caused some validators to see massive spikes in their bandwidth. At one point, a validator saw as high as 1 GB/s egress — the rate at which data is exported from the validator. Egress costs vary widely by cloud provider and geographic location. While a few providers offer unlimited egress, others, like AWS, charge comparatively high fees. Some Sui validators running on AWS during this bug saw their AWS fees peak to over $1000/day.
To solve the high egress problem, the Mysten team recommended validators limit the maximum number of connections that validators would accept from other nodes. The team plans to limit connections at Mainnet to around 50 connections. For testnet, they recommended operators set it to 0 after the problem arose. Setting connections to 0 means that fullnodes would not be able to sync to the network unless they are whitelisted by a validator. To solve this, the Mysten team spun up their own full nodes to help 3rd party full nodes sync.
Beyond the bandwidth issue, the validator costs were not significant. With network improvements and cost efficient validator setups, the operating costs of running a Sui validator should not be prohibitive.
Stability
Relative to other testnets, Sui Testnet Wave 2 was relatively straightforward to operate. Although the network experienced a few more hotfixes than Testnet Wave 1, we remain impressed with the Mysten team’s ability to quickly identify and address bugs. Most bugs were patched in less than 6 hours from when an issue was identified. Few issues persisted after an initial attempted fix.
For new testnet networks, the number of bugs is less important than their significance. A bug that requires a fundamental change to the protocol architecture or a refactoring of the codebase can be orders of magnitude more consequential than a one-line typo. Fortunately, we did not see any issues arise that required major protocol changes. Sui is closer to Mainnet-readiness than ever before.
Performance
To our surprise, Wave 2 did not involve any load testing. Though we could see the network’s actual throughput (which topped at 67 TPS), we were not able to assess the maximum throughput.
One issue we faced when evaluating network performance was seeing differences in transaction timestamps depending on which validator we sourced the data from. After reviewing the source code, we learned that Sui transactions are subjectively timestamped according to when the node sees the transaction certificate for the first time. The lack of a global source of truth made performance analysis more difficult. The team has decided to implement global timestamping which will significantly improve uniformity and consistency across nodes.
According to the Mysten team, network optimizations have dropped shared object transaction latency from 8–10 seconds to 3–4 seconds. Although we are excited to assess the network’s latency ourselves, we left this for future work. We made the decision due to the complexity of collecting an accurate measurement and the expectation of additional latency improvements over the coming months.
The Next Wave and Beyond
Wave 2 helped validators better understand Sui’s economics. We saw improvement in validator profitability over time as node operators built out more sophisticated gas pricing models. The dominance of validators scoring between 2 and 3 by the end of testnet was an important signal of validator preparedness. Mainnet price modeling will require additional analysis, since validators will need to calculate their individual operating costs and the SUI token price will fluctuate within epochs. However, the speed at which validators built profitable, high-scoring models on testnet is promising. We hope to see a majority of validators maintain profitability within a few weeks of Mainnet launch.
Wave 2 network stability issues were consistently identified and patched quickly. Application developers continue to onboard onto testnet. Over the course of the next few weeks, the Mysten Labs team will analyze the Testnet’s data, refine Sui’s infrastructure, apply fixes to identified issues, and prepare for the next wave.
As we continue to monitor Sui’s testnets, we’ve gained a tremendous amount of insights into how the network will operate. For Testnet Wave 3, our goal is to answer any outstanding questions that went unanswered from Wave 1 and observe any changes from Wave 2’s performance. Some of our questions include:
- How will we see the tallying rule used for validator self-governance?
- Will tallying scores of 0 be assigned liberally for suboptimal performance or honest-but-controversial behavior (like large validators setting aggressively low RGPs), or will it only be used for clearly malicious or unsatisfactory behavior?
- How will the presence of MEV impact validator gas models?
- If validators extracting MEV can quote lower gas prices and attract higher yields, how will other validators react?
- What will be the difference, on average, between gas prices quoted by MEV-extracting and non MEV-extracting validators?
- How soon will automation be implemented into gas price modeling and quoting? Will any aspect need to be manually processed?
- Will RGP models need to be continuously improved to adjust to new protocol upgrades or validator set “metas”, or will validators be able to set-it-and-forget-it?
- How will minimum and recommended validator requirements change over time?
- What is the network’s throughput limit, and how does the network behave as it approaches or exceeds that limit?
- What is the latency for owned-object and shared-object transactions?
- If validator connections to full nodes are limited to 50 at Mainnet, how much does this hinder decentralization or impact RPC services?
- In equilibrium, what percentage of validators will copy other validators RGP quotes versus computing their own?
- How does network stability and performance vary between different validator sets?
- How does validator performance correlate with transaction volume?
- Will any acute changes in network behavior be experienced or observed during epoch changes?
- What does the validator storage use look like, graphed over time?
- Can we map the network’s storage cost to data sizes? What is the proportion between bytes and gas units?
Our team looks forward to gaining additional insights from Sui Testnet Wave 3 as we develop a deeper understanding of the network. If you have any questions regarding Sui’s testnet or if you would like to connect with our team, feel free to reach out.
