Celo Discord Validator Digest #14
Governance slashing, foundation voting, attestation errors and sender ID pre-registration, bringing Ren to Celo, potential issue with Celo economics and reserve, and community updates.
One of the troubles we have faced as a validator for Celo is keeping up with all the information that comes up in the Celo's Discord discussions. This is especially true for smaller validators whose portfolios include several networks. To help everyone stay in touch with what is going on the Celo validator scene and contribute to the validator and broader Celo community, we have decided to publish the Celo Discord Validator Digest. Here are the notes for the period of 3-16 August 2020.
Discussions
Governance slashing
While downtime slashing is not yet working both in Baklava and mainnet, discussions of other options took place in the Baklava channel:
@victor | cLabs: ... It turns out that Governance slashing works a little different than Downtime or Double Signing slashing. Specifically, it only applies the CELO penalty, and does not directly remove the validator from their group. We could create a proposal that 1) Enabled Governance to directly invoke slashing functions on the Validators smart contract then 2) Execute a
forceDeaffiliate
call to remove the dead validator from their group.My suggestions would be to leave it for now, for two reasons. 1) It is not affecting network stability since it is only one node and 2) It's unclear that the same proposal would be a wise choice on Mainnet, and we want to keep Baklava looking as close to Mainnet as reasonable. Once the downtime slasher is upgraded, which is the first step to releasing the new Downtime slasher which was recently finalized, we could slash this dead validator, which would itself be an good test of the new slashing code.
Related: It looks like we now have 4 nodes that are offline.
@syncnode (George Bunea): I partially share your point of view, but let's not forget that in the current conditions we are in the situation that we have very limited actions to punish eventually bad actors which is not desirable on the long run.
@victor | cLabs: If there is a case where a bad actor needs to be removed entirely from the network, Governance slashing can be used to remove their locked funds on the network. This will also cause the validator to be unelected. I wasn't suggesting that here because my suggestion would be to leave their funds so that if they come back, they could get elected again. I could see an argument for completely removing funds validator accounts that are offline for long enough, but would prefer to go through the Downtime Slasher route, which is less aggressive.
@Thylacine | PretoriaResearchLab: I think we should get this bug fixed, deploy it to Baklava and use it as a true testnet and simulate what should happen on mainnet.
Governance slashing is interesting though. What happens if a validator is behaving within the protocol parameters (i.e. not slashable) but is still consistently performing malicious actions for example, transaction censoring or suspicious reordering / block stuffing? I'm not exactly sure how a block is actually constructed from the txPool in geth/PoS implementations so this might not actually makes sense, but I was just trying to think of an example of "bad behavior but allowed by the network".
@victor | cLabs: Ya, governance slashing is designed to be a flexible tool, for use as long as the community approves it, and in my mind it's reserved for fairly extreme cases, especially when actions through election voting is not a viable option to address the issue. If a validator found a way to game the Election system, or a whale with enough funds to unilaterally elect one or more validators starts to misbehave in a way that effects network stability, or user trust, that seems like the kind of case where Governance slashing will come into play.
Foundation voting
While the new votes from the Celo Foundation were activated, the election threshold kept on increasing prompting the community to suggest the expansion of the validator set:
@zviad | WOTrust | celovote.com: Welcome all new foundation votes validators. It is a little bit unfortunate though that 5 out of 6 couldn't get elected. For all five to actually get elected, 5th one would need ~130-150k extra votes in addition to foundation votes which is quite a lot.
We should probably still wait few more months but if election threshold remains this high, maybe it is time to increase total validator numbers by ~10% to 110.
(the downside of adding more validators is extra cUSD/CELO inflation from validator rewards, plus further reduction to rewards multiplier, but I think it still would make sense if these levels of election thresholds remain.)
@Henry | cLabs: Indeed, the election threshold is high, and cLabs is testing how the increase in validator set will affect throughput and block time. We'll keep the community updated on the progress.
Attestation errors and sender ID pre-registration
Some of the validators have seen the following error in Twilio: "Warning - 30018: Destination carrier requires sender ID pre-registration". Here's the explanation:
@zviad | WOTrust | celovote.com: Fyi, I have started to see few (2 total) messages from Twilio: "Warning - 30018: Destination carrier requires sender ID pre-registration".
There seems to be some extra registration required as described here: https://support.twilio.com/hc/en-us/articles/223133767-International-support-for-Alphanumeric-Sender-ID
...
After some minor research, looks like there are additional charges and things that need to be done to get this SenderId:
Russia: US$230 per month Czech Republic: US$6 per month Japan: US$0.0400 per SMS in addition to the standard price of US$0.0800 for a total of US$0.1200 per SMS Kenya: US$600 set up fee
Note that Twillio logs still shows stuff as "Delivered", but I am not sure if messages are actually being delivered (because these warning messages were also followed by missed attestations).
...
I think this is just a thing that certain countries/carriers enforce where SMS can be rejected if it doesn't have pre-registered SenderID, I am guessing as a way to combat SMS spam overall.
We probably need some sort of global attestation stats to see what the actual delivery percentages look like, since even during the #valora demo we had at least one or two people in the meeting who had issues getting the SMS messages. If overall attestation success rate is <80-90% that would probably be pretty bad.
@syncnode (George Bunea): Maybe there should be a way to manually address special situations?
And to be voted from case to case in a decentralized manner? In this way we will not let anyone out due to impossibility of sending or receiving an sms for an idiotic and totally out of our reach reason.
I am thinking maybe to create a committee like group that analyze exceptional cases and can manually validate a phone number. I don't know if it's doable now or what effort would be required to implement this, also I don't know who should budget this, it's just an opinion.
For sure, we will face special situations in the coming future. This is certain.
@zviad | WOTrust | celovote.com: I am not sure if anything manual will help since that seems difficult to scale. And response time is most important since registration flow needs to be as smooth and fast as possible. In general 2fa or auth token delivery flow with SMS is a well developed thing, its just a bit more complex than just setting up a simple Twilio account. Especially expecting all 100 validators to have this setup well seems unlikely.
There is a reason why Twilio has a separate product just for 2fa delivery: https://www.twilio.com/authy/features/otp
Their service internally uses multiple phone numbers, pre-registered senderIds and bunch of other things to increase delivery world wide. There is no way all validators will be able to replicate that kind of setup.
...
Actually after further looking at the contract and events, from users perspective as long as one of the attestation requests succeeds, verification flow is complete (I think, someone from cLabs can confirm)...:
DAYS: 1 -- issued: 14, fullfilled: 12, missed: 14.29% DAYS: 7 -- issued: 83, fullfilled: 74, missed: 10.84% DAYS: 30 -- issued: 133, fullfilled: 115, missed: 13.53% DAYS: 60 -- issued: 215, fullfilled: 191, missed: 11.16%
Number of unique
phone number + account
makes more sense this way too.
Meanwhile, the cLabs informed the validators that the team is working on improving the quality of the attestation service and conducting its rigorous testing:
@cmcewen | cLabs: ... We have been really focused on attestation quality and are working through these issues on both the wallet side and the attestation service side. obviously the real world never stacks up quite like testing, and we are doing some pretty rigorous testing with lots of different carriers in different geographies. I think we will have some recommendations to share soon, but this is definitely top of mind for the cLabs team and we understand the difficulty in getting the completion percentage up especially in the early days
Bringing Ren to Celo
Last month, alchemydc | zanshindojo mentioned a discussion about adding support for a Ren Protocol bridge on Celo. The discussion have seen some continuation last week:
@zviad | WOTrust | celovote.com: Not sure how far RenVM discussions went but I would throw my plus one support for that direction. There are still few things to work out with Ren`s model (especially since most of the code is not yet open), but overall that direction can be a big plus for Celo platform. I would say it is more important to bring renBTC and renETH on Celo rather than to bring renCELO to other networks. If Celo's reserve rebalancing starts doing, renBTC <-> CELO and renETH <-> CELO transactions on Celo chain itself, that will naturally bring liquidity to BTC and ETH trading on Celo chain.
Having that natural demand for liquidity opens up all the opportunities to develop exchanges and all other instruments on Celo network directly, so it can only help grow the platform.
To expand on this a bit more, I think this is one area where reinventing the wheel is probably not necessary and can be detrimental overall to growth of the platform. Of course it is possible to create yet another coin wrapping technology (so we have celBTC, celETH-s :D) where Celo network is the center of the universe itself, but that will probably only make adoption more difficult, and it would have only marginal benefits if any.
@alchemydc | zanshindojo: Ren seems to be gaining significant traction (despite widespread concerns about it being a black box with key components still closed source). Last I checked, Ren was #2, behind only wBTC (Kyber) for Bitcoin on Ethereum. @marek and I had a good chat w/ Michael (COO @ Ren) last week and are working on adding Celo to Ren's Multichain project.
I also see today that the future of Celo interoperability got a big boost, with @prestwich and the Summa team joining Celo: https://medium.com/summa-technology/this-is-goodbye-452e44db81ac
@zviad | WOTrust | celovote.com: ... Also for Celo reserve specifically, using platform like Ren shouldn't pose a significant risk since only ~5% (maybe at most 10%) of the assets would need to be held in ren* tokens for trading, rest of the reserve can be held on respective chains anyways.
Celo economics and reserve potential issues
zviad | WOTrust | celovote.com raised an interesting question about the Celo Reserve and its susceptibility to price manipulations:
As protocol is designed, Celo's onchain exchange reflects prices from outside markets very quickly. From oracles + arbitrage this happens consistently and efficiently and data shows that it is working well. However, I am not sure if it is working as efficiently in the other direction. Is there any mechanism that would lead to market re-adjusting the prices based on on-chain trading? As far as I can tell from last few days of trading that may not be happening too well. While arbitrage can happen in other direction too, because exchange updates prices every 5 minutes, you can "silently" trade quite a lot of volume without creating any arb opportunities. Also as far as I can tell market makers don't seem to pay any attention to the on-chain trades right now (cc: @Roman | cLabs ).
The hypothetical issue that this can create is that, there can potentially be an opportunity for a pump and dump manipulation. * Adversary could buy ~1mln dollars worth of CELO on central exchanges over a day or two, causing the price to raise quite a bit. Since price propagation in this direction is efficient, on chain rates will go up too. * Adversary could then slowly sell that CELO on-chain. Because there isn't as efficient feedback cycle to central exchanges in that direction, they could potentially sell at least few hundred thousand cUSD (if not more) worth of CELO per day without affecting central exchange prices. Hence selling it all back at the top of the price that they managed to pump the price to.
If such a manipulation is possible, that would hurt the reserve the most (maybe also market makers that operate on central exchanges, but probably not). In this case reserve would be acting as a weak market maker that you can take advantage of because it is unable to adjust prices downward properly.
To expand on this, fundamental issue that i see in this setup is that oracles don't take on-chain trades/rate in determining the current "price" of CELO. If there is a lot of trading volume that happens on-chain (like it has been on Aug 11,12,13), then that somehow has to make into price determination too, most likely.
@trevor | cLabs: That's a really good point, would like to hear @Roman | cLabs's take on this. I'm a little wary of having the oracle depend on on-chain trades because of the added complexity & (haven't really thought it through) what feels like a higher susceptibility to manipulation. But then again it is an exchange. The 5 minute bucket resets would make price aggregation from on chain trades nontrivial. I don't know exactly how market makers fill order books, but I wonder if we made on-chain exchange data easily consumable then market makers on CEXs would have an easier time adjusting.
@zviad | WOTrust | celovote.com: Yes definitely. Integrating on-chain price into oracle requires quite a bit of thought and care, since short term on-chain price can be manipulated fairly easily. So if some sort of price information comes from on-chain trades it would have to be over a longer time period. Or it would have to be calculated differently than just current exchange rate.
An option could be to calculate a spread from on chain trades, that would make it more similar to how pricing works on central exchanges. So instead of both buy and sell prices going in the same direction as it works in uniswap model, you would actually have ever increasing spread if trades are only going in one direction.
That would make it similar to what happens on central exchanges if you have trades only in one direction in a short time period.
If you have a formula that actually calculates "spread" (i.e. separate buy and sell prices) instead of just one price from on chain trades, now you can apply all other safe guards and logic that you have in oracles regarding spreads and choosing the middle price and all that.
Another option would be to incorporate on-chain exchange rate into oracle pricing information if it has only changed by <X% since beginning of the 5min bucket. This would be sort of equivalent to the "spread safeguard" that oracles have for central exchanges.
@Roman | cLabs: It’s a super interesting argument @zviad | WOTrust | celovote.com and I agree that the price feedback is less direct in the on-chain-to-off-chain direction than the other way around. I definitely want to give this more thought but here are some first thoughts already: I am not fully convinced yet that a trader is likely to make a profit by carrying out the described pump strategy. This is because not only the trader who invested a lot of capital to pump the CELO price but also all other CELO holders should have an incentive to sell CELO for cUSD (or USD for that matter) after the pump. The other CELO holders could basically free-ride the pump investment of the trader and take a large share of his profits by selling CELO on-chain after each bucket update and/or buy selling CELO on centralized exchanges. The pump trader therefore takes a pretty big risk that the CELO/USD price returns to its fundamental value eventually before he can sell the CELO tokens to the reserve at an average price that is larger than the average price he paid during the pump. I guess the likelihood of this pump strategy being profitable therefore depends on parameters like market liquidity, on-chain bucket sizes, the CELO holders’ ability to spot a pumped price, etc. Definitely feels interesting to think about how additional mechanisms like the ones you described could help with further reducing the likelihood of profitable pump trading!
@zviad | WOTrust | celovote.com: Agreed that in practice, such an attack definitely carries a risk. Also I am no expert in market manipulation and don't have any experience in it so it is hard to say how such things would actually transpire in practice if actually attempted. :) But on the other hand, I do think that the stats from last 2-3 days shows that something isn't 100% right with the on-chain/oracle pricing model. cUSD/CELO on chain trades were almost 2x more volume (and most of it in CELO -> cUSD direction) compared to USD/CELO trades from Bittrex + OkCoin, however effect of cUSD/CELO on chain trades seems to be very minimal on the overall price itself. To put it another way, if same trades that were done on-chain, had happened on Bittrex or OkCoin my suspicion is that current CELO price would quite a bit different (of course this is very difficult to verify but logically it makes sense to me at least).
Tbf, even if there is an opportunity for some sort of manipulation right now, the actual severity would still be very low because reserve is so over collateralized. And due to trading volume limits there is a ceiling on profits that one could get per day even in best case scenario. But probably still useful to think about this issue and tackle it earlier rather than later since such changes to oracles will only become more complex in future.
@syncnode (George Bunea): What you described makes sense only in the scenario where there would be only one player in the market, basically the one that pumps the token price and takes the profit, but in reality the market works like a living organism, it adapts and has a lot of players and market makers that are following closely any form of arbitrage opportunities and tend to balance any forms of differences. Also we have arbitrage player in the game already that take into consideration both CEX prices and on chain prices and I doubt we could see that scenario live in action.
@zviad | WOTrust | celovote.com: There are indeed multiple players in the market. One of them being the Reserve. Reserve is basically a predictable market maker on chain. Because it is predictable, if oracle pricing has issues, it can be exploited. On-chain / CEX arbitrage can't help in this case, since oracle pricing doesn't take on-chain trades into account, and it is possible to trade decent volume without triggering arbitrage thresholds. That is sort of the whole point that i am describing. If oracle pricing starts taking on-chain trades/volume/price into account, after that arbitrage will kick in and move price more properly across all markets.
@syncnode (George Bunea): I entirely support your opinion , especially that I raised a similar point weeks ago and I asked for decentralized oracle approach, something similar with what Terra is running now. We could have oracle voters from the community (validators or other professional service providers) that can vote in decentralized manner for the prices based on multiple data sources. This would bring stability to a whole new level.
Community
winslyn | Staking Fund presented a very cool governance dashboard at https://celo.stake.id/
Validators now have early access to the Celo's mobile-first payments app called Valora.
Henry | cLabs is unfortunately leaving cLabs as his internship is coming to an end on August 21. He was the glue that has kept this community stuck together. Best of luck, mate!
Like what we do? Support our validator group by voting for it!