Celo Discord Validator Digest #5
Slashing, key rotation, rounding error, missed rewards and incorrect uptime scores, useful info and community tools
One of the troubles we have faced as a validator for Celo is keeping up with all the information that comes up in the Celo's Discord discussions. This is especially true for smaller validators whose portfolios include several networks. To help everyone stay in touch with what is going on the Celo validator scene and contribute to the validator and broader Celo community, we have decided to publish the Celo Discord Validator Digest. Here are the notes for the period of 18-24 May 2020.
Discussions
Slashing
There was some confusion about the "statue of limitations" for slashing penalties (i. e. the period during which an offender can be punished for a slashable violation) which led to a discussion between zviad | WOTrust | celovote.com and Peter [ChainLayer.io]:
@zviad | WOTrust | celovote.com: ... Slashing as far as I understand has no statute of limitations either, so even though things can't be slashed right now, as soon as contract is fixed anyone who has slashable violations in the past can be slashed.
@Peter [ChainLayer.io]: slashing has a 4 epoch "statute of limitation"
Here's the precompile call https://github.com/celo-org/celo-monorepo/blob/master/packages/protocol/contracts/common/UsingPrecompiles.sol#L209
Here's the time limit in the precompile https://github.com/celo-org/celo-blockchain/blob/master/core/vm/contracts.go#L1010
^^ that's where that's mentioned (I had that same question)
@zviad | WOTrust | celovote.com: Oh interesting. Yeah I didn't see any limits in the slashing contract code itself, so assumed there was no limit. But it makes sense to have some limit since otherwise it would require VM to keep access to full history forever.
Key rotation
Looks like existence of RG contacts on mainnet added a new mechanics to the network that some might be unaware of. In particular, if you used your validator signer key to register metadata, you should remember to re-register the latter after performing key rotation or you metadata may turn out to be invalid:
@james | censusworks: ... Whether you have to regenerate metadata after you perform a key rotation will depend on which signer you use to register the metadata. If you use your validator signer, then I believe you'll have to reregister after a rotation since that signer will be invalid.
...
if you're using your vote signer to register metadata then it doesn't matter.
@Peter [ChainLayer.io]: ... Will attestations break if you rotate your validator signer while having signed metadata with that signer? Because I'm fairly certain it didn't during the stake-off.
@nambrot | cLabs: You will need to update your metadata immediately indeed as the metadata no longer verifies.
Another point to note is that you don't need to have funds on your news signer unless you need to run affiliate/deaffiliate commands:
@spa | swiftstaking.com: If I want to rotate validation signer, I would need to use a part of the spare 1cgld on initial RG to fund it right? Or should I fund it from somewhere else?
@zviad | WOTrust | celovote.com: New signer doesn't really need to have any balance on it, since
releasegold:authorizecommand is run through beneficiary.You will only need balance on new signer if you want to preform
deaffiliate/affiliatecommands to change the group.
Rounding error, missed rewards and incorrect uptime scores
Last week, some of the validators noticed a new error:
@james | censusworks: not sure if I'm missing something here or one of my locked gold accounts is losing a small amount of gold:
total: 9999999999999999999994 (~10000 10^18)This is from an elected validator that started off with a full 10k locked gold.
Rounding error somewhere?
@timmoreton | cLabs: Yes there is a rounding issue internal to how activated locked gold is managed that may make things a few Wei off.
Many noted that this issue could potentially have further impact:
@james | censusworks: Ah, yeah no one will care about a few wei from a financial perspective but will this end up breaking some automated systems down the track?
E. g., I could imagine someone deregistering a validator and attempting to reregister, but failing to do so because their locked gold is now too small.
That's a fairly contrived example but maybe there are more realistic ones where rounding issues could break things.
@Rob | Polychain: It had made our alerting quite loud (and confusing!) and would break our accounting reconciliation.
Next day, it became obvious that the rounding error also affects validator and voter rewards and uptime scores:
@zviad | WOTrust | celovote.com: Hey all, I am having a really weird issue. One of my validators didn't get paid last epoch! ... I tried checking everything, but unless I am missing something very obvious I don't see how this could have happened. This is the validator:
0xAb98b3C66Fb538c90359191067a5eA3aF8e5739B. It was definitely elected last epoch, it is still elected. It also has perfect uptime, since uptime score is max possible right now.2 of my other validators got paid as regular, but that one got no cUSD for this epoch transfer. My group also received payment only for 2 validators and not for 3rd one.
...
Is there code that will refuse to pay epoch rewards if required gold drops below 10k?
@asa | cLabs: Yes there is!
...
Activating rewards on votes in some cases leads to a 1 wei rounding error.
Which, if it drops you under the requirement, will mean no validator rewards.
...
If the group had the rounding error, then voter rewards were affected, if not, then no.
@zviad | WOTrust | celovote.com: Oh wow that is bad. Because now in every score calculation we will show up that we didn't have perfect rewards. And it might show up like that for a long time. That is a big problem, even though we haven't had any issues.
...
Well it can have long lasting effect on "scores" that are built by other people that show voter rewards. Because a lot of websites, like "validators.celo.org", show rewards percentage based on past payouts. So if you fully miss one epoch, that will affect your "voter rewards" score for a long time.
@james | censusworks: To repeat what some folks mentioned above: this can have some serious impact on the long-term viability of validator groups, e. g., if I was a gold holder looking at Censusworks with a 48.7% voter rewards and 62.5% uptime I definitely wouldn't be delegating any gold to us. Given that these numbers are completely inaccurate can we get this page [https://celo.org/validators/explore] taken down asap or replaced?
This is quite a sensitive time-period with potential voters receiving their gold from Coinlist and a lot of competition to maintain validator slots.
The cLabs team reacted quickly and offered short-term and long-term solutions to the problem:
@pranay: ✅ quick announcement re: the epoch rewards issue several of you have reported: https://forum.celo.org/t/epoch-rewards-issue/498
TL;DR -- The cLabs team spent the day reviewing the issue, and we believe we've identified the cause of the issue reported. The good news is there is a short term workaround, which requires validators and groups to lock up a small amount of cGLD over 10k (eg. 10,000.1) to avoid hitting this bug - you should have some extra in the ReleaseGold contracts. The long term fix requires an upgrade to the Election.sol smart contract, which will take ~6 weeks due to developing the smart contract upgrade process with backwards compatibility and governance.
Please read the forum post for more details on why the rounding error emerges, as well as how we intend to address the concerns re: the validator explorer. Thanks to everyone who reported the issue, and quickly helped others! Please let us know if any other concerns/feedback.
To check if your validator/group was affected run the following commands and make sure that the total gold locked for each account is not lower than the requirement:
celocli lockedgold:show $CELO_VALIDATOR_GROUP_RG_ADDRESScelocli lockedgold:show $CELO_VALIDATOR_RG_ADDRESSYou can also check rewards for your validator/group by running (adjust the --epoch flag value accordingly):
celocli rewards:show --validator=$CELO_VALIDATOR_RG_ADDRESS --epochs=32celocli rewards:show --group=$CELO_VALIDATOR_GROUP_RG_ADDRESS --epochs=32Useful info
The new version of Celo's Ledger app in now available at the Ledger developer store:
@kobigurk: Celo Spender 1.0.3 has been released to the Ledger developer store!
Release notes:
No “Pending Ledger review” screen anymore! This will probably solve any freezes that occurred.
RC1 tokens transfers now show proper details - both cGLD and cUSD (when enabled) will show correct amounts and destination addresses. This is thanks to a dynamic token mechanism in the spender app.
Contract data enabled by default.
syncnode (George Bunea) shared a tip on how to set up an active-active HA setup. You can run two validators with the same signer but one of them has to have mining disabled (be careful not to double sign!):
Geth has the option to stop the miner. It’s a command in the geth console. And you can run in this way with same signer on two or more machines. Of course with the risk of double sign if things are not properly setup.
The commands in question are:
docker exec -it celo-validator geth attach
# inside the console
> miner.stop()and
docker exec -it celo-validator geth attach
# inside the console
> miner.start()Community
Adrian | moonlet.io shared a preview of an app their team was working on:
Our main goal is to offer anyone who holds cGLD the possibility to vote for any @Validators ✅ Group using Moonlet. As well to build the right tools for Validators in order to use Moonlet to collect votes; available on iOS, Android and Chrome.
Hopefully, this initiative is intrusive enough and will add value for the whole CELO community, both Validators and cGLD holders. However we are keen to have your feedback and see what do you think about it. Would it be valuable?
FEATURES:
✅ Vote now. Our non-custodial cryptocurrency wallet allows you to vote/stake easily and secure all your different crypto assets. See below our UI/UX proposal.
Coming soon✅Handshake Login. It just got easier to sync the mobile app accounts, wallets, history and settings with Moonlet extension.
✅Ledger Support. Ledger Nano S and X integration adds one more layer of security, the best level of protection developed so far.
✅Notifications Enabled. Moonlet notifications feature is a powerful tool to inform users with regards to rewards, outbound and inbound transactions.
✅ERC-20, ZRC-2 Ready. Moonlet is ready for Ethereum ERC-20 and Zilliqa ZRC-2 tokens.
✅Multi-account, Multi-wallet UX. You can easily switch between 5 accounts at any point, as well between any wallet you are going to create.
✅Native and Web Deep Links. Validators could place deep links on their webpages or social media announcements to trigger the voting page.
Coming soon
The app is available here.
Pretoria Research Lab's block map now supports several new features:
Stay at head option to always refresh when new data arrives
Favourite tags to stay at top of validator signer list
Sorting by validator/signer/signed/missed
Change map scale from 100 to 1000 signed blocks
Added a map for the Baklava network also
Coming soon:
Email subscription
Single block updates rather than data being 100 blocks behind
Dark mode
There are now new Discord channels about the Celo Improvement Proposals (CIPS): #cip-improvement-proposals and matters related to various validator explorers: #validator-explorer.
Like what we do? Support our validator group by voting for it!