Celo Discord Validator Digest #23
Increase in attestation requests, useful info and community updates
One of the troubles we have faced as a validator for Celo is keeping up with all the information that comes up in the Celo's Discord discussions. This is especially true for smaller validators whose portfolios include several networks. To help everyone stay in touch with what is going on the Celo validator scene and contribute to the validator and broader Celo community, we have decided to publish the Celo Discord Validator Digest. It took a while to release this once due to the holiday break and not much happening in the validator channels. Here are the notes for the period of 7 December 2020 - 17 January 2021.
Discussions
Increase in attestation requests
The number of attestation requests has seen a rapid growth since early January. Here are some insights into what has been going on and issues associated with the increase:
@Ben | Staked: I'm occasionally seeing 422 errors from the attestation service like so:
res: {"statusCode":422,"header":"HTTP/1.1 422 Unprocessable EntityX-Powered-By: ExpressX-Request-Id: bd82941c0d8598d6d48a1b7169106549X-RateLimit-Limit: 100X-RateLimit-Remaining: 98Date: Thu, 14 Jan 2021 16:34:37 GMTX-RateLimit-Reset: 1610642097Content-Type: application/json; charset=utf-8Content-Length: 54ETag: W/"36-65s3OCf0L7Kci9Z8ladHvYo2aBs"Connection: keep-alive"}
Everything seems to be okay with the attestation service, but it keeps tossing this error and dropping attestations. Any advice on troubleshooting?
@Cody | cLabs: Thanks for flagging. There's a huge spike in SMS requests (and 422 errors) from Brazil (+55). Investigating now..
The increase seems to be caused by legitimate usage of @impactMarket: Celo Camp . If you go to their site (https://impactmarket.com/) you can see an exponential increase in beneficiaries starting on Jan 11th.
@Bernardo | impactMarket: Hold tight. 6k now, 10k+ next week. We are going to be fine, you are not the only ones getting errors.
@Cody | cLabs: I chatted with @aslawson | cLabs and she believes it's likely that the
422 Unprocessable Entity
is an error returned by Twilio when a property is malformed (potentially a phone number). Can you see any indication of this in your logs?@zviad | WOTrust | celovote.com: speaking of increased attestations, I just verified that fee withdraws now work due to this fix: https://github.com/celo-org/celo-monorepo/commit/17c47cf494b072fc96cad2e530c8650292f27444
However, it looks like Fee that is paid to attestation nodes is quite a bit less compared to what Twillio charges.
From my cursory glance it looks like accrued fees over both of my validators is ~2x less compared to the total charges on my Twillio account.
Total amounts are still very small, total accrued fees: ~23 cUSD, total fees paid in Twillio ~40$, but if this 100x-es it could become a problem.
I am guessing this is because of incomplete attestations.
Twillio SMS costs ~5 cents, I think attestation fee is also 5 cents, but we only get paid if attestation is completed. (Pus taxes and some other stuff also gets added on to the Twillio bill.)
Useful info
If you have encountered issues with the Ledger wallet on the latest release of celocli, here is the reason:
@kp: i've also been encountering the
ERR_INDEX_OUT_OF_RANGE
error with the globally-installed celocli (versions 0.0.60 and 1.0.0-beta) when running the following command:celocli releasegold:authorize --contract $CONTRACT_ADDRESS --role vote --signature $SIGNATURE --signer $SIGNER_ADDRESS --ledgerCustomAddresses '[0]' --useLedger
@Patrick|Validator.Capital|Moola: This is probably the same problem w/ the Ledger package that others have seen. You can downgrade to a previous version and should be able to submit the transaction. Here are steps:
@daluxx | StakesStone: I found the solution. My installation always used the global ledgerhq package, you can use the downgraded version with "yarn run". So this would be the temporary solution:
1) First downgrade ledgerhq package..: You need to downgrade the whole ledgerhq package (don't know where exactly the error comes from). I did the following if I remember correctly:
1)
mkdir celo && cd celo
2)
yarn add @celo/celocli
3) add "resolutions" to the package.json file
4)
yarn install
here's my package.json
{ "dependencies": { "@celo/celocli": "0.0.58" }, "resolutions": { "@ledgerhq/hw-transport-node-hid": "5.34.0", "@ledgerhq/cryptoassets": "5.34.0", "@ledgerhq/devices": "5.34.0", "@ledgerhq/errors": "5.34.0", "@ledgerhq/hw-app-eth": "5.34.0", "@ledgerhq/hw-transport-node-hid-noevents": "5.34.0", "@ledgerhq/hw-transport": "5.34.0", "@ledgerhq/logs": "5.34.0" } }
2) Then use celocli with "yarn run" in the local folder (then celocli does not use the global ledgerhq package, but the recently installed downgraded package): e.g.
yarn run celocli releasegold:transfer-dollars .....
Good advice from zviad | WOTrust | celovote.com considering the increase in attestation requests:
Woah, huge uptick in Attestations and account registrations. Remember everyone to make sure you have auto re-charging enabled for your Twillio + Nexmo accounts.
CIP 23 introducing Attestation Service operation incentives via slashing is open for comments and discussion:
@Cody | cLabs: CIP-23 is up and ready for review! CIP 32 proposes introducing Attestation Service operation incentives via slashing. This is the first step toward a more discerning incentive system. As such, it's focused on a min uptime bar (one attestation over 11 days) and will not increase validator rewards. Future proposals will likely support more granular uptime measures and the potential for operators to earn additional reward.
Discussions: https://github.com/celo-org/celo-proposals/issues/144
A note from Or | cLabs and james | censusworks regarding Baklava:
For everyone else, if your node isn't connected to
0хFF235Fb9F8B5f1303455712d84Ac6b2Ff2D90A35
, that's normal and to be expected since it's not with Churrito. However, if there are other validators it can't connect to, that likely indicates a problem. Here is some javascript that you can run in the geth console to output a list of validators it isn't connected to:valData = istanbul.valEnodeTableInfo; peersData = admin.peers; validators = []; idsToAddresses = {};• Object.keys(valData).forEach(function (addr) {id = valData[addr].enode.split("@")[0]; validators.push(id); idsToAddresses[id] = addr});peers = {}; peersData.forEach(function (peer) {peers[peer.enode.split("@")[0]] = true});validators.forEach(function (val) {if (!peers[val]) {console.log(idsToAddresses[val])}})
These addresses can then be looked up at https://baklava-blockscout.celo-testnet.org/ and under the Celo Info tab it should have some info about who that is. If you see your node isn't connected to several others, double check that your proxy (or your validator if you're not using a proxy) is not using the
--nodiscover
flag. For context, we recommend--nodiscover
for validators that are behind a proxy, because the proxy does the discovery in that case, not the validator. But the proxy shouldn't have that flag, and if you don't have a proxy then your validator shouldn't either since it doesn't have a proxy to do the discovery instead of it. If you check that and your node still shows problems, we'd like to debug it further with you by seeing which nodes it can't connect to and whether there is useful info in the logs about why.@james | censusworks: Also note that it should be run on the externally facing node, i.e., the proxy or validator if you don't have one
Community
cLabs is collecting feedback om how to improve the Celo Block Explorer:
@jb | cLabs: Hi Celo Community, in the spirit of improving the tools that are critical to everyone interacting with the Celo Blockchain, we created a survey to collect feedback on your experience with the Celo Block Explorer. If you have ever used explorer.celo.org, please take 5 minutes to complete this survey: https://forms.gle/gkC2xFbGEj2p1c5y7 Also feel free to DM me if you have more specific feedback you’d like to share in person. Happy to jump on a call!
The final validator round up for 2020 by zviad | WOTrust | celovote.com is out: https://wotrust.substack.com/p/celo-validator-roundup-december-2020
Attestation map at cauldron.pretoriaresearchlab.io got a little upgrade:
@Thylacine | PretoriaResearchLab: Folks, I've updated the attestation map viewer to only show the last N attestations, due to everyone having tons of attestations now and I'm running out of space to smush into one page. Later on I'll split it by epoch or block range. For now, it's just filtered for viewability.
Like what we do? Support our validator group by voting for it!