Is it possible to make a large (> 500 validators) and robust (liveness and safety up to f < n/3 byzantine faults) blockchain snappy (~1s finality)?

Over the last few months, @stephenbuttolph and I explored this question.

After many rounds of iteration (i.e. Stephen poking holes in my ideas), we arrived at Vena.

Vena drives finality at network speed, provides strong liveness guarantees, and natively outputs confirmation artifacts (aggregate signature for each block) over large validator sets.

This construction employs a form of tunable aggregation committee broadcast that when successful results in all participants, including the leader, sending less than n messages in a given round.

In the uncommon case that broadcast fails (robustness is parameterizable), Vena falls back to a single, all-to-all broadcast that guarantees eventual progress in the presence of f < n/3 static byzantine faults.

Vena's capacity to recover from "uncooperative" aggregation committee assignments enables aggressive tuning of the broadcast parameterization when the validator set is predominantly honest and online (reducing unneeded message broadcast) and pessimistic tuning when it is not.

If things deteriorate, the all-to-all broadcast fallback ensures safety and liveness will be maintained (as long as there are f < n/3 byzantine faults).

Vena even proposes an out-of-consensus mechanism for participants to dynamically tune robustness, if this occurs.

Optimizing for the "happy path" and falling back to a more expensive mechanism was inspired by Frosty (strong Snowman liveness).

Unlike Frosty, Vena does not require voting to enter an explicit "recovery epoch" or running separate consensus in it.

https://arxiv.org/abs/2404.14250

As evidenced by the abundance of interesting techniques, the deployment of consensus protocols on public blockchains in byzantine environments is rife with complex tradeoffs between:

* communication complexity
* confirmation latency
* fault tolerance

Vena provides an early exploration into an interesting compromise between these tradeoffs that suits large blockchains.

You can read about Vena in detail here:
https://hackmd.io/@patrickogrady/vena

If you have any suggestions for techniques to improve Vena's construction or wish to contribute to formal proofs of correctness, please reach out! 🙏 

You can read the unrolled version of this thread here: https://typefully.com/_patrickogrady/tCwiHwu

Designing a Snappy and Robust Blockchain: The Journey to Vena

🛠️ 🛠️ #Avalanche Durango 7 is out: https://github.com/ava-labs/avalanchego/releases/tag/v1.11.7 🛠️ 🛠️

This version (v1.11.7) is backwards compatible to v1.11.0. It is optional but recommended for all operators.

🔍 Focus: State Sync Robustness + Reduce DB Compaction Rate + Prepare for Metrics Overhaul

✅ Compatibility Check: VM Interface (v35) ✅ 

Durango 7 DOES NOT modify the VM interface. If you are running a Custom VM on v1.11.3-v1.11.6 (like Subnet-EVM@v0.6.5), it will work with v1.11.7.

https://github.com/ava-labs/subnet-evm/releases/tag/v0.6.5

State Sync Robustness

Durango 7 fixes a C-Chain State Sync panic bug (https://github.com/ava-labs/coreth/pull/568) and incorrect ETA estimates (https://github.com/ava-labs/coreth/pull/565).

Thanks to @darioush0 and @jceyonur for implementing these improvements 🙏.

Reduce DB Compaction Rate

Durango 6 introduced a bootstrapping regression that was ultimately caused by compacting the database too frequently.

@stephenbuttolph explained this issue in detail and implemented a fix here: https://github.com/ava-labs/avalanchego/pull/3065

Prepare for Metrics Overhaul

To prepare for the upcoming metrics overhaul (https://github.com/ava-labs/avalanchego/pull/3053), a number of metrics cleanup PRs were included in Durango 7.

The full metrics overhaul is expected to be included in Durango 8.

https://x.com/_patrickogrady/status/1793675907733111107

You can read the unrolled version of this thread here: https://typefully.com/_patrickogrady/mutPbmo

Durango.7 - Metrics Overhaul Preparation

🛠️ 🛠️ #Avalanche Durango 6 is out: https://github.com/ava-labs/avalanchego/releases/tag/v1.11.6 🛠️ 🛠️

This version (v1.11.6) is backwards compatible to v1.11.0. It is optional but recommended for all operators.

🔍 Focus: secp256r1 Staking Keys + Improve C-Chain State Sync Reliability + Fix Connection Tracking

🚧 Attn Node Operators: Metrics Refresh 🚧 

Durango 6 overhauls the format of many of AvalancheGo's exported metrics. This new format is easier to work with when creating dashboards and cheaper for hosted metrics platforms to ingest.

This work will continue in Durango 7.

✅ Compatibility Check: VM Interface (v35) ✅ 

Durango 6 DOES NOT modify the VM interface. If you are running a Custom VM on v1.11.3-v1.11.5 (like Subnet-EVM@v0.6.3), it will work with v1.11.6.

https://github.com/ava-labs/subnet-evm/releases/tag/v0.6.3

secp256r1 Staking Keys

Durango 6 defaults to using secp256r1 (P-256) staking keys instead of RSA 4096. This change dramatically reduces the size of the staking key, the staking key signature, and the time required to generate, sign, and verify it.

https://github.com/ava-labs/avalanchego/pull/3025

Improve C-Chain State Sync Reliability

Durango 6 fixes 2 regressions in C-Chain State Sync. The first regression could cause degraded performance after a successful state sync and the second could cause state sync to stall.

https://github.com/ava-labs/coreth/pull/557
https://github.com/ava-labs/coreth/pull/559

Fix Connection Tracking

Durango 6 fixes connection tracking on the P-Chain and C-Chain. This restores transaction pull gossip functionality on those chains (reducing p95+ transaction confirmation latency).

https://github.com/ava-labs/avalanchego/pull/2981
https://github.com/ava-labs/coreth/pull/543

ICYMI Frosty is OUT:

https://twitter.com/andrewlewispye/status/1785277538803044429?s=46

You can read the unrolled version of this thread here: https://typefully.com/_patrickogrady/ht0QCyh

Avalanche Durango 6 Release: secp256r1 Staking Keys

🛠️ 🛠️ #Avalanche Durango 5 is out: https://github.com/ava-labs/avalanchego/releases/tag/v1.11.5 🛠️ 🛠️

This version (v1.11.5) is backwards compatible to v1.11.0. It is optional but strongly recommended for all operators.

🔍 Focus: X/P/C Bootstrapping Optimization + Beta Consolidation + Consensus Engine Fixes

✅ Compatibility Check: VM Interface (v35) ✅ 

Durango 5 DOES NOT modify the VM interface. If you are running a Custom VM on v1.11.3-v1.11.4 (like Subnet-EVM@v0.6.3), it will work with v1.11.5.

https://github.com/ava-labs/subnet-evm/releases/tag/v0.6.3

X/P/C Bootstrapping Optimization

Durango 5 now fetches blocks in parallel for the Primary Network using an Interval Tree: https://github.com/ava-labs/avalanchego/pull/2752

Coupled with an improved peer selection algorithm for fetching, this reduces fetch time by 80%: https://github.com/ava-labs/avalanchego/pull/2946

Beta Consolidation

To prepare for the implementation of Error-Driven Snowflake (as outlined in Frosty), @AaronBuchwald consolidated VirtuousBeta and RogueBeta behind a singular Beta in the consensus engine.

https://github.com/ava-labs/avalanchego/pull/2949

Consensus Fixes

During a consensus refactor, @stephenbuttolph discovered a bug in the handling of votes (that could lead to some to be incorrectly dropped) and a bug in the handling of block issuance (which could lead to blocking on an already decided block).

Consensus Fixes (cont.)

The fix for the first issue (vote handling) is to ensure we properly track the in-degree of dependencies: https://github.com/ava-labs/avalanchego/pull/2965

The fix for the second issue (block issuance) is to ensure we abandon decided blocks: https://github.com/ava-labs/avalanchego/pull/2968

ICYMI (Bonus): @stephenbuttolph, @AaronBuchwald, and I joined @Lightspeedpodhq to discuss all the architectural changes recently proposed for Avalanche: 

https://twitter.com/Lightspeedpodhq/status/1785350486800969973

You can read the unrolled version of this thread here: https://typefully.com/_patrickogrady/isPsVCL

Avalanche Durango 5 Release: Bootstrapping Optimizations

🛠️ 🛠️ #Avalanche Durango 4 is out: https://github.com/ava-labs/avalanchego/releases/tag/v1.11.4 🛠️ 🛠️

This version (v1.11.4) is backwards compatible to v1.11.0. It is optional but recommended for all node operators.

🔍 Focus: P-Chain Block Cleanup + MerkleDB Memory Optimizations + Interval Tree Bootstrapping

✅ Compatibility Check: VM Interface (v35) ✅ 

Durango 4 does NOT modify the VM interface. If you are running a Custom VM on v1.11.3 (like Subnet-EVM@v0.6.3), it will work with v1.11.4.

https://github.com/ava-labs/subnet-evm/releases/tag/v0.6.3

🚧 P-Chain Block Cleanup 🚧

During startup, operators will see elevated disk IO for ~15 minutes as P-Chain blocks are rewritten to disk without their accompanying consensus status.

https://github.com/ava-labs/avalanchego/pull/2869

P-Chain Block Cleanup (cont.)

Because AvalancheGo only writes accepted blocks to disk (as of v1.10.x), there is no need to store the distinction between different types of persisted blocks.

MerkleDB Memory Optimizations

Over the past 2 weeks, ~24 MerkleDB memory optimization PRs have been merged to main (massive shoutout to @stephenbuttolph for driving this).

MerkleDB no longer performs any memory allocations during hashing 👀 

https://github.com/ava-labs/avalanchego/pull/2902

Interval Tree Bootstrapping

As part of an ongoing effort to reduce Avalanche Mainnet sync times, the legacy jobs queue was replaced with an interval tree that performs much less disk IO during bootstrapping (which is faster 😉).

https://github.com/ava-labs/avalanchego/pull/2756

ICYMI there are a TON of interesting ACP convos rn:
* https://github.com/avalanche-foundation/ACPs/pull/73 ( @AaronBuchwald )
* https://github.com/avalanche-foundation/ACPs/discussions/68 ( @AaronBuchwald )
* https://github.com/avalanche-foundation/ACPs/pull/75 ( @joshuagkim )
* https://github.com/avalanche-foundation/ACPs/discussions/71 ( @Michael_Kaplan1 )
* https://github.com/avalanche-foundation/ACPs/discussions/69 ( @Abenegia )

You can read the unrolled version of this thread here: https://typefully.com/_patrickogrady/UKwjrLf

Avalanche Durango 4: Interval Tree Bootstrapping

🛠️ 🛠️ #Avalanche Durango 3 is out: https://github.com/ava-labs/avalanchego/releases/tag/v1.11.3 🛠️ 🛠️

This version (v1.11.3) is backwards compatible to v1.11.0. It is optional but strongly recommended for all node operators.

🔍 Focus: Post-Durango Cleanup + Push Gossip Refactor (Phase 2) + BLS on Local Networks

🚧 Compatibility Check: VM Interface (v35) 🚧

This release introduces VM Interface v35 to support changes to the AppSender Interface (AppGossip).

If you are running a Subnet, make sure to upgrade your Custom VM to v35 (subnet-evm@v0.6.3) before running it with v1.11.3.

Post-Durango Cleanup

Now that Durango is activated, deprecated code from < Durango can now be removed. Here are a few examples:

* Remove Put/Peerlist Gossip
* Remove GZip Compressor
* Remove Pre-P2P SDK Message Handling (X/P)
* Enforce PreferredIDAtHeight in Chits Messages

Post-Durango Cleanup (cont.)

Non-Validators will most likely notice the impact of "Remove Put/Peerlist Gossip", which should reduce their p2p bandwidth by ~30%.

These mechanisms were rebuilt with more efficient Bloom Filter Pull-Based algorithms in v1.10.x.

Push Gossip Refactor (Phase 2)

Now that v1.11.2 is widely adopted, v1.11.3 can safely increase the number of peers that receive new transactions.

This change will slightly increase RPC bandwidth but will decrease E2E Time-to-Finality (TTF) for users.

https://github.com/ava-labs/avalanchego/pull/2836

BLS Keys on Local Networks

Thanks to @Nutymoon, Validators on Local Networks now start with BLS keys (instead of requiring registration after network start). This makes it much easier to play with Teleporter (which requires Validators with BLS keys).

https://github.com/ava-labs/avalanchego/pull/2794

[Bonus] BLS Adoption Rate 🚀 

Following the activation of ACP-62 (https://github.com/avalanche-foundation/ACPs/blob/main/ACPs/62-disable-addvalidatortx-and-adddelegatortx.md), 47+% of Avalanche Validators have now a registered BLS key!

Thanks to @snowpeerio for tracking this useful stat!

You can read the unrolled version of this thread here: https://typefully.com/_patrickogrady/nCF2kfK

Avalanche Durango 3 Release: Legacy Gossip Removal

🛠️ 🛠️ #Avalanche Durango 2 is out: https://github.com/ava-labs/avalanchego/releases/tag/v1.11.2 🛠️ 🛠️

This version (v1.11.2) is backwards compatible to v1.11.0. It is optional but strongly recommended for all node operators.

🔍 Focus: Redesign Push Gossip on the X/P/C-Chain

🚧 Compatibility Check: VM Interface (v34) 🚧

Durango 2 adds support for message-specific gossip parameters to the VM interface (used in the push gossip redesign).

If you run a Subnet, you must update your VM to v34 for v1.11.2 compatibility.

Last week, there was a "Block Finalization Stall" on the Avalanche Mainnet where a bug led to each node saturating their stake-weighted bandwidth allocation on peers with useless transaction gossip (preventing consensus messages from being processed).

https://status.avax.network/incidents/qmx0s7zk6gkm

Following the recovery of block finalization, the Platform Group at @AvaLabs shipped a full redesign of the gossip mechanism used in the P2P SDK (used in the X/P/C-Chain).

https://github.com/ava-labs/avalanchego/pull/2772

This new mechanism relies more heavily on transaction issuers to propagate transactions they care about through the network instead of validators. It takes inspiration from the optimization work that went into the Avalanche HyperSDK and Vryx.

https://hackmd.io/@patrickogrady/rys8mdl5p

Note to Operators:

Validators should see a reduction in AppGossip egress bandwidth and anyone running a Public API will see an increase in AppGossip egress bandwidth.

If you want to learn more about these changes (outside of what is detailed in the related PRs), stop by the next Avalanche Developer Community Call (which should happen next week).

Last but certainly not least, make sure your nodes are updated for the upcoming Durango Upgrade on 3/6. To avoid downtime (and potential loss of staking rewards), make sure to upgrade **BEFORE** the activation time:

https://twitter.com/_patrickogrady/status/1760421358612390132?s=20

You can read the unrolled version of this thread here: https://typefully.com/_patrickogrady/G1Q8FVB

Avalanche Durango 2 Release: Redesign Push Gossip

 🌀 The Avalanche Interoperability Era Starts at 11 AM ET on 3/6 🌀

Today, the production code (AvalancheGo@v1.11.0) for the proposed Durango Upgrade was published. If sufficient stake supports Durango, it will activate on Mainnet at 11 AM ET on Wednesday, March 6th, 2024.

Durango contains ACPs that are not compatible with < v1.11.0 (Cortina).

If Durango activates and you do not upgrade your node **before** the activation time, your node will be unable to process new blocks and will be marked as offline by other nodes (impacting uptime).

🚧 Compatibility Check: VM Interface (v33) 🚧

Durango introduces VM Interface v33. If you are running a Subnet, make sure to upgrade your Custom VM to v33 (subnet-evm@v0.6.0) before running it with v1.11.0.

You can download the Durango release here: https://github.com/ava-labs/avalanchego/releases/tag/v1.11.0

You can learn more about Durango here: https://www.avax.network/blog/durango-avalanche-warp-messaging-comes-to-the-evm

You can track support for ACP-23, ACP-24, ACP-25, ACP-30, ACP-31, ACP-41, and ACP-62 (included in Durango) on @snowpeerio:

https://snowpeer.io/acps

If you run a node on Fuji and already upgraded to v1.11.0-fuji, you should upgrade your node to v1.11.0 by **11 AM ET on Wednesday, March 6th, 2024**.

The production release (v1.11.0) contains some optimizations and fixes found during the Durango activation on Fuji that activate at the same time as the Mainnet activation.

If you run a Fuji Subnet, please wait for the subnet-evm@v0.6.0 release to upgrade your nodes to v1.11.0 (the VM Interface, as mentioned above, is now 33).

ICYMI AvalancheJS@v4 is now out (includes some massive performance upgrades and support for BLS staking transactions):

https://twitter.com/dhrubabasu_/status/1758224145916326318?s=20

You can read the unrolled version of this thread here: https://typefully.com/_patrickogrady/OJQotli

Avalanche Durango: C-Chain Warp Messaging

🌀 Welcome to the #Avalanche Interoperability Age 🌀

Today, the pre-release code for a proposed upgrade to the Avalanche Network (AWM Support on C-Chain) was published and will activate on the Fuji Testnet at 11 A.M. ET on Tuesday, February 13th, 2024.

https://github.com/ava-labs/avalanchego/releases/tag/v1.11.0-fuji

This proposed upgrade, codenamed Durango, consists of Avalanche Community Proposals (ACPs):

[23] P-Chain Native Transfers: https://github.com/avalanche-foundation/ACPs/blob/main/ACPs/23-p-chain-native-transfers.md

[24] Activate Shanghai EIPs on C-Chain: https://github.com/avalanche-foundation/ACPs/blob/main/ACPs/24-shanghai-eips.md

[25] Virtual Machine Application Errors: https://github.com/avalanche-foundation/ACPs/blob/main/ACPs/25-vm-application-errors.md

[30] Integrate Avalanche Warp Messaging into the EVM: https://github.com/avalanche-foundation/ACPs/blob/main/ACPs/30-avalanche-warp-x-evm.md

[31] Enable Subnet Ownership Transfer: https://github.com/avalanche-foundation/ACPs/blob/main/ACPs/31-enable-subnet-ownership-transfer.md

[41] Remove Pending Stakers: https://github.com/avalanche-foundation/ACPs/blob/main/ACPs/41-remove-pending-stakers.md

[62] Disable `AddValidatorTx` and `AddDelegatorTx`: https://github.com/avalanche-foundation/ACPs/blob/main/ACPs/62-disable-addvalidatortx-and-adddelegatortx.md

You can read more about the proposed Durango Upgrade here:

https://www.avax.network/blog/durango-avalanche-warp-messaging-comes-to-the-evm

As a reminder (have heard some confusion about this lately), AWM messages are not sent through the P-Chain.

The P-Chain just serves as a key registry that is used by nodes to lookup signers for a source Subnet to verify messages (from their local database).

Here is a good diagram that explains AWM from a presentation I gave at the @initc3org Winter Retreat in January 2023:

(Message delivery/signing % is up to the Subnets to specify...all AWM is concerned with is verifying that signers from one Subnet authenticated a message)

You can read the unrolled version of this thread here: https://typefully.com/_patrickogrady/jGYfnjR

Avalanche Durango Upgrade for Fuji

🛠️ 🛠️ #Avalanche Cortina 19 is out: https://github.com/ava-labs/avalanchego/releases/tag/v1.10.19 🛠️ 🛠️

This version (v1.10.19) is backwards compatible to v1.10.0. It is optional but recommended.

🔍 Focus: Options Before Verification + Fix P-Chain Validator Set Race + Fix C-Chain Bloom Filter Reconstruction

Options Before Verification

To prepare for deferred verification, support to invoke "Options" before "Verify" on "ProposalBlocks" has been added. This block type is used in the P-Chain.

https://github.com/ava-labs/avalanchego/pull/2363

Compatibility Check: VM Interface (v31)

Cortina 19 does NOT modify the VM interface. If you are running a Custom VM on v1.10.18, it will work with v1.10.19.

Fix P-Chain Validator Set Race

Cortina 18 introduced a regression that allowed for concurrent access of the P-Chain validator set (via the newly integrated P2P SDK in the P-Chain).

Fix P-Chain Validator Set Race (cont.)

This bug could cause non-deterministic corruption of the validator set cache, which in turn could AvalancheGo to get stuck and require a restart to recover.

https://github.com/ava-labs/avalanchego/pull/2672

Fix C-Chain Bloom Filter Reconstruction

When reconstructing the transaction bloom filter in the C-Chain, there was a bug that led to only the pending transaction that triggered reconstruction to be re-added (instead of all pending transactions).

https://github.com/ava-labs/coreth/pull/461

ICYMI @AvaLabs is now hiring for the following roles: https://boards.greenhouse.io/avalabs

ICYMI Make sure to check out the Vryx Specification:

https://twitter.com/_patrickogrady/status/1752028872781213703?s=20

You can read the unrolled version of this thread here: https://typefully.com/_patrickogrady/jPxh37j