Feature Proposal: Weight credits by timeliness

[bji]

Problem

There is no incentive for validators to vote as close as they can to the most recent block. As a result it may be optimal for a validator to 'lag behind' as far as is practically possible; delaying voting as long as possible gives a validator the greatest chance of detecting forks and choosing the most likely best fork.

Proposed Solution

Weight credits according to the timeliness of their votes. E.g. a vote for block A in block A + 1 gets 32 credits, a vote for block A in block A + 2 gets 31 credits, etc. Might even make the value fall off according to some nonlinear function to further encourage liveness of votes.

[riptl]

Eth2 uses a similar concept for rewarding validator attestation performance. Validators get rewarded according to attestation inclusion delay (reward := 1/n with n := <delay in slots>).

https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#components-of-attestation-deltas

Specifically

rewards[index] += Gwei(max_attester_reward // attestation.inclusion_delay)

Generally, the later a vote gets submitted, the less useful it will be to consensus. The proposed change is already widely used in other proof-of-stake networks to disincentivize late voting.

[bji]

Hi, I have written up a more detailed description of the proposal, including data gathered to show the effects:

https://www.shinobi-systems.com/timely_voting_proposal/

I am also working on an updated version of the PR that will include changes to the new VoteStateUpdate instruction to bring it in line with the proposed credits calculations (in addition to doing it for the Vote program, which is a lot easier).

I will post any updated data that I obtain here; I periodically collect results for new epochs since the performance characteristics of Solana are improving over time which leads to some differences in the impact of the proposed change, to the point where I believe I will likely propose a slightly different "parameterization" for the credits calculation than I had written about previously.

[bji]

Data for epoch 321:

https://www.shinobi-systems.com/timely_voting_proposal/v_321_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/d_321_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/v_321_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/d_321_4_16_1.html

Data for epoch 322:

https://www.shinobi-systems.com/timely_voting_proposal/v_322_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/d_322_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/v_322_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/d_322_4_16_1.html

[bji]

Data for epoch 323:

https://www.shinobi-systems.com/timely_voting_proposal/323/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/323/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/323/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/323/d_4_16_1.html

Data for epoch 324:

https://www.shinobi-systems.com/timely_voting_proposal/324/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/324/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/324/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/324/d_4_16_1.html

[bji]

Note that more recent data includes a "4 12 1" parameterization, which is:

• 4 slots grace period with full vote credits, 12 credits awarded for votes in the grace period
• 11 credits awarded for votes with latency 5
• 10 credits awarded for votes with latency 6
• 9 credits awarded for votes with latency 7
• etc
• Down to 1 credit awarded for votes with latency 15 and all latencies thereafter

I think that with the better performance of the cluster starting with 1.10, these numbers are more appropriate. But feedback is welcome!

[bji]

Data for epoch 325:

https://www.shinobi-systems.com/timely_voting_proposal/325/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/325/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/325/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/325/d_4_16_1.html

[bji]

Data for epoch 326:

https://www.shinobi-systems.com/timely_voting_proposal/326/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/326/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/326/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/326/d_4_16_1.html

[bji]

Data for epoch 327:

https://www.shinobi-systems.com/timely_voting_proposal/327/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/327/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/327/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/327/d_4_16_1.html

Data for epoch 328:

https://www.shinobi-systems.com/timely_voting_proposal/328/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/328/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/328/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/328/d_4_16_1.html

Data for epoch 329:

https://www.shinobi-systems.com/timely_voting_proposal/329/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/329/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/329/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/329/d_4_16_1.html

Data for epoch 330:

https://www.shinobi-systems.com/timely_voting_proposal/330/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/330/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/330/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/330/d_4_16_1.html

Data for epoch 331:

https://www.shinobi-systems.com/timely_voting_proposal/331/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/331/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/331/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/331/d_4_16_1.html

Data for epoch 332:

https://www.shinobi-systems.com/timely_voting_proposal/332/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/332/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/332/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/332/d_4_16_1.html

[bji]

Average of epochs 323 - 332 (inclusive):

https://www.shinobi-systems.com/timely_voting_proposal/avg_323_332/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/avg_323_332/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/avg_323_332/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/avg_323_332/d_4_16_1.html

[bji]

Data for epoch 333:

https://www.shinobi-systems.com/timely_voting_proposal/333/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/333/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/333/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/333/d_4_16_1.html

Data for epoch 334:

https://www.shinobi-systems.com/timely_voting_proposal/334/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/334/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/334/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/334/d_4_16_1.html

Data for epoch 335:

https://www.shinobi-systems.com/timely_voting_proposal/335/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/335/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/335/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/335/d_4_16_1.html

Data for epoch 336:

https://www.shinobi-systems.com/timely_voting_proposal/336/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/336/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/336/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/336/d_4_16_1.html

[bji]

Data for epoch 337:

https://www.shinobi-systems.com/timely_voting_proposal/337/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/337/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/337/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/337/d_4_16_1.html

Data for epoch 338:

https://www.shinobi-systems.com/timely_voting_proposal/338/v_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/338/d_4_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/338/v_4_16_1.html https://www.shinobi-systems.com/timely_voting_proposal/338/d_4_16_1.html

[bji]

Data for epoch 339:

https://www.shinobi-systems.com/timely_voting_proposal/339/v_3_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/339/d_3_12_1.html

Data for epoch 340:

https://www.shinobi-systems.com/timely_voting_proposal/340/v_3_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/340/d_3_12_1.html

Note I have switched to a "3 12 1" parameterization, which is:

3 slots grace period with full vote credits, 12 credits awarded for votes in the grace period
11 credits awarded for votes with latency 4
10 credits awarded for votes with latency 5
9 credits awarded for votes with latency 6
etc
Down to 1 credit awarded for votes with latency 14 and all latencies thereafter

The cluster clearly is composed of validators who in aggregate easily achieve latencies less than 3 the vast majority of the time, so that grace period now seems most appropriate.

[Benhawkins18]

If we wait too long to address vote timeliness, it's only a matter of time until either binaries or source code gets out to intentionally lag to maximize APY.

Even if there were strong negative social incentives not to lag, there are enough private, self funded, validators that might uptake such a modification.

Confirmation times would suffer and if enough validators made the change, then liveness could go.

[bji]

I don't have any working code to implement this, I just have been collecting the data using google bigtables and calculating the effects of the change by computing votes and their latencies using that data.

I thought that the change would be easy but it turns out it's not. The major issue is that the delta between the voted-on slots and the slot that the vote is received at, must be stored somewhere at the time that the vote is received. I had originally thought that just waiting until votes pop off the tower and calculating latency then would be possible, but it is not.

I had proposed maybe altering the Lockout structure to include a "credits to award for this slot" field, but that structure is ABI frozen, which I don't know exactly what that means, but I assumed it means that it can't be structurally changed.

However I was thinking, it's not necessary to change the structure, just interpret the "confirmation_count" field differently. It's 32 bits which is way overkill for counting confirmation counts up to 32. The top 8 bits could be used to store a "credits for this Lockout" value.

So:

• Vote or VoteStateUpdate arrives
• Clock is used to determine slot that the vote arrived in
• For all slots in the Vote, compute the credits that it should earn based on the difference between voted-in slot and voted-on slot
• Store that value in the Lockouts that are pushed in the tower as a result of the vote
• When Lockouts pop off the back of the tower, instead of awarding 1 credit, award the number of credits that were stored in the Lockout

[carllin]

However I was thinking, it's not necessary to change the structure, just interpret the "confirmation_count" field differently. It's 32 bits which is way overkill for counting confirmation counts up to 32. The top 8 bits could be used to store a "credits for this Lockout" value.

This could work. Could also use like 4 bits, since the latency before rewards go to 1 only goes up to 16

Note I have switched to a "3 12 1" parameterization, which is

I'm not entirely sure if this parameterization is the ideal, since it might be better to take a 1 credit loss than to pick the wrong fork. If we assume

1. The fork rate fork_rate is roughly the skip rate ~10%, with the average chance of taking the minor fork is minor_fork_chance 20% (need some data on this)
2. Picking the wrong fork costs you on average average_minor_fork_vote_lockout slots (Idk what the number of votes people usually vote on minor forks, maybe up to lockout 8? would be good to get some data on this)

Then that means over N slots, voting as soon as possible will cost you an expected:

fork_rate * minor_fork_chance * average_minor_fork_vote_lockout * N

voting opportunities.

The cost in credits of this then depends on the voting strategy you're comparing it to. If your strategy is delaying by 4 slots, the cost would be 11 credits per slot. Let's call this number delayed_strategy_cost_per_slot so then the expected cost of the naive strategy voting on time would be:

fork_rate * minor_fork_chance * average_minor_fork_vote_lockout  * N * delayed_strategy_cost_per_slot

So we have to find a function that sets delayed_strategy_cost_per_slot such that the rewards decrease fast enough to make these costs unworthwhile, maybe that function resembles something closer to eth's where the penalty is much steeper

[bji]

However I was thinking, it's not necessary to change the structure, just interpret the "confirmation_count" field differently. It's 32 bits which is way overkill for counting confirmation counts up to 32. The top 8 bits could be used to store a "credits for this Lockout" value.

This could work. Could also use like 4 bits, since the latency before rewards go to 1 only goes up to 16

Thank you very much for your reply and for considering this proposal.

I chose 8 bits just to give some headroom for future changes. But since you seem to be leaning towards even fewer "max credits" awarded to votes within the grace period, fewer bits would be fine with me as well.

Note I have switched to a "3 12 1" parameterization, which is

I'm not entirely sure if this parameterization is the ideal, since it might be better to take a 1 credit loss than to pick the wrong fork. If we assume

1. The fork rate fork_rate is roughly the skip rate ~10%, with the average chance of taking the minor fork is minor_fork_chance 20% (need some data on this)
2. Picking the wrong fork costs you on average average_minor_fork_vote_lockout slots (Idk what the number of votes people usually vote on minor forks, maybe up to lockout 8? would be good to get some data on this)

I don't think the fork rate is the skip rate, I think it's much lower, like 25% or less of the total skip rate. But definitely real numbers are needed, because this is just my conjecture based on watching for LockedOut on my validator, which happens at a rate far, far lower than the skip rate; and also because it makes sense that most skips do not produce a fork because the skipped leader was simply delinquent and the next leader chains directly off of the leader prior to the delinquent one; and there is no forking.

Then that means over N slots, voting as soon as possible will cost you an expected:

fork_rate * minor_fork_chance * average_minor_fork_vote_lockout * N

voting opportunities.

The cost in credits of this then depends on the voting strategy you're comparing it to. If your strategy is delaying by 4 slots, the cost would be 11 credits per slot. Let's call this number delayed_strategy_cost_per_slot so then the expected cost of the naive strategy voting on time would be:

fork_rate * minor_fork_chance * average_minor_fork_vote_lockout  * N * delayed_strategy_cost_per_slot

So we have to find a function that sets delayed_strategy_cost_per_slot such that the rewards decrease fast enough to make these costs unworthwhile, maybe that function resembles something closer to eth's where the penalty is much steeper

Certainly happy to try to gather data and help with this analysis. If you just look at my data though you'll see that known laggers get punished exceedingly hard. Now whether or not they'd be able to adapt to a smaller lagging with smaller positive impact on credits, I don't know, but my gut feeling is that with only three slots of "free" latency, a validator takes a risk by waiting one or two slots to evaluate fork choices, because any vote they cast thereafter has a good chance of missing the grace window due to packet loss or missing the leader or whatever.

But again, real data is better than conjecture for sure. I have access to Google Bigtables and could write a more sophisticated analysis tool if it would help. I am not sure if lockouts can be detected reliable from this data since it only includes votes that landed, not votes on dead forks that caused lockouts but, maybe validator lockouts can be inferred from long periods of not voting? Especially if many validators get locked out on the same slots.

[bji]

By the way I am very amenable to more "punishing" credit schedules; I selected the schedule I did because I wanted to be as forgiving as possible while still punishing known lagging strategies, because I didn't want validators to feel like the change was too scary.

Additionally, the implementation should allow for easy tuning of the numbers and a later change could always ratched up the penalties or reduce them as needed.

[AshwinSekar]

Just a heads up the confirmation count is going to be an u8 once the compaction feature flag is turned on https://github.com/solana-labs/solana/blob/master/sdk/program/src/vote/state/mod.rs#L93

[carllin]

@AshwinSekar that's only in the VoteStateUpdate, I think this proposal was to modify the confirmation counts in the VoteState itself to track the latency

[bji]

I have added a proposal for eliminating VoteStateUpdate. This will make implementation of this timely voting credits proposal MUCH easier.

https://github.com/solana-labs/solana/issues/27473

[bji]

Some older data that I hadn't uploaded:

https://www.shinobi-systems.com/timely_voting_proposal/341/v_3_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/341/d_3_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/342/v_3_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/342/d_3_12_1.html

[bji]

Epochs 351 and 352:

https://www.shinobi-systems.com/timely_voting_proposal/351/v_3_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/351/d_3_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/352/v_3_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/352/d_3_12_1.html

[bji]

Epoch 353:

https://www.shinobi-systems.com/timely_voting_proposal/353/v_3_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/353/d_3_12_1.html

[bji]

Epochs 354 and 355:

https://www.shinobi-systems.com/timely_voting_proposal/354/v_3_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/354/d_3_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/355/v_3_12_1.html https://www.shinobi-systems.com/timely_voting_proposal/355/d_3_12_1.html

[bji]

Epoch 356:

https://www.shinobi-systems.com/timely_voting_proposal/356/v_3_10_1.html https://www.shinobi-systems.com/timely_voting_proposal/356/d_3_10_1.html

Note that to match the pull request for the change that implements this feature, I've switched to a 3 10 1 parameterization.

Just for additional information, here's an example of 2 8 1 parameterization. This more clearly shows which validators are lagging.

https://www.shinobi-systems.com/timely_voting_proposal/356/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/356/d_2_8_1.html

[bji]

Epoch 364:

https://www.shinobi-systems.com/timely_voting_proposal/364/v_3_10_1.html https://www.shinobi-systems.com/timely_voting_proposal/364/d_3_10_1.html

I might be wrong but it looks like more validators have chosen to lag for more credits. I see a few in the top 10 that are clearly lagging with > 3 avg latency.

[bji]

In honor of our new top lagger, I have gathered to data for the three most recent epochs:

Epoch 377:

https://www.shinobi-systems.com/timely_voting_proposal/377/v_3_10_1.html https://www.shinobi-systems.com/timely_voting_proposal/377/d_3_10_1.html

Epoch 378:

https://www.shinobi-systems.com/timely_voting_proposal/378/v_3_10_1.html https://www.shinobi-systems.com/timely_voting_proposal/378/d_3_10_1.html

Epoch 379:

https://www.shinobi-systems.com/timely_voting_proposal/379/v_3_10_1.html https://www.shinobi-systems.com/timely_voting_proposal/379/d_3_10_1.html

And here's some data showing a more aggressive lagging penalty. I think it would be even better to use these parameters:

https://www.shinobi-systems.com/timely_voting_proposal/377/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/377/d_2_8_1.html

https://www.shinobi-systems.com/timely_voting_proposal/378/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/378/d_2_8_1.html

https://www.shinobi-systems.com/timely_voting_proposal/379/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/379/d_2_8_1.html

[bji]

There was a request for most recent epoch data. Here it is:

For 3 10 1 parameters:

Epoch 380:

https://www.shinobi-systems.com/timely_voting_proposal/380/v_3_10_1.html https://www.shinobi-systems.com/timely_voting_proposal/380/d_3_10_1.html

Epoch 381:

https://www.shinobi-systems.com/timely_voting_proposal/381/v_3_10_1.html https://www.shinobi-systems.com/timely_voting_proposal/381/d_3_10_1.html

Epoch 382:

https://www.shinobi-systems.com/timely_voting_proposal/382/v_3_10_1.html https://www.shinobi-systems.com/timely_voting_proposal/382/d_3_10_1.html

For 2 8 1 parameters:

Epoch 380:

https://www.shinobi-systems.com/timely_voting_proposal/380/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/380/d_2_8_1.html

Epoch 381:

https://www.shinobi-systems.com/timely_voting_proposal/381/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/381/d_2_8_1.html

Epoch 382:

https://www.shinobi-systems.com/timely_voting_proposal/382/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/382/d_2_8_1.html

[bji]

More recent epoch data:

Epoch 392 (3 10 1):

https://www.shinobi-systems.com/timely_voting_proposal/392/v_3_10_1.html https://www.shinobi-systems.com/timely_voting_proposal/392/d_3_10_1.html

Epoch 392 (2 8 1):

https://www.shinobi-systems.com/timely_voting_proposal/392/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/392/d_2_8_1.html

Epoch 393 (3 10 1):

https://www.shinobi-systems.com/timely_voting_proposal/393/v_3_10_1.html https://www.shinobi-systems.com/timely_voting_proposal/393/d_3_10_1.html

Epoch 393 (2 8 1):

https://www.shinobi-systems.com/timely_voting_proposal/393/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/393/d_2_8_1.html

Epoch 394 (3 10 1):

https://www.shinobi-systems.com/timely_voting_proposal/394/v_3_10_1.html https://www.shinobi-systems.com/timely_voting_proposal/394/d_3_10_1.html

Epoch 394 (2 8 1):

https://www.shinobi-systems.com/timely_voting_proposal/394/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/394/d_2_8_1.html

Epoch 395 (3 10 1):

https://www.shinobi-systems.com/timely_voting_proposal/395/v_3_10_1.html https://www.shinobi-systems.com/timely_voting_proposal/395/d_3_10_1.html

Epoch 395 (2 8 1):

https://www.shinobi-systems.com/timely_voting_proposal/395/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/395/d_2_8_1.html

Epoch 396 (3 10 1):

https://www.shinobi-systems.com/timely_voting_proposal/396/v_3_10_1.html https://www.shinobi-systems.com/timely_voting_proposal/396/d_3_10_1.html

Epoch 396 (2 8 1):

https://www.shinobi-systems.com/timely_voting_proposal/396/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/396/d_2_8_1.html

Average of Epochs 392-396: (3 10 1)

https://www.shinobi-systems.com/timely_voting_proposal/avg_392_396/v_3_10_1.html https://www.shinobi-systems.com/timely_voting_proposal/avg_392_396/d_3_10_1.html

Average of Epochs 392-396: (2 8 1)

https://www.shinobi-systems.com/timely_voting_proposal/avg_392_396/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/avg_392_396/d_2_8_1.html

[bji]

Switching to only reporting the 2 8 1 parameterization now.

Epoch 397

https://www.shinobi-systems.com/timely_voting_proposal/397/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/397/d_2_8_1.html

Epoch 398

https://www.shinobi-systems.com/timely_voting_proposal/398/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/398/d_2_8_1.html

Epoch 399

https://www.shinobi-systems.com/timely_voting_proposal/399/v_2_8_1.html https://www.shinobi-systems.com/timely_voting_proposal/399/d_2_8_1.html