This “Ethereum mining pool construction guide” comes from the first-hand experience of Dragonfly Capital research partner Ivan Bogatyy in building the MiningDAO.io mining pool, and outlines how to reduce the uncle block rate.
Written by: Ivan Bogatyy, research partner of Dragonfly Capital, a blockchain investment institution. Compiler: Perry Wang
The mining pool is the main force in the Ethereum ecosystem. With the exponential growth of miners’ extractable value (MEV), the upgrade of EIP-1559 and the upcoming merger, mining pools have become more important and more powerful participants in the ecosystem.
Explain to Mengxin: The mining pool is a software provider that enables many mining machines to pool their mining capabilities and share rewards. From two perspectives, mining pools are indispensable in mining based on the Proof of Work (PoW) consensus mechanism:
- First of all, because the income of individual miners will fluctuate greatly;
- Secondly, because it is becoming more and more difficult to build software infrastructure around mining. Through the concentration of resources, individual miners can reduce income fluctuations and therefore have more predictable business operations.
But this kind of power is also accompanied by huge responsibilities. The mining pool has a lot of power. This is because the mining pool ultimately determines which blocks its mining machine processes and which transactions can be contained in these blocks. The mining pool decides which MEV to withdraw and who can withdraw it, they vote on the Gas limit and participate in major political struggles. Therefore, for the Ethereum culture, it is very important that the entry threshold of the mining pool is as low as possible to maximize decentralization.
So when I decided to build a mining pool, I was surprised to find that it was a very challenging job! There is very little public knowledge sharing on how to run a competitive mining pool with high responsiveness and low uncle block rate.
So I thought: Okay, let’s solve this problem.
Building a mining pool consists of two parts:
- Set up a full-node client with good peer-to-peer network and fast processing speed;
- Connect the full node to the mining pool software, which manages the hash rate and distributes the workload among all the miners.
In this article, both will be discussed.
This “Ethereum mining pool construction guide” comes from our first-hand experience in building the MiningDAO.io mining pool, and outlines how we can reduce the uncle block rate from 10%-14% to about 4%-5%, which is similar to the previous one. Part of the organization in the 10 mining pools remained flat or even better.
Set up Ethereum full node client
To run the mining pool, you need to run the Ethereum full node client. The client will be responsible for receiving new blocks and pending transactions, as well as generating its own blocks and broadcasting them to other nodes. This section describes how to correctly set up a full-node client.
Server hardware requirements
Running a fully synchronized node requires fairly good hardware. We recommend at least 32GB of memory (RAM) and at least 2TB of SSD storage (you will always need to synchronize the Ethereum chain with the HDD).
Bandwidth is also very important. It is best to be as close to other nodes as possible to receive new blocks as quickly as possible. We recommend using cloud hosting dedicated machines for cloud services commonly used by other mining pools: OVH and Hetzner in Europe, and Alibaba Cloud and Amazon Cloud AWS in Asia.
Geth or OpenEthereum? Geth!
The next decision is which Ethereum client to use. The most popular and well-tested options are Geth and OpenEthereum (formerly known as Parity). Geth is a leader in protocol development and is always up-to-date.
For comparison, we used Parity-2.7.2 (the latest stable branch before the refactoring of OpenEthereum) and OpenEthereum to conduct some small-scale experiments, but the test results of both in terms of block import time and block production time are very bad , Resulting in unacceptably high uncle block rate.
We welcome anyone to conduct a more thorough A/B test and contact us to provide us with more data, but at this stage, we only recommend Geth.
Here is the command line we used:
geth –datadir=/ssd/gethdata –syncmode=fast –cache=21000
–maxpeers=250 –txpool.globalslots=1000
–http –http.api=eth
–miner.etherbase=’0xADDRESS’ –mine –miner.threads=0
–miner.extradata=’MiningDAO’
–miner.notify=’ http://127.0.0.1:8107 ‘&>> ~/geth-log.txt
Here –cache=21000 means to allocate 21GB for memory state storage (this is the peak that Geth can handle), and the rest of the command line will be explained below.
More importantly, the modifications to the Geth code we describe below can be found here , as a downloadable repository, or here , as a patch to be applied.
Minimize the frequency of empty blocks
There are two things that will destroy the value return obtained by miners: digging out uncle blocks and digging out empty blocks.
The two are almost equally bad: the uncle block reward is 1.75 ETH, the empty block reward is 2 ETH, and there is no transaction fee surplus in both cases. In contrast, the total reward for a complete block with transaction fees usually amounts to 3-4 ETH, sometimes even more.
Why do mining pools sometimes produce empty blocks , and how to minimize their output frequency?
When another mining pool mines a new block (such as height N), any other block of height N may become an uncle block. Therefore, whenever a new block is found, Geth will immediately switch the work of the miner and dig an empty block at the height of N+1. This empty block does not include transaction fees, but it is better than mining a block destined to become an uncle block.
Subsequently, Geth constructed a “real” block at a height of N+1 and switched the work of the miners again. It takes time (0.1-0.3 seconds) to construct such a “real” block, so a two-step process is required. But during this transition period of 0.1 to 0.3 seconds, other miners are digging an empty block.
It may be tempting to collect all pending transactions in order to maximize the fee income, but greedy like txpool.globalslots will greatly increase the amount of processing Geth needs to build a “real” block (it takes up to 1 second) Or longer). Our recommended value is not greater than 1000 or 2000.
To know more details about this, please check https://github.com/ethereum/go-ethereum/issues/21899
Minimize the frequency of uncle block generation
After solving the empty blocks, we can start the difficult part. To minimize the uncle block rate, two things are key:
- When other mining pools produce new blocks, grasp the news as soon as possible;
- When your mining pool generates a new block, spread it as widely as possible (so that other people start mining behind it).
As mentioned earlier, the first step to achieve a good peer-to-peer network (p2p) is to run your full node on a cloud server close to other nodes with good bandwidth.
Second, good bandwidth allows nodes to handle more direct peers, thereby reducing the number of p2p hops required to receive new information. The flag for the number of peers in the Geth command line is –maxpeers.
Below we will explain some more subtle and powerful techniques to maximize the block import speed and block broadcast speed.
Use bloXroute
bloXroute is a service dedicated to improving the connection between miners and reducing their uncle block rate. Most mining pools are connected to bloXroute, and even major mature mining pools have reported that using bloXroute will bring significant improvements. The index measurement carried out by KeeperDAO further confirms that bloXroute has a huge advantage over similar services.
Our experiment also shows a significant improvement. On the new sync node with default peer settings, approximately 90% of all new blocks come from bloXroute first (only 10% come from all other peer nodes).
Even after our nodes were fine-tuned to connect to top peers, 40%-60% of new blocks came first from bloXroute.
After following the bloXroute setup tutorial, don’t forget to add the bloXroute node to the “trusted peers” set for your Geth , you will use it later. A trusted peer is a preset node that Geth will always connect to, regardless of how the peer is randomly initialized. Trusted peers also do not count towards the connection limit. Adding the bloXroute gateway to the trusted peer will ensure that Geth’s connection will not be accidentally disconnected.
We further recommend connecting to the Tai Chi network. The Taiji Network Project is a block broadcasting network developed by the Spark Mine Pool. You can maintain the connection to the Tai Chi network by adding each endpoint of the Tai Chi network to the same trusted peer file.
Actively broadcast your block
Whenever Geth successfully mines a new block, it sends the block to broadcast on the network. By default, Geth only broadcasts it to a random subset of size sqrt(n_peers), which broadcasts blocks to some of their peers, and so on.
Even if all peers are equally effective, this mechanism is not ideal, but when some peers are more powerful than others, and these peers are not ultimately included in the subset, this mechanism is The disadvantages are particularly obvious.
In particular, the first thing to do when mining a new block is to send it to bloXroute in order to forward it to all participating mining pools. If the bloXroute gateway does not eventually appear in that random sqrt(n_peers) subset, then your chances of getting uncle blocks will be greatly increased!
Next, you will want to send the block to the peer with the highest quality, and then to all the remaining peers.
We have open sourced the following Geth patch and recommend applying it to your client. It broadcasts all newly mined blocks to all trusted nodes (including bloXroute), and then broadcasts to all remaining peers.
Cultivate the widest peers
Vanilla Geth aims to achieve maximum decentralization and flat network structure. This choice is very suitable for enthusiasts and supports a strong ecosystem consisting of thousands of nodes. However, as we saw in the previous section, these default parameters do not work well for nodes that perform critical responsibilities or that would cause high costs if they fail.
In fact, not all peers are equally effective. Some nodes have slow connection speeds and will neither provide new blocks nor help your block broadcast. Others, especially nodes of other mining pools, will generate a steady stream of new block data.
Based on Spark Pool’s recommendations, we adjusted the Geth parameters to record which nodes sent us new blocks first. After several months of collecting this data, we were able to find the best peer (through the “static”/”trusted” node setting in Geth) that is always connected. Here is a Python script that we use to process the above data and convert it into a trusted_nodes.json list that Geth can ingest.
Since the MiningDAO mining pool exists in each geographic region (North America, Europe, Asia), we have performed data mining on the top peer list in each geographic region. Unfortunately, we cannot publicly share the IP of these nodes to avoid DDoS attacks on these nodes. For serious inquirers with good reasons, we can share it privately. I am also happy to share our own nodes in each geographic area for other mining pools to connect!
Set up mining pool software
After setting up the full node correctly, the next step is to set up the mining pool software itself. The software will be responsible for handling connections from all individual miners, tracking workers’ shares and managing expenditures.
Choose mining pool software
We briefly analyzed the following 4 options: Miningcore , open-ethereum-pool , NOMP (node ​​open mining portal) and MPOS (mining portal open source) . We later learned about Flexpool Solo , but did not experiment with it.
We have gained a wealth of experience with Miningcore for two reasons. First, it saves all past data on the disk of the SQL database, which is different from open-ethereum-pool, which only saves data in RAM through Redis. Disk storage provides stability against restart and the ability to analyze historical data. Second, we like Miningcore’s highly readable, object-oriented code.
In the end, MiningDAO adopted our own mining pool software, using Miningcore as the model, written in Go language to improve speed. We hope to open source our software as soon as possible, but at the same time we recommend using Miningcore.
Fix the delay problem of the mining pool software
One of the main problems we encountered when using Miningcore was the way it handled job updates. By default, Miningcore pings the full-node remote procedure call (RPC) every 0.5 seconds to see if the latest job has changed. This setting may be suitable for other cryptocurrencies with a long block production time (for Bitcoin with a block production time of 10 minutes, this is a negligible problem), but for Ethereum, this This setting can result in an unacceptably high uncle block rate.
As a background introduction, there is a simple way to calculate the increase in the uncle block rate caused by any processing delay. The block generation time is Poisson distributed , which means that no matter how long it has been since the last mined block, the probability of finding the next block in the next second (or milliseconds or other time unit) is always the same. For example, the goal of Ethereum is 13 seconds to produce a block, which means that the probability of a block in the next second is always 1 second / 13 seconds ≈ 7.7%. Therefore, if your mining pool has a delay of 0.1 second anywhere in the pipeline for any reason, there will be an additional uncle block of 0.1 second/13 seconds ≈ 0.77% due to the delay. The uncle block will come from the 0.1 second time period when your miner handles obsolete work.
Back to Miningcore. Using the above formula, if there is a 0.5 second delay in the mining industry update, it will result in 0.5 seconds / 13 seconds ≈ 4% extra uncle blocks (absolute ratio, not relative percentage). Of course, such a high rate of unforced errors is unacceptable. We have tried extensively to reduce the update frequency from 0.5 seconds to 50 milliseconds and below, but found that this setting is quite unreliable: work updates are still significantly delayed.
A better solution is to use Geth’s notifyWork function so that Geth will actively send it to the mining pool software when a job update occurs. We patched Miningcore to support this option and released a modified version . After transitioning to notifyWork, we found that the communication delay between Geth and Miningcore was almost negligible, so our uncle block rate was significantly reduced.
Summarize
I hope this article is useful and guide more people to operate Ethereum mining pools; the culture of self-made mining pools is crucial to maintaining the openness and decentralization of Ethereum.
To summarize briefly, we initially used vanilla Miningcore’s default parameters and vanilla Miningcore software. The uncle block rate of this default setting is about 10%-14%. After gradually adopting the various modifications outlined in this article, we have reduced our uncle block rate to 4%-5% , which is equal to or better than the current level of the top 10 mining pools (the uncle block rate shown in Etherscan) Slightly higher because we sometimes experiment in production).
Our Geth modifications can be used as repo here find , it is also available here as a patch found. You can here find our Miningcore modifications, and can here find the mine pool configuration file.
If you have further ideas on how to improve this parameter setting, please send us a pull request or email !
Acknowledgement:
Miningcore acceleration was developed by Alexander Melnikov . Thanks to these friends for their suggestions and ideas: Alex Obadia ( Flashbots ), Eyal Markovich , Shen Chen ( bloXroute ), Xin Xu ( Sparkpool ), Yang Ze ( Sparkpool ), Chris ( Flexpool ) and Haseeb Qureshi ( Dragonfly Capital ).
Source link: medium.com
