Wallet development experience sharing: BTC

[TOC]

BTC node building

The first step of BTC is to build nodes. Because BTC is the most popular and popular, there are many articles about BTC node building on the network, or problems in synchronizing nodes. I won't go into details here (mainly there is no environment for building nodes). Here is an article: Building Bitcoin test node under blockchain Linux . Two websites can be considered if no node is built: blockcypher,blockchain.

BTC's account model - UTXO

For the definition of UTXO, please refer to Understand bitcoin's UTXO, address and transactions , I think this article is quite comprehensive about UTXO. It is mentioned in: in bitcurrency, every input and output of a transaction is actually UTXO. The input UTXO is the rest of the previous transaction, more accurately, the output UTXO of the previous transaction. This statement should be understood from JSON data.

Each transaction contains an output greater than or equal to, as shown in the following figure:

The output list includes the output quantity (value), input script (script), addresses and script type. We mainly focus on the input quantity.

JSON of each transaction contains zero or more inputs (no input for mining revenue), as shown in the following figure:

The input list contains the prev? Hash of the last transaction corresponding to this input, the output? Index of the last transaction corresponding to this input, the input script, the script? Type and other fields. The two most important fields in the input are the hash and output subscript of the last transaction. From these two fields, we can easily find the output of the last transaction corresponding to this input, so as to find the number of this input from the output.

Calculated balance

From the above account model, we know that BTC's account is composed of UTXO list. From the initial stage of creation to the current, all transactions of each account are a series of inputs and outputs. These utxos are connected in series through the rules of inputs and outputs, forming a chain structure. Therefore, to calculate the account balance, we can finally obtain the balance by calculating this series of utxos, but in the In the actual development, it is very performance consuming, so in the development, we often consider obtaining the calculation results directly from the third-party blockchain browser through the open API. In fact, based on the API developed by the third-party blockchain browser, we can directly complete many operations without building nodes:

Reference code:

    /**
     * balance
     * @param address
     * @param mainNet
     * @return
     */
    public static String balance(String address, boolean mainNet){
        String host = mainNet ? "blockchain.info" : "testnet.blockchain.info";
        String url = "https://" + host + "/balance?active=" + address;
        OkHttpClient client = new OkHttpClient();
        String response = null;
        try {
            response = client.newCall(new Request.Builder().url(url).build()).execute().body().string();
        } catch (IOException e) {
            e.printStackTrace();
        }
        Map<String, Map<String, Object>> result = new Gson().fromJson(response, Map.class);
        Map<String, Object> balanceMap = result.get(address);
        BigDecimal finalBalance = BigDecimal.valueOf((double) balanceMap.get("final_balance"));
        BigDecimal balance = finalBalance.divide(new BigDecimal(100000000));
        return balance.toPlainString();
    }

Test code:

	/**
	 * Obtain balance
	 * @throws Exception
	 */
	@Test
	public void testGetBTCBalance() throws Exception{
		String address = "17A16QmavnUfCW11DAApiJxp7ARnxN5pGX";
		String balance = BtcUtil.balance(address, true);
		logger.warn("balance: {}", balance);
	}

Calculation of miner's fee:

It can be seen from the above conclusion that each transaction consists of zero, one or more inputs and one or more outputs. Each input points to the output of the previous transaction, so that each transaction is serially formed by these inputs and outputs (UTXO). Generally speaking, a transaction will have one or more inputs. The total number of these inputs is just or greater than the number of this transaction. There will be one or more outputs. The output mainly includes the collection address and quantity of this transaction, as well as the change address and change quantity. The change address is usually the original address. The total number of inputs and the total number of outputs are always unequal , because the miner's fee is included in each transaction, we can deduce the calculation method of miner's fee, that is, the total input of each transaction minus the total output:

Reference code:

    /**
     * Calculation of miners' fees
     * @param txid
     * @param mainNet
     * @return
     */
    public static String fee(String txid, boolean mainNet){
        String host = mainNet ? "blockchain.info" : "testnet.blockchain.info";
        String url = "https://" + host + "/rawtx/" + txid;
        OkHttpClient client = new OkHttpClient();
        String response = null;
        try {
            response = client.newCall(new Request.Builder().url(url).build()).execute().body().string();
        } catch (IOException e) {
            e.printStackTrace();
        }
        JSONObject jsonObject = JSONObject.parseObject(response);

        // Total statistical input
        JSONArray inputs = jsonObject.getJSONArray("inputs");
        BigDecimal totalIn = BigDecimal.ZERO;
        for (int i = 0; i < inputs.size(); i++) {
            JSONObject inputsData = inputs.getJSONObject(0);
            JSONObject prevOut = inputsData.getJSONObject("prev_out");
            totalIn = totalIn.add(prevOut.getBigDecimal("value"));
        }

        // Total statistical output
        JSONArray outs = jsonObject.getJSONArray("out");
        BigDecimal totalOut = BigDecimal.ZERO;
        for (int i = 0; i < outs.size(); i++) {
            JSONObject outData = outs.getJSONObject(i);
            totalOut = totalOut.add(outData.getBigDecimal("value"));
        }

        return totalIn.subtract(totalOut).divide(new BigDecimal(100000000)).toPlainString();
    }

Test code:

	/**
	 * Calculation of miners' fees
	 * https://blockchain.info/rawtx/$tx_hash
	 */
	@Test
	public void testGetMinerFee(){
		String txid = "b8df97b51f54df1c1f831e0e9e5561c03822f6c5a5a59e0118b15836657a4970";
		logger.warn("Fee: {}", BtcUtil.fee(txid, true));
	}

There are some differences between the transaction data obtained through the open API of the third-party blockchain browser and the transaction data obtained from the self built node. If the node is self built, I recommend using azazar/bitcoin-json-rpc-client Or other SDKs that encapsulate bitcoin RPC interface are the simplest and most convenient way to implement. They encapsulate a lot of objects. We don't need to manually obtain the required data from JSONObject objects. Moreover, through these SDKs, we can really call the bitcoin node interface as we call methods.

Get unused list

Reference code:

    /***
     * Get unused list
     * @param address : address
     * @return
     */
    public static List<UTXO> getUnspent(String address, boolean mainNet) {
        List<UTXO> utxos = Lists.newArrayList();
        String host = mainNet ? "blockchain.info" : "testnet.blockchain.info";
        String url = "https://" + host + "/zh-cn/unspent?active=" + address;
        try {

            OkHttpClient client = new OkHttpClient();
            String response = client.newCall(new Request.Builder().url(url).build()).execute().body().string();
            if (StringUtils.equals("No free outputs to spend", response)) {
                return utxos;
            }
            JSONObject jsonObject = JSON.parseObject(response);
            JSONArray unspentOutputs = jsonObject.getJSONArray("unspent_outputs");
            List<Map> outputs = JSONObject.parseArray(unspentOutputs.toJSONString(), Map.class);
            if (outputs == null || outputs.size() == 0) {
                System.out.println("Abnormal transaction, insufficient balance");
            }
            for (int i = 0; i < outputs.size(); i++) {
                Map outputsMap = outputs.get(i);
                String tx_hash = outputsMap.get("tx_hash").toString();
                String tx_hash_big_endian = outputsMap.get("tx_hash_big_endian").toString();
                String tx_index = outputsMap.get("tx_index").toString();
                String tx_output_n = outputsMap.get("tx_output_n").toString();
                String script = outputsMap.get("script").toString();
                String value = outputsMap.get("value").toString();
                String value_hex = outputsMap.get("value_hex").toString();
                String confirmations = outputsMap.get("confirmations").toString();
                UTXO utxo = new UTXO(Sha256Hash.wrap(tx_hash_big_endian), Long.valueOf(tx_output_n), Coin.valueOf(Long.valueOf(value)),
                        0, false, new Script(Hex.decode(script)));
                utxos.add(utxo);
            }
            return utxos;
        } catch (Exception e) {
            return null;
        }
    }

Test code:

	/**
	 * Get unused list
	 */
	@Test
	public void testGetUnSpentUtxo(){
		String address = "17A16QmavnUfCW11DAApiJxp7ARnxN5pGX";
		List<UTXO> unspent = BtcUtil.getUnspent(address, true);
		logger.warn("unspent: {}", unspent);
	}

Offline signature

Reference code:

    /**
     * Offline signature
     * @param unSpentBTCList
     * @param from
     * @param to
     * @param privateKey
     * @param value
     * @param fee
     * @param mainNet
     * @return
     * @throws Exception
     */
    public static String signBTCTransactionData(List<UTXO> unSpentBTCList, String from, String to, String privateKey, long value, long fee, boolean mainNet) throws Exception {
        NetworkParameters networkParameters = null;
        if (!mainNet)
            networkParameters = MainNetParams.get();
        else
            networkParameters = TestNet3Params.get();

        Transaction transaction = new Transaction(networkParameters);
        DumpedPrivateKey dumpedPrivateKey = DumpedPrivateKey.fromBase58(networkParameters, privateKey);

        ECKey ecKey = dumpedPrivateKey.getKey();

        long totalMoney = 0;
        List<UTXO> utxos = new ArrayList<>();
        //Traverse the unused list and assemble the appropriate item s
        for (UTXO us : unSpentBTCList) {
            if (totalMoney >= (value + fee))
                break;
            UTXO utxo = new UTXO(us.getHash(), us.getIndex(), us.getValue(), us.getHeight(), us.isCoinbase(), us.getScript());
            utxos.add(utxo);
            totalMoney += us.getValue().value;
        }

        transaction.addOutput(Coin.valueOf(value), Address.fromBase58(networkParameters, to));
        // transaction.

        //The total amount of consumption list - transferred amount - handling fee is equal to the amount to be returned to yourself
        long balance = totalMoney - value - fee;
        //Output - transfer to yourself
        if (balance > 0) {
            transaction.addOutput(Coin.valueOf(balance), Address.fromBase58(networkParameters, from));
        }
        //Enter unused list items
        for (UTXO utxo : utxos) {
            TransactionOutPoint outPoint = new TransactionOutPoint(networkParameters, utxo.getIndex(), utxo.getHash());
            transaction.addSignedInput(outPoint, utxo.getScript(), ecKey, Transaction.SigHash.ALL, true);
        }

        return Hex.toHexString(transaction.bitcoinSerialize());
    }

After signing, the result can be broadcast. No self built node can be used blockcypher/send Broadcast your own deal. In the above transaction, the change itself, of course, can also be set to other wallet addresses. Secondly, in this transaction, the transaction fee is calculated in the front step, and its calculation method will be mentioned below.

Broadcasting trade

If you set up a node by yourself, you can call the interface to broadcast the transaction directly. This is for the students who do not set up a node but want to complete the whole transaction process. We can find a lot of API s on search engines that can broadcast transactions for us, and I use them here blockcypher/send.

Reference code:

    /**
     * Network wide broadcast transaction
     * @param tx
     * @param mainNet
     * @return
     */
    public static String sendTx(String tx, boolean mainNet){
        String url = "";
        if(mainNet) {
            url = "https://api.blockcypher.com/v1/btc/main/txs/push";
        }else {
            url = "https://api.blockcypher.com/v1/btc/test3/txs/push";
        }
        OkHttpClient client = new OkHttpClient();
        JSONObject jsonObject = new JSONObject();
        jsonObject.put("tx", tx);
        String response = null;
        try {
            response = client.newCall(new Request.Builder().url(url).post(RequestBody.create(MediaType.parse("application/json"), jsonObject.toJSONString())).build()).execute().body().string();
        } catch (IOException e) {
            e.printStackTrace();
        }
        return response;
    }

Calculation of miners' fees

For the explanation of the calculation formula of miner's fee, please refer to BTC handling fee calculation, how to set handling fee . Through the guidance of this article, to calculate the miner's fee, we need to get the rate first, that is, how much per byte is equal to.

Reference code:

    /**
     * Acquisition rate
     * @param level 3 fastestFee 2 halfHourFee 1 hourFee default fastestFee
     * @return
     */
    public static String feeRate(int level){
        OkHttpClient client = new OkHttpClient();
        String url = "https://bitcoinfees.earn.com/api/v1/fees/recommended";
        String response = null;
        try {
            response = client.newCall(new Request.Builder().url(url).build()).execute().body().string();
        } catch (IOException e) {
            e.printStackTrace();
        }
        JSONObject jsonObject = JSONObject.parseObject(response);
        switch (level){
            case 1:
                return jsonObject.getBigDecimal("hourFee").toPlainString();
            case 2:
                return jsonObject.getBigDecimal("halfHourFee").toPlainString();
            default:
                return jsonObject.getBigDecimal("fastestFee").toPlainString();
        }
    }

Test code:

	@Test
	public void testGetFeeRate(){
		logger.warn("feeRate: {}", BtcUtil.feeRate(3));
	}

After you get the rate, you can calculate the miner's fee. Generally speaking, a transaction contains several inputs. When the sum of these inputs is just enough to pay for the number of transactions, the volume of output is the smallest. Only the output of one receiving address is available. When the sum of these inputs is greater than the number of transactions, the output quantity includes the output of one receiving address and one zero change output, which is separated from the above Line signed code can also easily understand this.

Reference code:

    /**
     * Get miners' fees
     * @param amount
     * @param utxos
     * @return
     */
    public static Long getFee(long amount, List<UTXO> utxos) {
        Long feeRate = Long.valueOf(feeRate(3));//Acquisition rate
        Long utxoAmount = 0L;
        Long fee = 0L;
        Long utxoSize = 0L;
        for (UTXO us : utxos) {
            utxoSize++;
            if (utxoAmount >= (amount + fee)) {
                break;
            } else {
                utxoAmount += us.getValue().value;
                fee = (utxoSize * 148 + 34 * 2 + 10) * feeRate;
            }
        }
        return fee;
    }

Test code:

	@Test
	public void testGetFee(){
		String address = "17A16QmavnUfCW11DAApiJxp7ARnxN5pGX";
		List<UTXO> unspent = BtcUtil.getUnspent(address, true);
		Long fee = BtcUtil.getFee(100 * 100000000, unspent);
		logger.warn("fee: {}", BigDecimal.valueOf(fee / 100000000.0).toPlainString());
	}

Optimize miners' fees

Through the calculation formula of miner's fee (input*148+34*out+10)*rate, we can easily think of ways to reduce miner's fee. There are two main aspects: one is to choose a lower miner's fee, which can significantly reduce miner's fee. Because the formula can clearly reflect the multiple relationship between rate and volume of input and output, reducing rate is the most effective way to reduce miner's fee, but relatively The negative effects of these methods are also direct, which will affect the efficiency of packaging. The second is to reduce the volume of input and output. We are assembling a list that can pay for this transaction. It is often to directly traverse the list of unused transactions and make cumulative judgments. But in fact, we can use some algorithms to minimize the list of unused transactions to pay for the current transaction. This algorithm actually uses as few list items as possible to make both sides of the transaction equation Yes, according to this conclusion, the simplest way to achieve this is to sort the list in reverse order before using it

Test code:

	@Test
	public void testGetFee(){
		String address = "17A16QmavnUfCW11DAApiJxp7ARnxN5pGX";
		List<UTXO> unspents = BtcUtil.getUnspent(address, true);
		Long fee1 = BtcUtil.getFee(100 * 100000000, unspents);
		Collections.sort(unspents, (o1, o2) -> BigInteger.valueOf(o2.getValue().value).compareTo(BigInteger.valueOf(o1.getValue().value)));
		Long fee2 = BtcUtil.getFee(100 * 100000000, unspents);

		logger.warn("Miner's fee before sorting: {}, Miner's fee after sorting: {}", BigDecimal.valueOf(fee1 / 100000000.0).toPlainString(), BigDecimal.valueOf(fee2 / 100000000.0).toPlainString());
	}

Comparison results:

Miner's fee before sorting: 0.00137968, miner's fee after sorting: 0.00001808

According to this idea, I initiated two transactions in the test chain, with the transaction amount of 0.012BTC, and compared the two transactions as follows:

Before optimization:

After optimization:

It can be seen that there are two inputs before optimization and only one after optimization, and the miner's cost after optimization is less than that before optimization.

Generate wallet address

Reference code:

	public static final Map<String, String> btcGenerateBip39Wallet(String mnemonic, String mnemonicPath) {

		if (null == mnemonic || "".equals(mnemonic)) {
			byte[] initialEntropy = new byte[16];
			SecureRandom secureRandom = new SecureRandom();
			secureRandom.nextBytes(initialEntropy);
			mnemonic = generateMnemonic(initialEntropy);
		}

		String[] pathArray = mnemonicPath.split("/");
		List<ChildNumber> pathList = new ArrayList<ChildNumber>();
		for (int i = 1; i < pathArray.length; i++) {
			int number;
			if (pathArray[i].endsWith("'")) {
				number = Integer.parseInt(pathArray[i].substring(0, pathArray[i].length() - 1));
			} else {
				number = Integer.parseInt(pathArray[i]);
			}
			pathList.add(new ChildNumber(number, pathArray[i].endsWith("'")));
		}

		DeterministicSeed deterministicSeed = null;
		try {
			deterministicSeed = new DeterministicSeed(mnemonic, null, "", 0);
		} catch (UnreadableWalletException e) {
			throw new RuntimeException(e.getMessage());
		}
		DeterministicKeyChain deterministicKeyChain = DeterministicKeyChain.builder().seed(deterministicSeed).build();
		BigInteger privKey = deterministicKeyChain.getKeyByPath(pathList, true).getPrivKey();
		ECKey ecKey = ECKey.fromPrivate(privKey);
		String publickey = Numeric.toHexStringNoPrefixZeroPadded(new BigInteger(ecKey.getPubKey()), 66);

		// formal
		String mainNetPrivateKey = ecKey.getPrivateKeyEncoded(MainNetParams.get()).toString();
		Map<String, String> map = Maps.newHashMap();
		map.put("mnemonic", mnemonic);
		map.put("mainNetPrivateKey", mainNetPrivateKey);
		map.put("publickey", publickey);
		map.put("address", ecKey.toAddress(MainNetParams.get()).toString());
		return map;
	}

Test code:

	@Test
	public void testGenerateBtcWallet(){
		Map<String, String> map = AddrUtil.btcGenerateBip39Wallet(null, Constants.BTC_MNEMONIC_PATH);
		String mnemonic = map.get("mnemonic");
		String privateKey = map.get("mainNetPrivateKey");
		String publicKey = map.get("publicKey");
		String address = map.get("address");
		logger.warn("address: {}, mnemonic: {}, privateKey: {}, publicKey: {}", address, mnemonic, privateKey, publicKey);
	}

The wallet address of bitcoin has a feature that can be distinguished from the formal network or the test network. Generally, the beginning of the wallet address of bitcoin is the number 1 or 3, which is the formal network, and the beginning is the test network. The wallet address of the test network and the formal network are not interconnected.

If you are interested in my article, please pay attention to my public address.

Tags: Programming Blockchain network JSON Linux

Posted on Wed, 15 Jan 2020 03:54:32 -0500 by jmboblee