Implementation of JAVA Query Based on ipv6 Static Data File of zxinc Website

Recently there is a need to resolve ipv6 addresses. Because of the large amount of data, it is not possible to request a pre-used http query interface. This will certainly seriously affect use, so we found it when searching on the web zxinc This website provides offline files for query, and then downloads the latest offline ipv6 data. There are also some programming language (python,php,c++) reader implementations inside, but there is no implementation of JAVA I want. So referring to python's implementation, I implemented the Java version of reading data by myself.Share it here for everyone to use. If there are any problems with the code or what can be optimized, please correct them.

ipv6

First, let's introduce the composition of the IPv6 address: the IPv6 address is 128 bits, which means that the IPv6 address is composed of 1280 or 1 bits, divided into 8 blocks with 16 bits each.Each block can be represented as four hexadecimal numbers separated by a colon':'.
There are two rules for ipv6 address writing. Take FF01:0000:3EE8:DFE1:0063:0000:0000:F001 as an example to illustrate these two rules:

1. Remove leading 0
   Result after removal FF01:0000:3EE8:DFE1:63:0000:0000:F001
2. If more than two blocks contain consecutive zeros, all blocks are omitted and replaced by':'. If there are all zeros, they can be abbreviated as a zero
   The results were FF01:0:3EE8:DFE1:63::F001

BigInteger

Before you implement modern code, familiarize yourself with BigInteger, such as big integer literally. In java, the int type takes up four bytes, the long type takes up eight bytes, and each byte has eight bits. So the maximum value that an int can express is 2^(48-1) - 1=2147483647, while the long type can express is 2^(88-1) - 1 = 9223372036854775807.While we are doing ipv6 parsing, we often need to do left shift operation on long type. At this time, it is possible that the number of displacements will exceed the range that long type can express. At this time, the result will be displayed in negative form. This result is naturally not what we want, so we should use BigInteger class to do bit operation on ipv6 parsing.Below is a list of BigInteger's bitwise operation-related methods. Refer to this article for BigInteger: Java Large Number High Precision Function (BigInteger)

BigInteger n = new BigInteger( String ); BigInteger m = new BigInteger( String );

ipv6 to Long

ipv6 to long type, direct reference zxinc The python code in replaces any possible overflow

public BigInteger ipv6ToNum(String ipStr) { // String ipStr = "2400:3200::1"; // Up to 8 segments separated by a colon int ipCount = 8; List<String> parts = new ArrayList<>(Arrays.asList(ipStr.split(":"))); // At least three paragraphs, such as: `::1` if (parts.size() < 3) { System.out.println("error ip address"); } String last = parts.get(parts.size() - 1); if (last.contains(".")) { long l = ipv4ToNum(last); parts.remove(parts.size() - 1); // (l >> 16) & 0xFFFF; parts.add(new BigInteger(((l >> 16) & 0xFFFF) + "").toString(16)); parts.add(new BigInteger((l & 0xFFFF) + "").toString(16)); } int emptyIndex = -1; for (int i = 0; i < parts.size(); i++) { if (StringUtils.isEmpty(parts.get(i))) { emptyIndex = i; } } int parts_hi, parts_lo, parts_skipped; if (emptyIndex > -1) { parts_hi = emptyIndex; parts_lo = parts.size() - parts_hi - 1; if (StringUtils.isEmpty(parts.get(0))) { parts_hi -= 1 ; if (parts_hi > 0) { System.out.println("error ip address"); } } if (StringUtils.isEmpty(parts.get(parts.size() - 1))) { parts_lo -= 1; if (parts_lo > 0) { System.out.println("error ip address"); } } parts_skipped = ipCount - parts_hi - parts_lo; if (parts_skipped < 1) { System.out.println("error ip address"); } } else { // Full Address if (parts.size() != ipCount) { System.out.println("error ip address"); } parts_hi = parts.size(); parts_lo = 0; parts_skipped = 0; } BigInteger ipNum = new BigInteger("0"); if (parts_hi > 0) { for (int i = 0; i < parts_hi; i++) { ipNum = ipNum.shiftLeft(16); String part = parts.get(i); if (part.length() > 4) { System.out.println("error ip address"); } BigInteger bigInteger = new BigInteger(part, 16); int i1 = bigInteger.intValue(); if (i1 > 0xFFFF) { System.out.println("error ip address"); } ipNum = ipNum.or(bigInteger); } } ipNum = ipNum.shiftLeft(16 * parts_skipped); for (int i = -parts_lo; i < 0; i++) { // ipNum <<= 16; ipNum = ipNum.shiftLeft(16); String part = parts.get(parts.size() + i); if (part.length() > 4) { System.out.println("error ip address"); } BigInteger bigInteger = new BigInteger(part, 16); int i1 = bigInteger.intValue(); if (i1 > 0xFFFF) { System.out.println("error ip address"); } // ipNum |= i1; ipNum = ipNum.or(bigInteger); } System.out.println(ipNum); return ipNum; }

When working with ipv4, we usually turn byte[] back to long, but when working with ipv6, we can't handle long because there may be an overflow, so we use BigInteger, which we described earlier.

private BigInteger byteArrayToBigInteger(byte[] b) { BigInteger ret = new BigInteger("0"); // Loop Read Each Byte Completes long's 8 Byte Assembly by Shift for(int i = 0; i < b.length; i++){ // value |=((long)0xff << shift) & ((long)b[i] << shift); int shift = i << 3; BigInteger shiftY = new BigInteger("ff", 16); BigInteger data = new BigInteger(b[i] + ""); ret = ret.or(shiftY.shiftLeft(shift).and(data.shiftLeft(shift))); } return ret; }

It is the statement commented here that converts 0xff to long, causing a number overflow that has stuck me here for a long time.It's best to use BigInteger for all of them.
Here, the byte array is converted to BigInteger for reference: java:bytes[] Three ways to long The first method in the article.
Here, byteArrayToBigInteger is equivalent to the code below, which is written in a simpler way, while the method below, which uses long as the final result, runs the risk of overflow:

//This is how binary is converted to long type public static long getLongAt(byte[] buffer,int offset) { long value = 0; //The first value corresponds to the long type above converted to binary. //First remove the first left 64 bits and FF000000 phase is the eight bits, then phase or the original eight bits value |= buffer[offset + 0] << 56 & 0xFF00000000000000L; value |= buffer[offset + 1] << 48 & 0x00FF000000000000L; value |= buffer[offset + 2] << 40 & 0x0000FF0000000000L; value |= buffer[offset + 3] << 32 & 0x000000FF00000000L; value |= buffer[offset + 4] << 24 & 0x00000000FF000000L; value |= buffer[offset + 5] << 16 & 0x0000000000FF0000L; value |= buffer[offset + 6] << 8 & 0x0000000000000FF0L; value |= buffer[offset + 7] & 0x00000000000000FFL; return value; }

Perhaps you need to be aware of these places, with the full code attached below:

import java.io.IOException; import java.io.RandomAccessFile; import java.io.UnsupportedEncodingException; import java.math.BigInteger; /** * ip Library source:http://ip.zxinc.org/ * All of the following data types (short/int/int64/IP address/offset address, etc.) are small endings * * File Header * 0~3 String "IPDB" * 4-5 short Version number, now 2.Version numbers 0x01 to 0xFF are guaranteed to be compatible. * 6 byte Offset address length (2~8) * 7 byte IP Address length (4 or 8 or 12 or 16, now only 4(ipv4) and 8(ipv6)) * 8~15 int64 Number of records * 16-23 int64 Offset of the first record in the index area * 24 byte Number of address fields (1~255) [new version of Grunge, default at this stage is 2] * 25-31 reserve Reserved, filled with 00 * 32~39 int64 Offset of database version string [version 2 new, version 1 not] * * Record Area * array String [Number of address fields] * andQqwry.datRoughly the same, but no end IP address * 01 Abandoned at the beginning * 02+Offset address [offset length] indicates redirection * 20~FF Begins with a normal string, encoded in UTF-8, and ends with NULL * * Index Area * struct{ * IP[IP Address Length] Start IP Address * Offset [offset length] record offset address * }Index [number of records]; * */ public class Ipv6Service { private String file = "D:\\project\\idea\\spring-boot-demo\\ipv6wry.db"; //Single mode instance private static Ipv6Service instance = new Ipv6Service(); // private RandomAccessFile randomAccessFile = null; private byte[] v6Data; // Offset Address Length private int offsetLen; // Offset of the first record in the index area private long firstIndex; // Total Records private long indexCount; public long getIndexCount() { return indexCount; } private Ipv6Service() { try(RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r")) { v6Data = new byte[(int) randomAccessFile.length()]; randomAccessFile.readFully(v6Data, 0, v6Data.length); } catch (IOException e) { System.out.println("Failed to read file!"); } // Get offset address length byte[] bytes = readBytes(6, 1); offsetLen = bytes[0]; // Offset of the first record in the index area bytes = readBytes(16, 8); BigInteger firstIndexBig = byteArrayToBigInteger(bytes); firstIndex = firstIndexBig.longValue(); // 570063 System.out.println("Offset of the first record in the index area:" + firstIndex); // Total Records bytes = readBytes(8, 8); BigInteger indexCountBig = byteArrayToBigInteger(bytes); indexCount = indexCountBig.longValue(); } /** * @return Single instance */ public static Ipv6Service getInstance() { return instance; } private byte[] readBytes(long offset, int num) { byte[] ret = new byte[num]; for(int i=0; i < num; i++) { ret[i] = v6Data[(int) (offset + i)]; } return ret; } /** * Converted little-endian byte order * byte[]Convert to long * @param b * @return ret */ private BigInteger byteArrayToBigInteger(byte[] b) { BigInteger ret = new BigInteger("0"); // Loop Read Each Byte Completes long's 8 Byte Assembly by Shift for(int i = 0; i < b.length; i++){ // value |=((long)0xff << shift) & ((long)b[i] << shift); int shift = i << 3; BigInteger shiftY = new BigInteger("ff", 16); BigInteger data = new BigInteger(b[i] + ""); ret = ret.or(shiftY.shiftLeft(shift).and(data.shiftLeft(shift))); } return ret; } /** * Binary Search * @param ip * @param l * @param r * @return */ public long find (BigInteger ip, long l, long r){ if (r - l <= 1) return l; long m = (l + r) >>> 1; long o = firstIndex + m * (8 + offsetLen); byte[] bytes = readBytes(o, 8); BigInteger new_ip = byteArrayToBigInteger(bytes); if (ip.compareTo(new_ip) == -1) { return find(ip, l, m); } else { return find(ip, m, r); } } public static long ipv4ToNum(String ipStr) { long result = 0; String[] split = ipStr.split("\\."); if (split.length != 4) { System.out.println("error ip address"); } for (int i = 0; i < split.length; i++) { int s = Integer.valueOf(split[i]); if (s > 255) { System.out.println("error ip address"); } result = (result << 8) | s; } return result; } // The ipv6ToNum method has pasted code at the top of the article. To reduce the code, remove public BigInteger ipv6ToNum(String ipStr) { BigInteger ipNum = new BigInteger("0"); return ipNum; } public long getIpOff(long findIp) { return firstIndex + findIp * (8 + offsetLen); } public long getIpRecOff(long ip_off) { byte[] bytes = readBytes(ip_off + 8, offsetLen); BigInteger ip_rec_off_big = byteArrayToBigInteger(bytes); return ip_rec_off_big.longValue(); } public String getAddr(long offset) { byte[] bytes = readBytes(offset, 1); int num = bytes[0]; if (num == 1) { // Redirection mode 1 // [IP][0x01][Absolute offset address for country and region information] // Use next 3 bytes as offset to call bytes to get information bytes = readBytes(offset + 1, offsetLen); BigInteger l = byteArrayToBigInteger(bytes); return getAddr(l.longValue()); } else { // Redirection Mode 2 + Normal Mode // [IP][0x02][Absolute offset of information][...] String cArea = getAreaAddr(offset); if (num == 2) { offset += 1 + offsetLen; } else { offset = findEnd(offset) + 1; } String aArea = getAreaAddr(offset); return cArea + "|" + aArea; } } private String getAreaAddr(long offset){ // Get the zone information string by giving an offset value byte[] bytes = readBytes(offset, 1); int num = bytes[0]; if (num == 1 || num == 2) { bytes = readBytes(offset + 1, offsetLen); BigInteger p = byteArrayToBigInteger(bytes); return getAreaAddr(p.longValue()); } else { return getString(offset); } } private String getString(long offset) { long o2 = findEnd(offset); // It is possible that only national information does not have regional information. byte[] bytes = readBytes(offset, Long.valueOf(o2 - offset).intValue()); try { return new String(bytes, "utf8"); } catch (UnsupportedEncodingException e) { return "Unknown data"; } } private long findEnd(long offset) { int i = Long.valueOf(offset).intValue(); for (; i < v6Data.length; i++) { byte[] bytes = readBytes(i, 1); if ("\0".equals(new String(bytes))) { break; } } return i; } public static void main(String[] args) { Ipv6Service ipv6Service = Ipv6Service.getInstance(); BigInteger ipNum = ipv6Service.ipv6ToNum("2409:8754:2:1::d24c:4b55"); BigInteger ip = ipNum.shiftRight(64).and(new BigInteger("FFFFFFFFFFFFFFFF", 16)); // Find index offset for ip long findIp = ipv6Service.find(ip, 0, ipv6Service.getIndexCount()); // Get indexed records long ip_off = ipv6Service.getIpOff(findIp); long ip_rec_off = ipv6Service.getIpRecOff(ip_off); String addr = ipv6Service.getAddr(ip_rec_off); System.out.println(addr); } }

The top is the complete code, which is written with reference to the python sample code given on the zxinc website.

Last

Writing this java code takes a lot of effort because you are unfamiliar with bitwise operations in binary.Understanding the form of an ipv6 address; familiarizing yourself with binary bitwise operations; and then falling into the pit where the long type overflows and coming over step by step will ultimately be a great way to achieve a lookup.If there is something wrong in the code, you are welcome to point out that we can communicate with each other, or what methods can be optimized. Thank you.

1. Java Large Number High Precision Function (BigInteger)
2. java:bytes[] Three ways to long
3. Binary in Java
4. About the Binary Conversion for Java Implementation (Bit Operations)
5. Binary and base bit operations in java
6. IPv6 Address Type

17 June 2020, 12:29 | Views: 8606