Compatibility of protobuf version of TensorFlow

It mainly solves the problem of version mismatch and error reporting:
This file was generated by an older version of protoc which is incompatible with your Protocol Buffer headers. Please regenerate this file with a newer version of protoc.

Problem Description:

When using tensorflow, an error will be reported if the protoc ol version is inconsistent, because the protobuf version used in tensorflow-1.15.5 is 3.8.0, but other dependent libraries we use are compiled based on 3.9.0, so they need to be consistent.

#error This file was generated by an older version of protoc which is
#error incompatible with your Protocol Buffer headers.  Please
#error regenerate this file with a newer version of protoc.

TensorFlow compiles using its own Protocol Buffer version. If you want to mix your own C + + code with the header file generated by TensorFlow, you need to use the same version.
Please use the same version with tensorflow to recompile your proto files.

There are two solutions:

1. Reinstall protobuf and compile other dependencies with the same version of protobuf as TensorFlow.
If TensorFlow is compiled using the source code (, Bazel will download the corresponding version in TensorFlow / TensorFlow / contrib / makefile / Gen / protobuf / bin / protocol). You can view the version number through the protocol – version, and then change your protobuf version to the same (reinstall protobuf)
2. Modify the native dependency of tensorflow. When compiling tensorflow, use your own protoc ol version to generate relevant header files of tensorflow.
The library version used by TensorFlow is defined in tensorflow/workspace.bzl. In principle, as long as it is compatible with TensorFlow and all other dependencies, it should be possible to generate a custom build of TensorFlow.

Specific solution steps

The second method is adopted here. The specific operations are as follows:
The protobuf version used by TensorFlow is downloaded from the image warehouse or the specified official website address. The download address is defined in tensorflow-1.15.5/tensorflow/workspace.bzl:
However, this address does not contain all the protobuf versions, only the protobuf versions corresponding to the existing versions of TensorFlow.

Modify the specification of protobuf version in tensorflow/workspace.bzl, calculate the corresponding sha256 and fill in the corresponding variables.
Note that you need to add the corresponding protobuf version to the image warehouse of tensorflow and keep it in the front. Because:

tf_http_archive(urls) must have redundant URLs. The URL must be present and it must come first. Even if you don't have permission to mirror the file, please put the correctly formatted mirror URL there anyway, because someone will come along shortly thereafter and mirror the file.

    # 310ba5ee72661c081129eb878c1bbcec936b20f0 is based on 3.8.0 with a fix for protobuf.bzl.
        # "",
        # "",
    #PROTOBUF_SHA256 = "b9e92f9af8819bbbc514e2902aec860415b70209f31dfc8c4fa72515a5df9d59"
    PROTOBUF_SHA256 = "725730117ea1722de5e2f99fe0d9de1cd2c4e48ff2c298fc7a2d8cb4e48811ef"
    #PROTOBUF_STRIP_PREFIX = "protobuf-310ba5ee72661c081129eb878c1bbcec936b20f0"
    PROTOBUF_STRIP_PREFIX = "protobuf-3.9.0"

    # protobuf depends on @zlib, it has to be renamed to @zlib_archive because "zlib" is already
    # defined using bind for grpc.
    PROTOBUF_PATCH = "//third_party/protobuf:protobuf.patch"

        name = "com_google_protobuf",
        patch_file = clean_dep(PROTOBUF_PATCH),
        sha256 = PROTOBUF_SHA256,
        strip_prefix = PROTOBUF_STRIP_PREFIX,
        system_build_file = clean_dep("//third_party/systemlibs:protobuf.BUILD"),
        system_link_files = {
            "//third_party/systemlibs:protobuf.bzl": "protobuf.bzl",
        urls = PROTOBUF_URLS,

In addition, tensorflow will make some modifications to protobuf according to the usage, which will take effect through git patch (/ home/svn/tensorflow-1.15.5/third_party/protobuf/protobuf.patch)
The following figure shows two modifications made by tensorflow to protobuf-3.8.0. However, we want to use 3.9.0, so we need to modify it accordingly. We only keep the renaming of zlib in the first place, because the second place has been updated in 3.9.0.

reference resources:

Tags: Linux Back-end AI TensorFlow

Posted on Fri, 29 Oct 2021 08:26:54 -0400 by Vebut