storm code reading

2021SC@SDUSC

storm code reading (III)

2021SC@SDUSC

Implementation of some interfaces in topology builder

ConfigGetter

ConfigGetter is a class defined in TopologyBuilder.java. It implements the ComponentConfigurationDeclarer interface and inherits from the BaseConfigurationDeclarer class.

ConfigGetter function: set the configuration item in the program, overwrite the default configuration item, and the format of the configuration item is JSON (essentially changing the value of json_conf in the corresponding ComponentCommon object).

The specific codes are as follows:

    protected class ConfigGetter<T extends ComponentConfigurationDeclarer> extends BaseConfigurationDeclarer<T> {
        String id;

        public ConfigGetter(String id) {
            this.id = id;
        }

        @SuppressWarnings("unchecked")
        @Override
        public T addConfigurations(Map<String, Object> conf) {
            if (conf != null) {
                if (conf.containsKey(Config.TOPOLOGY_KRYO_REGISTER)) {
                    throw new IllegalArgumentException("Cannot set serializations for a component using fluent API");
                }
                if (!conf.isEmpty()) {
                    String currConf = commons.get(id).get_json_conf();
                    commons.get(id).set_json_conf(mergeIntoJson(parseJson(currConf), conf));
                }
            }
            return (T) this;
        }

        /**
         * return the current component configuration.
         *
         * @return the current configuration.
         */
        @Override
        public Map<String, Object> getComponentConfiguration() {
            return parseJson(commons.get(id).get_json_conf());
        }

        @Override
        public T addResources(Map<String, Double> resources) {
            if (resources != null && !resources.isEmpty()) {
                String currConf = commons.get(id).get_json_conf();
                Map<String, Object> conf = parseJson(currConf);
                Map<String, Double> currentResources =
                    (Map<String, Double>) conf.computeIfAbsent(Config.TOPOLOGY_COMPONENT_RESOURCES_MAP, (k) -> new HashMap<>());
                currentResources.putAll(resources);
                commons.get(id).set_json_conf(JSONValue.toJSONString(conf));
            }
            return (T) this;
        }

        @SuppressWarnings("unchecked")
        @Override
        public T addResource(String resourceName, Number resourceValue) {
            Map<String, Object> componentConf = parseJson(commons.get(id).get_json_conf());
            Map<String, Double> resourcesMap = (Map<String, Double>) componentConf.computeIfAbsent(
                Config.TOPOLOGY_COMPONENT_RESOURCES_MAP, (k) -> new HashMap<>());

            resourcesMap.put(resourceName, resourceValue.doubleValue());

            return addConfiguration(Config.TOPOLOGY_COMPONENT_RESOURCES_MAP, resourcesMap);
        }

        @SuppressWarnings("unchecked")
        @Override
        public T addSharedMemory(SharedMemory request) {
            SharedMemory found = sharedMemory.get(request.get_name());
            if (found != null && !found.equals(request)) {
                throw new IllegalArgumentException("Cannot have multiple different shared memory regions with the same name");
            }
            sharedMemory.put(request.get_name(), request);
            Set<String> mems = componentToSharedMemory.computeIfAbsent(id, (k) -> new HashSet<>());
            mems.add(request.get_name());
            return (T) this;
        }
    }

Where id represents the unique identifier of the component. In normal non transaction flow processing, the serialization method of components cannot be set, but the system default serialization method can only be used. ConfigGetter will overwrite the default configuration item of the component according to the newly set configuration item. Finally, the configuration item will be serialized into JSON format.

SpoutGetter

Spotgetter implements basically the same functions as CinfigGetter, except that spotter does not need to declare the input.

The specific codes are as follows:

    protected class SpoutGetter extends ConfigGetter<SpoutDeclarer> implements SpoutDeclarer {
        public SpoutGetter(String id) {
            super(id);
        }
    }

BoltGetter

BoltGetter inherits from ConfigGetter, realizes the configuration customization of Bolt, implements the boltdeclar interface, and overloads various grouping methods (such as fieldsGrouping and shuffleGrouping) in boltdeclar (inputeclar). The implementation of grouping method is essentially based on_ In commons, find the corresponding ComponentCommon object through the boltId used, and set the inputs property.

The specific codes are as follows:

    protected class BoltGetter extends ConfigGetter<BoltDeclarer> implements BoltDeclarer {
        private String boltId;

        public BoltGetter(String boltId) {
            super(boltId);
            this.boltId = boltId;
        }

        public BoltDeclarer fieldsGrouping(String componentId, Fields fields) {
            return fieldsGrouping(componentId, Utils.DEFAULT_STREAM_ID, fields);
        }

        public BoltDeclarer fieldsGrouping(String componentId, String streamId, Fields fields) {
            return grouping(componentId, streamId, Grouping.fields(fields.toList()));
        }

        public BoltDeclarer globalGrouping(String componentId) {
            return globalGrouping(componentId, Utils.DEFAULT_STREAM_ID);
        }

        public BoltDeclarer globalGrouping(String componentId, String streamId) {
            return grouping(componentId, streamId, Grouping.fields(new ArrayList<String>()));
        }

        public BoltDeclarer shuffleGrouping(String componentId) {
            return shuffleGrouping(componentId, Utils.DEFAULT_STREAM_ID);
        }

        public BoltDeclarer shuffleGrouping(String componentId, String streamId) {
            return grouping(componentId, streamId, Grouping.shuffle(new NullStruct()));
        }

        public BoltDeclarer localOrShuffleGrouping(String componentId) {
            return localOrShuffleGrouping(componentId, Utils.DEFAULT_STREAM_ID);
        }

        public BoltDeclarer localOrShuffleGrouping(String componentId, String streamId) {
            return grouping(componentId, streamId, Grouping.local_or_shuffle(new NullStruct()));
        }

        public BoltDeclarer noneGrouping(String componentId) {
            return noneGrouping(componentId, Utils.DEFAULT_STREAM_ID);
        }

        public BoltDeclarer noneGrouping(String componentId, String streamId) {
            return grouping(componentId, streamId, Grouping.none(new NullStruct()));
        }

        public BoltDeclarer allGrouping(String componentId) {
            return allGrouping(componentId, Utils.DEFAULT_STREAM_ID);
        }

        public BoltDeclarer allGrouping(String componentId, String streamId) {
            return grouping(componentId, streamId, Grouping.all(new NullStruct()));
        }

        public BoltDeclarer directGrouping(String componentId) {
            return directGrouping(componentId, Utils.DEFAULT_STREAM_ID);
        }

        public BoltDeclarer directGrouping(String componentId, String streamId) {
            return grouping(componentId, streamId, Grouping.direct(new NullStruct()));
        }

        private BoltDeclarer grouping(String componentId, String streamId, Grouping grouping) {
            commons.get(boltId).put_to_inputs(new GlobalStreamId(componentId, streamId), grouping);
            return this;
        }

        @Override
        public BoltDeclarer grouping(GlobalStreamId id, Grouping grouping) {
            return grouping(id.get_componentId(), id.get_streamId(), grouping);
        }

        @Override
        public BoltDeclarer partialKeyGrouping(String componentId, Fields fields) {
            return customGrouping(componentId, new PartialKeyGrouping(fields));
        }

        @Override
        public BoltDeclarer partialKeyGrouping(String componentId, String streamId, Fields fields) {
            return customGrouping(componentId, streamId, new PartialKeyGrouping(fields));
        }

        @Override
        public BoltDeclarer customGrouping(String componentId, CustomStreamGrouping grouping) {
            return customGrouping(componentId, Utils.DEFAULT_STREAM_ID, grouping);
        }

Because there are many and similar grouping methods, we take fieldsGrouping as an example.

The boltId variable defined in the second line is used to hold the unique identifier of the Bolt component.
The next BoltGetter is the constructor, which calls the constructor of ConfigGetter to realize the customization function of configuration items.
Next, fieldsGrouping and its overload method are defined. The parameters include componentId, streamId and fields (list of field names used for grouping). If there is no specified streamId, default will be used as streamId by default.

The following boltdeclar grouping is the main implementation method of the inputeclar interface. The input meaning is: according to the grouping method defined by the grouping parameter, the streams defined with streamId in the component specified by the parameter componentId are placed in the input definition of this component. The return this pointer in the last line of the method can add more inputs to this component through function cascade.
This is followed by an important overload of the method. The input is GlobalStreamId and grouping method.

Tags: storm

Posted on Thu, 14 Oct 2021 14:58:33 -0400 by nonphixion