Observability: use Elastic Stack to analyze geospatial data

Previous articles“ Observability: using Elastic Stack to analyze geospatial data (1) ”, I detailed how to import data from OpenSky Network API interface to Elasticsearch and analyze the data visually. Maybe it's enough for the right situation, because it can help us extract a lot of useful things from a lot of real-time data.

In today's article, we will refer to previous articles“ How to use the Elasticsearch ingest node to enrich logs and indicators ” . We can use the elastic search iest node to enrich our data, and make further analysis of these data.

To achieve this, we must first understand the icao Field. This field means:

The ICAO airport code or position indicator is a four letter code used to designate airports around the world. These codes are defined by ICAO and published in ICAO document 7910: position indicators for use by air traffic control and airline operations (e.g. flight plans).

Each of our previous documents is like this:

{ "velocity" : 0.0, "icao" : "ad0851", "true_track" : 264.38, "time_position" : 1591190152, "callsign" : "AAL2535", "origin_country" : "United States", "position_source" : "ADS-B", "spi" : false, "request_time" : 1591190160, "last_contact" : 1591190152, "@timestamp" : "2020-06-03T13:16:03.723Z", "on_ground" : true, "location" : "32.7334,-117.2035" }

In addition, we can address https://opensky-network.org/datasets/metadata/ Find a file like this:

Here, we can find one called aircraftDatabase.csv File. Its contents are as follows:

In the table above, we find a field called icao24. This field can be associated with our previous documents so that we can get more information about a particular flight.

Create enrich index

Because the downloaded document is a csv file. We can use data visualizer to import.

Click the Override settings link above:

Click the Apply button:

Click the Import button above:

We call this index aircraft. Click Advanced:

Confirm the mapping again. If there is no problem, click the Import button:

Because this file is large, it will take a little time to import:

After that, we can find an index called aircraft in elastic search:

GET _cat/indices

As shown above, a new aircraft index has been generated.

Create Enrich policy

Next, let's create enrich policy. It tells us how to enrich data. Enter the following command in Kibana:

PUT /_enrich/policy/flights_policy { "match": { "enrich_fields": [ "acars", "adsb", "built", "category_description", "engines", "first_flight_date", "icao_aircraft_type", "line_number", "manufacturer_icao", "manufacturer_name", "model", "modes", "notes", "operator", "operator_callsign", "operator_iata", "operator_icao", "owner", "reg_until", "registered", "registration", "seat_configuration", "serial_number", "status", "test_reg", "type_code" ], "indices": [ "aircraft" ], "match_field": "icao" } }

We use execute enrich policy API Create an enrich index for this policy:

POST /_enrich/policy/flights_policy/_execute

Next, we create a new one called flights_ aircraft_ Enrich's pipeline:

PUT /_ingest/pipeline/flights_aircraft_enrichment { "description": "joins incoming ADSB state info with richer aircraft metadata", "processors": [ { "enrich": { "field": "icao", "policy_name": "flights_policy", "target_field": "aircraft" } } ] }

So far, we have successfully created rich strategies. Next, we will show how to use this pipeline to enrich our data.

Rich data

In order to use the pipeline defined above, we refer to the previous articles again“ Observability: using Elastic Stack to analyze geospatial data (1) ”Flighths in_ logstash.conf File and modify the output section as follows:

output { stdout { codec => rubydebug } elasticsearch { manage_template => "false" index => "flights" # pipeline => "flights_aircraft_enrichment" hosts => "localhost:9200" } }

Let's take out the comment on the line above:

# pipeline => "flights_aircraft_enrichment"

This becomes:

output { stdout { codec => rubydebug } elasticsearch { manage_template => "false" index => "flights" pipeline => "flights_aircraft_enrichment" hosts => "localhost:9200" } }

Before starting Logstash, we can delete the previous flights index:

DELETE flights

Then execute the following command:

PUT flights { "mappings": { "properties": { "@timestamp": { "type": "date" }, "baro_altitude": { "type": "float" }, "callsign": { "type": "keyword" }, "geo_altitude": { "type": "float" }, "icao": { "type": "keyword" }, "last_contact": { "type": "long" }, "location": { "type": "geo_point" }, "on_ground": { "type": "boolean" }, "origin_country": { "type": "keyword" }, "position_source": { "type": "keyword" }, "request_time": { "type": "long" }, "spi": { "type": "boolean" }, "squawk": { "type": "long" }, "time_position": { "type": "long" }, "true_track": { "type": "float" }, "velocity": { "type": "float" }, "vertical_rate": { "type": "float" } } } }

To rerun Logstash:

sudo ./bin/logstash -f fligths_logstash.conf

We checked flights' mapping in Kibana:

GET flights/_mapping

We can see some new fields added:

We can search:

"_source" : { "aircraft" : { "owner" : "Wells Fargo Trust Co Na Trustee", "reg_until" : "2021-04-30", "modes" : false, "built" : "1984-01-01", "acars" : false, "manufacturer_icao" : "BOEING", "serial_number" : "23018", "manufacturer_name" : "Boeing", "icao_aircraft_type" : "L2J", "operator_callsign" : "GIANT", "operator_icao" : "GTI", "engines" : "GE CF6-80 SERIES", "icao" : "a8a763", "registration" : "N657GT", "model" : "767-281", "type_code" : "B762", "adsb" : false }, "true_track" : 272.81, "velocity" : 5.14, "spi" : false, "origin_country" : "United States", "@timestamp" : "2020-06-04T10:41:00.558Z", "request_time" : 1591267250, "time_position" : 1591267168, "last_contact" : 1591267168, "callsign" : "GTI165", "icao" : "a8a763", "location" : "39.0446,-84.6505", "on_ground" : true, "position_source" : "ADS-B" } }

We can see a field called aircraft, which contains all the rich information about the aircraft.

Using Kibana to analyze data

Identify the top 10 aircraft models

Because new fields come in, we have to recreate the new inde pattern:

What we can see most is PC-12/47E.

Find out the distribution of aircraft manufacturers

We see that BOING has the largest market share. AIRBUS is in second place.

Aircraft age distribution

We can see that the most airplanes are produced in 2019.

Relationship between aircraft type and flight altitude

It can be seen that the A320-214 aircraft flies the highest.

Graph

Use Graph to find the direct relationship between data. If you don't know much about Graph, please refer to my previous tutorial“ Introduction to Elastic Graph".