Functional computing automation operation and maintenance practice 2 -- event triggered eip automatic transfer

Function calculation

Ali cloud Function calculation Is an event driven fully hosted computing service. Through function calculation, you don't need to manage infrastructure such as servers, just write code and upload it. Function calculation will prepare computing resources for you, run your code in a flexible and reliable way, and provide log query, performance monitoring, alarm and other functions. With function computing, you can quickly build any type of application and service without management and operation and maintenance. What's better, you only need to pay for the resources consumed by the actual running of the code, while there is no cost for the code not running.

Cloud monitoring

Ali cloud Cloud monitoring Provide an enterprise level open one-stop monitoring solution for cloud users out of the box. IT covers IT infrastructure monitoring, external network quality dial-up monitoring, business monitoring based on events, custom indicators and logs. To provide you with a more efficient, comprehensive and cost-effective monitoring service. < br / > cloud monitoring provides rich events, which are still in the process of enrichment( Event monitoring of cloud product system ), rich event trigger user-defined processing functions can achieve more perfect automatic operation and maintenance.

Thematic portal = > Function calculation for automatic operation and maintenance

Example scenario

Suppose that two ECS machines A, B and A are down. At this time, EIP on machine A needs to be migrated to standby machine B. This can realize the automatic migration of EIP by using the alarm and function calculation of cloud monitoring. Cloud monitoring detects the event of A down, and then triggers the function execution. The function realizes the automatic migration of EIP

Operation steps

Note: remember to set the permissions to operate ecs and eip for the role of the function's service 1

  • mock debugging

Code

# -*- coding: utf-8 -*-
import logging
import json, random, string, time
from aliyunsdkcore import client
from aliyunsdkvpc.request.v20160428.AssociateEipAddressRequest import AssociateEipAddressRequest
from aliyunsdkvpc.request.v20160428.UnassociateEipAddressRequest import UnassociateEipAddressRequest
from aliyunsdkvpc.request.v20160428.DescribeEipAddressesRequest import DescribeEipAddressesRequest
from aliyunsdkcore.auth.credentials import StsTokenCredential
LOGGER = logging.getLogger()
clt = None
def handler(event, context):
  creds = context.credentials
  sts_token_credential = StsTokenCredential(creds.access_key_id, creds.access_key_secret, creds.security_token)
  '''
  {
    "product": "ECS",
    "content": {
        "executeFinishTime": "2018-06-08T01:25:37Z",
        "executeStartTime": "2018-06-08T01:23:37Z",
        "ecsInstanceName": "timewarp",
        "eventId": "e-t4nhcpqcu8fqushpn3mm",
        "eventType": "InstanceFailure.Reboot",
        "ecsInstanceId": "i-bp18l0uopocfc98xxxx" 
    },
    "resourceId": "acs:ecs:cn-hangzhou:123456789:instance/i-bp18l0uopocfc98xxxx",
    "level": "CRITICAL",
    "instanceName": "instanceName",
    "status": "Executing",
    "name": "Instance:SystemFailure.Reboot:Executing", 
    "regionId": "cn-hangzhou"
  }
  '''
  evt = json.loads(event)
  content = evt.get("content");
  ecsInstanceId = content.get("ecsInstanceId");
  regionId = evt.get("regionId");
  
  global clt
  clt = client.AcsClient(region_id=regionId, credential=sts_token_credential)
  
  name = evt.get("name");
  eipId = "eip-bp1nexxxc7zjfnex6i0";
  standbyEcsId = "i-bp19yycxx7yroukxpv"
  name = name.lower()
  
  if name in ['Disk:Stalled:Executing'.lower(), 'Instance:SystemFailure.Reboot:Executing'.lower(), "Instance:InstanceFailure.Reboot:Executing".lower()]:
    print("move eip to standbyEcs");
    move_eip(ecsInstanceId, standbyEcsId, eipId) # move eip to standbyEcs
  else:
    # other event to do 
    pass
  
def getEipStatus(eip):
  request = DescribeEipAddressesRequest()
  request.set_AllocationId(eip)
  request.add_query_param("RegionId", "cn-hangzhou")
  response = _send_request(request)
  if isinstance(response, dict) and "RequestId" in response:
    EipAddresses = response.get('EipAddresses', {})
    EipAddress = EipAddresses['EipAddress'][0]
    status = EipAddress['Status']
    return status
  else:
    LOGGER.error("getEipAddressDesc {} fail".format(eip))
def unAssociateEip(ecs_id, eip):
  request = UnassociateEipAddressRequest()
  request.set_AllocationId(eip)
  request.set_InstanceType('EcsInstance')
  request.set_InstanceId(ecs_id)
  response = _send_request(request)
  if isinstance(response, dict) and "RequestId" in response:
    LOGGER.info("UnassociateEipAddress {} from {} succ".format(ecs_id, eip))
  else:
    LOGGER.error("UnassociateEipAddress {} from {} fail".format(ecs_id, eip))
def associateEip(ecs_id, eip):
  associte_request = AssociateEipAddressRequest()
  associte_request.set_AllocationId(eip)
  associte_request.set_InstanceType('EcsInstance')
  associte_request.set_InstanceId(ecs_id)
  associte_response = _send_request(associte_request)
  if isinstance(associte_response, dict) and "RequestId" in associte_response:
    LOGGER.info("AssociateEipAddress {} to {} succ".format(eip, ecs_id))
    return True
  return False
def move_eip(from_ecs, to_ecs, eip):
  unAssociateEip(from_ecs, eip)
  # wait unAssociateEip ...
  time.sleep(3)
  # retry 30s util sucess
  for i in xrange(10):
    eip_status = getEipStatus(eip).lower()
    if eip_status == 'available':
      if associateEip(to_ecs, eip):
        LOGGER.info("AssociateEipAddress {} to {} succ".format(eip, to_ecs))
        return
    else:
      LOGGER.info("eip status = {}".format(eip_status))
      time.sleep(3)
  LOGGER.info("AssociateEipAddress {} to {} fail".format(eip, to_ecs))
# send open api request
def _send_request(request):
    request.set_accept_format('json')
    try:
        response_str = clt.do_action_with_exception(request)
        LOGGER.info(response_str)
        response_detail = json.loads(response_str)
        return response_detail
    except Exception as e:
        LOGGER.error(e)

Tags: JSON network

Posted on Mon, 02 Dec 2019 22:18:19 -0500 by cab