Yarp makes scheduling in the system more flexible

brief introduction

Yarp is a reverse proxy component developed by the Microsoft team. In addition to the conventional http and https conversion communication, its biggest feature is that it can be customized, and it is easy to develop customized proxy channels according to specific scenarios.

Details: https://devblogs.microsoft.com/dotnet/announcing-yarp-1-0-release

Source code warehouse: https://github.com/microsoft/reverse-proxy

Document address: https://microsoft.github.io/reverse-proxy/

 

Basic use

1. Create ASP.NET Core   Empty item

Using Visual Studio:

 
Create using the. NET CLI command line:
dotnet new web -o MyProxy

2. Modify the code Program.cs file

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddReverseProxy()
    .LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"));
var app = builder.Build();
app.MapGet("/Ping", () => "Hello World!");
app.MapReverseProxy();
app.Run();

3. Modify the configuration file appsettings.json

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "ReverseProxy": {
    "Routes": {
      "routeAll": {
        "ClusterId": "clusterBaidu",
        "Match": {
          "Path": "{**catch-all}"
        }
      }
    },
    "Clusters": {
      "clusterBaidu": {
        "Destinations": {
          "baidu": {
            "Address": "https://www.baidu.com/"
          }
        }
      }
    }
  }
}

The configuration here is to forward all requests to Baidu.

In Program.cs, a Get route Ping is also registered.

4. Start project

You can see that Baidu's page is displayed after the port number monitored by the browser access program. Open F12 and see that the request header is also local, not Baidu's domain name.

Test manually registered route Ping:

Can display normal.

5. Problem sorting

(1) Can Yarp only do this simple forwarding?

No, there is a description of the configuration file.

(2) Is there anything to pay attention to in the JSON configuration file?

have In the configuration file of this demonstration, the corresponding values of ReverseProxy:Clusters:cluster1:Destinations:destination1:Address are: https://www.baidu.com/ , if you remove WWW, you will jump to Baidu home page after the project is started, not agent forwarding. Removing the / compliance at the end has no effect.

(3) Will Yarp affect the routes registered in the program?

The routing registered within the program will not be affected. In Program.cs, no matter app.MapReverseProxy(); When you visit Ping, you return to Hello World at the top or bottom!

var app = builder.Build();
app.MapReverseProxy();
app.MapGet("/Ping", () => "Hello World!");
app.Run();

 

Advanced exploration

1. Multi address proxy

  Modify the configuration file appsettings.json to realize the default route jump to Baidu. When accessing / movie, visit station b.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "ReverseProxy": {
    "Routes": {
      "routeBaidu": {
        "ClusterId": "clusterBaidu",
        "Match": {
          "Path": "{**catch-all}"
        }
      },
      "routeBiliBili": {
        "ClusterId": "clusterBiliBili",
        "Match": {
          "Path": "/movie/{**catch-all}"
        }
      }
    },
    "Clusters": {
      "clusterBaidu": {
        "Destinations": {
          "baidu": {
            "Address": "https://www.baidu.com/"
          }
        }
      },
      "clusterBiliBili": {
        "Destinations": {
          "bilibili": {
            "Address": "https://www.bilibili.com/"
          }
        }
      }
    }
  }
}

  Test results:

After entering the route / movie later, you can jump to station b. However, the web page of station b is not fully displayed and there are no pictures. This is a policy problem on the website. There are no such problems for the data interface.

A detailed description of the configuration file can be viewed https://microsoft.github.io/reverse-proxy/articles/config-files.html

2. Rule matching

There are too many resources on the web page. In order to facilitate testing, two api interfaces are enabled. The addresses are: http://localhost:5241/ And https://localhost:7184/

Register the / test route in the two api interfaces.

// http://localhost:5241/
app.MapGet("/test", () => "Welcome to Api111!");

// https://localhost:7184/
app.MapGet("/test", () => "Welcome to Api222!");

Start two api programs.

C:\Users\Test>curl http://localhost:5241/test
Welcome to Api111!

C:\Users\Test>curl https://localhost:7184/test
Welcome to Api222!

Modify the configuration file appsettings.json of the MyProxy project  

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "ReverseProxy": {
    "Routes": {
      "routeOne": {
        "ClusterId": "clusterOne",
        "Match": {
          "Path": "/test/{**catch-all}",
          "QueryParameters": [
            {
              "Name": "number",
              "Values": [ "1" ]
            }
          ]
        }
      },
      "routeTwo": {
        "ClusterId": "clusterTwo",
        "Match": {
          "Path": "/test/{**catch-all}",
          "QueryParameters": [
            {
              "Name": "number",
              "Values": [ "2" ]
            }
          ]
        }
      },
      "routeBaidu": {
        "ClusterId": "clusterBaidu",
        "Match": {
          "Path": "{**catch-all}"
        }
      }
    },
    "Clusters": {
      "clusterOne": {
        "Destinations": {
          "apiOne": {
            "Address": "http://localhost:5241/"
          }
        }
      },
      "clusterTwo": {
        "Destinations": {
          "apiTwo": {
            "Address": "https://localhost:7184/"
          }
        }
      },
      "clusterBaidu": {
        "Destinations": {
          "baidu": {
            "Address": "https://www.baidu.com/"
          }
        }
      }
    }
  }
}

Path: listen for routing addresses.

QueryParameters: match parameters.

QueryParameters:Name: parameter name.

QueryParameters:Values: parameter values.

The listening port of MyProxy is http://localhost:5024/ The results are as follows:

C:\Users\Test>curl http://localhost:5024/ping
Hello World!

C:\Users\Test>curl http://localhost:5024/test
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test was not found on this server.</p>
</body></html>

C:\Users\Test>curl http://localhost:5024/test?number=1
Welcome to Api111!

C:\Users\Test>curl http://localhost:5024/test?number=2
Welcome to Api222!

It can guide the corresponding address according to the parameters and parameter values.

3. Problem sorting

(1) Why can't / movie display web pages normally.

Because some interfaces of station b have opened the anti-theft chain and cross domain detection.

(2) In parameter matching, what happens if the matched routes are the same, the monitored parameters are the same, and the parameter values are the same?

An error will be reported when accessing the routing address.

(3) Priority of route matching?

The routing priority registered in the program is the highest, followed by the one loaded by Yarp in the configuration file.

 

Small trial ox knife

Overtime is written in business, to be completed.

 

Stepping on the pit collection

1,non-ASCII 

If the project wants to represent a web page, when using the download function, the interface returns 502.

info: Yarp.ReverseProxy.Forwarder.HttpForwarder[48]
      ResponseHeaders: The destination returned a response that cannot be proxied back to the client.
      System.InvalidOperationException: Invalid non-ASCII or control character in header: 0x00E4
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpHeaders.ThrowInvalidHeaderCharacter(Char ch)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpHeaders.ValidateHeaderValueCharacters(StringValues headerValues)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseHeaders.SetValueFast(String key, StringValues value)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpHeaders.Microsoft.AspNetCore.Http.IHeaderDictionary.set_Item(String key, StringValues value)
         at Yarp.ReverseProxy.Forwarder.HttpTransformer.CopyResponseHeaders(HttpHeaders source, IHeaderDictionary destination)
         at Yarp.ReverseProxy.Forwarder.HttpTransformer.TransformResponseAsync(HttpContext httpContext, HttpResponseMessage proxyResponse)
         at Yarp.ReverseProxy.Transforms.Builder.StructuredTransformer.TransformResponseAsync(HttpContext httpContext, HttpResponseMessage proxyResponse)
         at Yarp.ReverseProxy.Forwarder.HttpForwarder.SendAsync(HttpContext context, String destinationPrefix, HttpMessageInvoker httpClient, ForwarderRequestConfig requestConfig, HttpTransformer transformer)

Go to GitHub and find Issues

The download interface can be accessed normally and the file stream can be obtained completely. It's no use rewriting all the response headers. This kind of non open source commercial site can't guess the character coding.

Finally, we compromised and used a. NET service to download it on the server and then forward it.

When you proxy unconventional service interfaces, you must test more.

 

Tags: C# .NET

Posted on Mon, 29 Nov 2021 17:06:25 -0500 by hammadtariq