Add a new feature to generate krakend json from swagger json

[prakkarp]

Add functionality to the krakenD designer to generate the endpoints from the swagger json. This would reduce the time taken to generate the krakend json using the designer. Most of the cloud API gateway like AWS api gateway, apigee and others offer this functionality. So this is something very useful to have.

[fvillain]

This would be indeed very useful !! Any timeline on that ?

[kpacha]

are you talking about the swagger definitions of your backends or the swagger file describing your desired KrakenD?

in my opinion, the first case would be very hard to solve, the second one will provide just a list of endpoints with not very much details and no backends.

do you have any suggestion?

PS: the enterprise tooling includes a generator in the other direction: from a running KrakenD it generates a swagger definition for the clients of the gateway

[fvillain]

In my head it was the second option : swagger file describing our krakend configuration. This means mapping with maybe others specfiles via refs in order to generate a complete doc of the exposed API (with full information: schemas, components, ...)

Do you have some documentation readable on the entrerpise tool your talking about ? To see how it works and if it would fit our use-case ?

[mattcamp]

I too would like to see the ability to import an OpenAPI document (aka "swagger doc") to configure KrakenD.

Many other enterprise API managers (AWS API Gateway, Azure API Manager, IBM's API Connect, Apigee, etc) all support importing OpenAPI spec docs so this seems like a gap currently.

[AdityaBhansali1]

In our project, we take the backend service swagger as input to a swagger-codegen module which converts it to a kraken config file. In our case it is a 1-1 mapping with same API structure between the gateway and backend, so works well for us.

[adebree]

@AdityaBhansali1 can you elaborate on how you do that? We are looking into solving that exact same problem.

[saloni27singal]

@AdityaBhansali1 we are also trying to solve the same

[mingfang]

what was the resolution to this issue?

[ozonni]

we also need this. We have like 200 different endpoints and we add new, doing it manually seems to be a hustle

[AdityaBhansali1]

@adebree, @saloni27singal : Apologies for the delay. Below are some more details on how we consume multiple backend service swaggers to generate a final krakend configuration dynamically.

We utilize the swagger codegen project for the above transformation. This project allows generating client libraries for various languages given an open api spec as input. It also has the flexibility to customize the generator. We wrote our own Config generator by extending DefaultCodegenConfig, and used mustache templates which would result in a krakend config file.

At a high level, the customized generator automatically extracts information from open api and then injects them into the mustache template and generates the resulting krakend config file which matches the template definition. All this happens using the single command as below.

for FILE in *.json; do
	java -jar modules/swagger-codegen-cli/target/swagger-codegen-cli.jar generate \
	-i $FILE \
	-l com.pega.platform.gateway.swaggergenerator.KrakenDConfigCodegen
done

The looped runs of above command will generate several multiple krakend config files which are several parts of krakend Flexible configuration.

Finally we run krakend by running only the top level config file.

krakend run -c "global.json"

You can also pass environment variables to this command and utilize them in the mustache to generate different configs based on input properties. This is useful when your config generation happens inside a docker container initialized via a helm chart. The customization of the containers can happen during the start of containers by passing the desired environment variables.

Please let me know if you need more information.

[egandro]

Please let me know if you need more information.

Multiple questions.

1. Can you share your "com.pega.platform.gateway.swaggergenerator.KrakenDConfigCodegen" and the teimplates?

2. Can you filter routes from the downstream microservices? e.g. only certain paths or only certain HTTP Verbs?

I found this for filtering the swagger files - https://github.com/Mermade/openapi-filter - eg. tagged with "API-Mobile" or "API-Web" and you can only export this routes/operations

3. I also like to ask if you integrated a tool like swagger-ui in krakend in your setup?

I am using swaggest/rest in go - so there is a Swagger/Open API that can be provided.

https://github.com/swaggest/swgui

[1gravity]

Why was this issue closed? It's good to hear that someone has built a solution that works for them but that doesn't help anyone else. This should be out-of-the-box functionality, not every company has the resources to build their own codegen generator.

[LuciferSam86]

Why was this issue closed? It's good to hear that someone has built a solution that works for them but that doesn't help anyone else. This should be out-of-the-box functionality, not every company has the resources to build their own codegen generator.

Seeing the enterprise edition there's a Swagger generator in their Kraken Studio. I'm afraid they're not interested to bring the swagger generator into the CE, but I hope I am wrong.

[alombarte]

We don't have today any tool, Enterprise or Open-Source, that is able to generate a krakend.json from an OpenAPI spec (but it exists the other way around).

If you are interested in this functionality, please upvote the first comment. In 2 years it had less than 30 votes.

As @1gravity pointed out, not everyone has the resources to build a codegen tool. We don't think we have the time and resources either to write a generic solution for everyone, with all the possible combinations, openapi versions, etc.

What we suggest is that anyone needing this (which is actually thinking KrakenD as a no-op proxy, instead of implementing a GW as it should be) writes a custom 20 lines script to do the magic. It can be in any language. If you are scared of Go, it doesn't have to be. The following snippet is 5 minutes of work in a non-Go language. Do not see it as a ready-to-use script, neither a good practice, but a quick hack to demonstrate the simplicity of a generator.

<?php

$fc = file_get_contents('swagger.json');
$swagger = json_decode($fc, true);

// KrakenD config:
$k['version'] = 2;
$k['host'] = [( $swagger['schemes'][0] ?: 'http' ) . '://' . $swagger['host']];
$k['name'] =   $swagger['info']['title'];

$paths = array_keys($swagger['paths']);
foreach( $paths as $endpoint) {
    $methods = array_keys($swagger['paths'][$endpoint]);
    
    foreach( $methods as $method) {
        $k["endpoints"][] = array(
            "endpoint" => $endpoint,
            "method" => strtoupper($method),
            "output_encoding" => "no-op",
            "backend" => [
                "encoding" => "no-op",
                "url_pattern" => $endpoint
            ]
        );
    }
}
file_put_contents('krakend.json',json_encode($k, JSON_PRETTY_PRINT|JSON_UNESCAPED_SLASHES));

[alombarte]

Moved the issue to the krakend designer as it is the place where it could be implemented

[giosueDelgado]

Is very important to have this feature because on large project that use swagger / openapi if is available the import feature is very fast to migrate and use this API Gateway.

[milanjaros]

As @alombarte described generating KrakenD JSON from swagger is "5 minutes of work" task. The devil is in the details. @giosueDelgado, the "import feature" you think about could be totally different from the "import feature" (transition) we are going to follow on our project. For example, we started to use KrakenD more than a year ago and we had a different approach than we have today. We thought we will use a lot of KrakendD features (like data manipulation, circuit breaker, caching, etc.) but we ended up with just no-op proxy. So, the use case changes from "intensive manual gateway configuration" to just "exposing private APIs to public API". As you can see, the motivation and needs could differ.

Custom gateway configuration format

Maybe it could be useful for someone (just as an inspiration) if I share a python snippet that I created as a Proof of Concept to just "simplify manual gateway configuration" for a team. Even we use templating we have like 25 *.tmpl files with the sum of 6000 lines of configuration. I wanted to try to shrink the Endpoint configuration to the minimum, so we would have just template.yml, something like:

/inbox:
  GET:
    - host: [ "http://inbox.user" ]
/profile/images:
  GET:
    - url_pattern: /images
      host: [ "http://profiles.user" ]
  PUT:
    - url_pattern: /images
      host: [ "http://profiles.user" ]
# The most complicated configuration would look like
/contracts/upload:
  POST:
    - url_pattern: /upload
      host: [ "http://contracts.files" ]
      endpoint:
        headers_to_pass: [ "{{ .default_headers }}", "Content-Encoding" ]
        querystring_params: [ "*" ]

As you can see, the structure is inspired by OpenAPI (I wanted to make it simple for both mobile and backend developers who are used to writing OAS) but on the other hand, it is a "new proprietary format" because as you know OpenAPI does not contain such things as url_pattern or host and you can't just extend it if you use code generation and validation tools. My proposal was to copy the path to url_pattern and take the rest from defaults.json which could look like:

{
    "extra_config": "{{ marshal .jwt_token_validator }}",
    "output_encoding": "no-op",
    "headers_to_pass": [ "{{ .default_headers }}" ],
    "backend": [
        {
            "host": ["{{ .host_users }}"],
            "encoding": "no-op"
        }
    ]
}

Finally, there is a script where you can enjoy precisely "chosen" Christmas tree design pattern.🎄😉 I hope that there are just enough comments. It is also fair to mention, we do not use it in production because our motivation has changed.

import oyaml as yaml
import json
import sys, copy

# Load files
if(len(sys.argv) != 3):
    exit("usage: generate_configuration.py <template.yml> <defaults.json>")
template = open(sys.argv[1])
defaults = open(sys.argv[2])
urls = yaml.safe_load(template)
default = json.load(defaults)

# set item to array and extend it if needed
def ensure_array_has_item(variable, item, index):
    if item not in variable:
        variable[item] = []
    if len(variable[item]) <= index:
        variable[item].append({})

out = []
# Append all endpoints from defined URLs
for url in urls:
    endpoint = {}
    # Load defaults
    for prop in default:
        endpoint[prop] = copy.deepcopy(default[prop])
    # Update endpoints from defined HTTP methods and their items
    methods = urls[url]
    for method in methods:
        # This does he openapi.yaml to krakend.json transformation
        endpoint["endpoint"] = url
        endpoint["method"] = method
        items = methods[method]
        # Copy config of endpoins
        for i, item in enumerate(items):
            ensure_array_has_item(endpoint, "backend", i)
            endpoint["backend"][i]["method"] = method
            for prop in item:
                # Set all properties of a HTTP method
                if prop != "endpoint":
                    endpoint["backend"][i][prop] = item[prop]
                else:
                    # Set endpoint properties to (parent) endpoint
                    for endpoint_prop in item[prop]:
                        endpoint[endpoint_prop] = item[prop][endpoint_prop]
            # Set URL from (parent) endpoint to endpoint backend if not set
            if "url_pattern" not in endpoint["backend"][i]:
                endpoint["backend"][i]["url_pattern"] = url

    out.append(endpoint)

print(json.dumps(out, indent=4))

[HamzaOralK]

I am writing a piece of code that converts OpenAPI definition to krakend.json for our APIs. Our microservices serve their API definition internally so every update to API definition will restart the gateways as well and gateway will pick up new definition. It shows a lot of promise though, still in PoC mode though.

Created something like this. One can compile or use the image there to dynamically download necessary swagger files and convert them into krakend and boot up the krakend itself.

[alombarte]

This functionality exists already in the Enterprise edition, and allows generating a KrakenD configuration by passing an openapi spec:

$krakend generate from openapi -c ~/Downloads/openapi.yaml -o krakend.json

https://www.krakend.io/docs/enterprise/developer/openapi/

From our side I don't see this feature ported to the designer any time soon, but if there is any contributor willing to do it please send a PR.