Feature request: dedupe emitted types referenced from multiple files

[evert]

trafficstars

Apologies if this has been asked before, I did a few searches but couldn't quite find this.

I have a directory of schemas. Each of those schemas sometimes share references to common definitions.

This tool automatically expands the references, and they all get in-lined (and thus duplicated). It would be nice if references could be transformed to a typescript type, and referenced from a common file.

For example, 2 files might use:

{
  "myProp": { "$ref": "defs/my-prop.json" }
}

It would be really great if this were transformed to:

import { MyProp } from './defs/my-prop';

interface SomeObj {
  myProp: MyProp
}

[jylauril]

I'll second this request! This would be really great to be able to reference the types with imports instead of literally duplicating it in every file.

Our use case is that we have about 80 or so JSON schema files that we need to individually convert, and a lot of them are referencing other files. I've been thinking about adding some post processing steps to manually fix those, but better option would definitely be if json-schema-to-typescript would support this!

[zlk89]

+1 It would be great to have this feature and an API to convert all files under a directory in batch. We have more than 1k json schema files, and duplicating shared references make output size explode

[smil2k]

The closest I found so far was:

"elements": { "type": "array", "items": { "$ref": "Element.json", "tsType": "Element" } },

Additionally my .js script puts everything into one definition file at the end, so it is kind of working as you would expect.

[NathanJAdams]

Totally agree, I'm having to work around this at the moment. Importing is a pretty standard pattern so it's surprising everything gets inlined.

[bcherny]

Proposals for how to do this welcome! How exactly should this work when using multiple input files, multiple output files, etc.?

[NathanJAdams]

Writing this with the caveat that I've barely glanced at the code so I may be making incorrect assumptions.

With multiple output files, one option would be to output every type, interface, enum etc. into it's own file and add a relative import for every unique $ref.

With multiple input files, an option would be to look for existing files and only create one if it doesn't already exist. Although that does introduce the assumption that $refs with an identical path contain identical contents. Keeping unique file names might get tricky if that assumption can't be made. Another option would be to create a subfolder for each input file (if needed) and put unique and identical files in the base folder and identically named but different files in the specific subfolder they're referenced from. Something similar to the dependency structure that yarn uses when installing.

I'd be happy to make a PoC if it helps?

Edit Just looked at the code and seen that compiling returns a string Promise. In the index file, another function could be added that returns a Map of content keyed by the $ref path. It's a similar idea to the above just with the $ref paths being used as the content key rather than creating files itself.

[NathanJAdams]

I've had to make my own utility to get around this

[Roaders]

This is a highly desired feature for me as well. Having the same low level type (such as IEmployee) duplicated in multiple different files is a big pain. It means for a start that you can't export everything from multiple files with a barrel. Assuming that all instances of IEmployee are the same would be fine for me. There are further complications though. For example if I am generating types for IOffice we could call the file office.d.ts and put all references in references.d.ts. Other files could also use references.d.ts such as ICompany that also contains IEmployee. The problem comes when ICompany also references IOffice which is in it's own file, not in the references file.

[Roaders]

I have created a tool that will take the d.ts files created by json-schema-to-typescript and remove duplicated types: ts-dedupe:

https://www.npmjs.com/package/ts-dedupe

I have only tested it on my project and it works fine for me but I can envisage lots of edge cases in something like this. I'd appreciate it if anyone with the same issue as me could give this a try and create issues for anything that doesn't work.

running it is as simple as:

ts-dedupe -d duplicatesFile.ts

although you might want to create a tsconfig specifically for the files generated by json-schema-to-typescript:

ts-dedupe -d duplicatesFile.ts -p tsconfig.generated.json

any feedback very welcome

[nabati]

How about a simpler version of this @bcherny;

If you specify multiple input files but output everything to a single file you just keep track of which names have already been generated and which have not. In case of conflict, the first definition would be used.

What do you think?

[roboncode]

Per suggestion of @nabati I created a file in my tools/merge-types.js

Pre-requisites:

npm i -D glob minimist

tools/merge-types.js

const glob = require('glob')
const path = require('path')
const fs = require('fs')
const argv = require('minimist')(process.argv.slice(2));
const typesPath = path.join(__dirname, '..', 'src', 'types')

glob(typesPath + '/*.d.ts', (err, files) => {
  if (err) {
    console.error(err)
    return
  }

  let types = []

  files.forEach(file => {
    const content = fs.readFileSync(file, 'utf8')
    const exports = content.split('export ')
    exports.forEach(v => {
      const result = v.match(/interface\s+\w+/g)
      if (result) types.push('export ' + exports[1])
    })
    fs.unlinkSync(file)
  })

  fs.writeFileSync(path.join(typesPath, `${argv.name}.d.ts`), types.join('\n'))
})

package.json

"scripts": {
  "compile-schemas": "json2ts --cwd src/schemas -i 'src/schemas/*.json' -o src/types && npm run merge-types",
  "merge-types": "node ./tools/merge-types.js --name=mytypes",
}

[drhr]

I started working on this. Currently I only have the "multiple output files" scenario handled, and even that I still need to refactor and test. I need to re-think my approach to include the single output file / stdout scenario.

It's a ways off from a PR, but thought I'd at least share some progress:

the flags I'm using

For reference, I'm using these options.

json2ts \
  -i /path/to/json \
  -o /path/to/generated \
  --unreachableDefinitions \
  --declareExternallyReferenced \
  --ignoreMinAndMaxItems \
  --no-additionalProperties \
  --no-bannerComment \
  --no-style.semi \
  --style.singleQuote

input from multiple json files

❯ tree json
json
├── dependent.json
├── dependent2.json
└── subdir
    └── independent.json

❯ fd -t f json json -0 | xargs -0 -n1 -I{} bash -c 'echo "// {}"; cat {}; echo;'
// json/dependent.json
{
  "$schema": "http://json-schema.org/schema#",
  "type": "object",
  "properties": {
    "prop1": {
      "$ref": "/path/to/json/dependent2.json"
    }
  }
}

// json/dependent2.json
{
  "$schema": "http://json-schema.org/schema#",
  "type": "object",
  "properties": {
    "prop1": {
      "$ref": "/path/to/json/subdir/independent.json"
    },
    "prop2": {
      "$ref": "/path/to/json/subdir/independent.json#/definitions/source"
    }
  }
}

// json/subdir/independent.json
{
  "$schema": "http://json-schema.org/schema#",
  "type": "object",
  "definitions": {
    "source": {
      "type": "object",
      "properties": {
        "prop": {
          "type": "boolean"
        }
      }
    }
  }
}

output to multiple files

Note the import statements.

❯ fd -t f d.ts generated -0 | xargs -0 -n1 -I{} bash -c 'echo "// {}"; cat {}; echo;'
// generated/dependent.d.ts
import {Dependent2} from './dependent2'

export interface Dependent {
  prop1?: Dependent2
}

// generated/dependent2.d.ts
import {Independent, Source} from './subdir/independent'

export interface Dependent2 {
  prop1?: Independent
  prop2?: Source
}

// generated/subdir/independent.d.ts
export interface Independent {}
/**
 * This interface was referenced by `Independent`'s JSON-Schema
 * via the `definition` "source".
 */
export interface Source {
  prop?: boolean
}

I haven't yet thought about how this change should affect certain flags, such as declareExternallyReferenced. But feel free to make suggestions.

[mkadirtan]

This would be great addition!

[drhr]

I have the stdout case handled as well now, but still have a little ways to go cleaning up my changes, adding tests, etc. I don't think I have any other pressing questions at this point, I'll save further comments for the PR.