polkadot-types-from-chain removes imports which are used later in the same file

[atodorov]

• I'm submitting a ...

• [x] Bug report
• [ ] Feature request
• [ ] Support request
• [x] Other

• What is the current behavior and expected behavior?

Rebuilding type definitions removes imports which are actually being used, which in turn results to build issues with my package later on.

I am expecting that these imports will not be removed.

• Please tell us about your environment:

• Version: @polkadot/typegen": "^10.10.1",, "@polkadot/api": "^10.10.1",

• Environment: Linux

  • [X] Node.js - v18.18.2
  • [ ] Browser
  • [ ] Other (limited support for other environments)

• Language:

  • [ ] JavaScript
  • [x] TypeScript (include tsc --version: 5.3.2)
  • [ ] Other

** Steps to reproduce **

$ git clone https://github.com/gluwa/creditcoin3
$ cd creditcoin3/
$ cargo build --release
   Compiling creditcoin3-node v3.6.3 (/home/gluwa/creditcoin-3.0/creditcoin3/node)
    Finished release [optimized] target(s) in 53.77s

$ ./target/release/creditcoin3-node --dev --validator --pruning archive  --node-key d182d503b7dd97e7c055f33438c7717145840fd66b2a055284ee8d768241a463             --base-path ./alice-data


$ cd cli/
$ yarn install
$ ./get-metadata.sh

$ git status
On branch testing/csub-862-only-update-typedefs
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   creditcoin.json


$ yarn build:types
yarn run v1.22.19
$ yarn generate:defs && yarn generate:meta
$ ts-node --skip-project node_modules/.bin/polkadot-types-from-defs --package creditcoin3/lib/interfaces --input ./src/lib/interfaces --endpoint ./creditcoin.json
ERROR: Unable to load user definitions: /home/gluwa/creditcoin-3.0/creditcoin3/cli/src/lib/interfaces/definitions.ts is not a file
/home/gluwa/creditcoin-3.0/creditcoin3/cli/src/lib/interfaces/types.ts
	Generating
	Writing

/home/gluwa/creditcoin-3.0/creditcoin3/cli/src/lib/interfaces/augment-types.ts
	Generating
	Writing

/home/gluwa/creditcoin-3.0/creditcoin3/cli/src/lib/interfaces/lookup.ts
	Generating
	Writing

$ ts-node --skip-project node_modules/.bin/polkadot-types-from-chain --package creditcoin3/lib/interfaces  --endpoint ./creditcoin.json --output ./src/lib/interfaces
Generating from metadata, 106,995 bytes
ERROR: No custom definitions found: /home/gluwa/creditcoin-3.0/creditcoin3/cli/src/lib/interfaces/definitions.ts is not a file
ERROR: No lookup definitions found: Unknown file extension ".ts" for /home/gluwa/creditcoin-3.0/creditcoin3/cli/src/lib/interfaces/lookup.ts
/home/gluwa/creditcoin-3.0/creditcoin3/cli/src/lib/interfaces/augment-api-consts.ts
	Generating
	Writing

/home/gluwa/creditcoin-3.0/creditcoin3/cli/src/lib/interfaces/augment-api-errors.ts
	Generating
	Writing

/home/gluwa/creditcoin-3.0/creditcoin3/cli/src/lib/interfaces/augment-api-events.ts
	Generating
	Writing

/home/gluwa/creditcoin-3.0/creditcoin3/cli/src/lib/interfaces/augment-api-query.ts
	Generating
	Writing

/home/gluwa/creditcoin-3.0/creditcoin3/cli/src/lib/interfaces/augment-api-rpc.ts
	Generating
	Writing

/home/gluwa/creditcoin-3.0/creditcoin3/cli/src/lib/interfaces/augment-api-runtime.ts
	Generating
	Writing

/home/gluwa/creditcoin-3.0/creditcoin3/cli/src/lib/interfaces/augment-api-tx.ts
	Generating
	Writing

/home/gluwa/creditcoin-3.0/creditcoin3/cli/src/lib/interfaces/augment-api.ts
	Generating
	Writing

Done in 1.99s.


$ yarn format  ### because all files in this repo are explicitly formatted

$$ git diff src/
diff --git a/cli/src/lib/interfaces/augment-api-consts.ts b/cli/src/lib/interfaces/augment-api-consts.ts
index c2418f9..32df384 100644
--- a/cli/src/lib/interfaces/augment-api-consts.ts
+++ b/cli/src/lib/interfaces/augment-api-consts.ts
@@ -8,13 +8,6 @@ import '@polkadot/api-base/types/consts';
 import type { ApiTypes, AugmentedConst } from '@polkadot/api-base/types';
 import type { Vec, u128, u16, u32, u64, u8 } from '@polkadot/types-codec';
 import type { Codec } from '@polkadot/types-codec/types';
-import {
-    FrameSupportPalletId,
-    FrameSystemLimitsBlockLength,
-    FrameSystemLimitsBlockWeights,
-    SpWeightsRuntimeDbWeight,
-    SpVersionRuntimeVersion,
-} from '@polkadot/types/lookup';
 
 export type __AugmentedConst<ApiType extends ApiTypes> = AugmentedConst<ApiType>;
 
diff --git a/cli/src/lib/interfaces/augment-api-events.ts b/cli/src/lib/interfaces/augment-api-events.ts
index 5dac02c..8b4b8c0 100644
--- a/cli/src/lib/interfaces/augment-api-events.ts
+++ b/cli/src/lib/interfaces/augment-api-events.ts
@@ -22,21 +22,6 @@ import type {
 } from '@polkadot/types-codec';
 import type { ITuple } from '@polkadot/types-codec/types';
 import type { AccountId32, H160, H256, Perbill, Permill } from '@polkadot/types/interfaces/runtime';
-import {
-    FrameSupportTokensMiscBalanceStatus,
-    EvmCoreErrorExitReason,
-    EthereumLog,
-    SpRuntimeDispatchError,
-    SpConsensusGrandpaAppPublic,
-    PalletImOnlineSr25519AppSr25519Public,
-    PalletStakingExposure,
-    PalletNominationPoolsCommissionChangeRate,
-    PalletNominationPoolsPoolState,
-    Creditcoin3RuntimeProxyFilter,
-    PalletStakingForcing,
-    PalletStakingValidatorPrefs,
-    FrameSupportDispatchDispatchInfo,
-} from '@polkadot/types/lookup';
 
 export type __AugmentedEvent<ApiType extends ApiTypes> = AugmentedEvent<ApiType>;
 
diff --git a/cli/src/lib/interfaces/augment-api-query.ts b/cli/src/lib/interfaces/augment-api-query.ts
index f7f2097..67bcbeb 100644
--- a/cli/src/lib/interfaces/augment-api-query.ts
+++ b/cli/src/lib/interfaces/augment-api-query.ts
@@ -7,61 +7,10 @@ import '@polkadot/api-base/types/storage';
 
 import type { ApiTypes, AugmentedQuery, QueryableStorageEntry } from '@polkadot/api-base/types';
 import type { Data } from '@polkadot/types';
-import type { Bytes, Option, U256, U8aFixed, Vec, bool, u128, u32, u64 } from '@polkadot/types-codec';
+import type { Bytes, Null, Option, U256, U8aFixed, Vec, bool, u128, u32, u64 } from '@polkadot/types-codec';
 import type { AnyNumber, ITuple } from '@polkadot/types-codec/types';
 import type { AccountId32, H160, H256, Perbill, Percent, Permill } from '@polkadot/types/interfaces/runtime';
 import type { Observable } from '@polkadot/types/types';
-import {
-    SpConsensusBabeAppPublic,
-    SpConsensusBabeBabeEpochConfiguration,
-    SpConsensusBabeDigestsPreDigest,
-    SpConsensusBabeDigestsNextConfigDescriptor,
-    PalletBalancesAccountData,
-    PalletBalancesIdAmount,
-    PalletBalancesBalanceLock,
-    PalletBalancesReserveData,
-    EthereumBlock,
-    EthereumReceiptReceiptV3,
-    FpRpcTransactionStatus,
-    EthereumTransactionTransactionV2,
-    PalletEvmCodeMetadata,
-    PalletFastUnstakeUnstakeRequest,
-    PalletGrandpaStoredPendingChange,
-    PalletGrandpaStoredState,
-    PalletIdentityRegistration,
-    PalletIdentityRegistrarInfo,
-    PalletImOnlineSr25519AppSr25519Public,
-    PalletNominationPoolsBondedPoolInner,
-    PalletNominationPoolsClaimPermission,
-    PalletNominationPoolsPoolMember,
-    PalletNominationPoolsRewardPool,
-    PalletNominationPoolsSubPools,
-    SpStakingOffenceOffenceDetails,
-    PalletProxyAnnouncement,
-    PalletProxyProxyDefinition,
-    SpCoreCryptoKeyTypeId,
-    Creditcoin3RuntimeOpaqueSessionKeys,
-    PalletStakingActiveEraInfo,
-    PalletStakingEraRewardPoints,
-    PalletStakingExposure,
-    PalletStakingValidatorPrefs,
-    PalletStakingForcing,
-    PalletStakingStakingLedger,
-    PalletStakingNominations,
-    PalletStakingRewardDestination,
-    PalletStakingSlashingSlashingSpans,
-    PalletStakingSlashingSpanRecord,
-    PalletStakingUnappliedSlash,
-    FrameSystemAccountInfo,
-    FrameSupportDispatchPerDispatchClassWeight,
-    SpRuntimeDigest,
-    FrameSystemEventRecord,
-    FrameSystemPhase,
-    FrameSystemLastRuntimeUpgradeInfo,
-    PalletTransactionPaymentReleases,
-    PalletBagsListListBag,
-    PalletBagsListListNode,
-} from '@polkadot/types/lookup';
 
 export type __AugmentedQuery<ApiType extends ApiTypes> = AugmentedQuery<ApiType, () => unknown>;
 export type __QueryableStorageEntry<ApiType extends ApiTypes> = QueryableStorageEntry<ApiType>;
@@ -376,6 +325,8 @@ declare module '@polkadot/api-base/types/storage' {
                 [H160, H256]
             > &
                 QueryableStorageEntry<ApiType, [H160, H256]>;
+            suicided: AugmentedQuery<ApiType, (arg: H160 | string | Uint8Array) => Observable<Option<Null>>, [H160]> &
+                QueryableStorageEntry<ApiType, [H160]>;
             /**
              * Generic query
              **/
diff --git a/cli/src/lib/interfaces/augment-api-tx.ts b/cli/src/lib/interfaces/augment-api-tx.ts
index b1480d7..ce6304a 100644
--- a/cli/src/lib/interfaces/augment-api-tx.ts
+++ b/cli/src/lib/interfaces/augment-api-tx.ts
@@ -15,36 +15,6 @@ import type { Data } from '@polkadot/types';
 import type { Bytes, Compact, Option, U256, Vec, bool, u128, u16, u32, u64 } from '@polkadot/types-codec';
 import type { AnyNumber, IMethod, ITuple } from '@polkadot/types-codec/types';
 import type { AccountId32, Call, H160, H256, Perbill, Percent, Permill } from '@polkadot/types/interfaces/runtime';
-import {
-    SpConsensusBabeDigestsNextConfigDescriptor,
-    SpConsensusSlotsEquivocationProof,
-    SpSessionMembershipProof,
-    EthereumTransactionTransactionV2,
-    SpConsensusGrandpaEquivocationProof,
-    PalletIdentityJudgement,
-    PalletIdentityBitFlags,
-    PalletIdentityIdentityInfo,
-    PalletImOnlineHeartbeat,
-    PalletImOnlineSr25519AppSr25519Signature,
-    PalletNominationPoolsBondExtra,
-    PalletNominationPoolsClaimPermission,
-    PalletNominationPoolsCommissionChangeRate,
-    PalletNominationPoolsConfigOpU128,
-    PalletNominationPoolsConfigOpU32,
-    PalletNominationPoolsConfigOpPerbill,
-    PalletNominationPoolsPoolState,
-    PalletNominationPoolsConfigOpAccountId32,
-    Creditcoin3RuntimeProxyFilter,
-    Creditcoin3RuntimeOpaqueSessionKeys,
-    PalletStakingRewardDestination,
-    PalletStakingPalletConfigOpU128,
-    PalletStakingPalletConfigOpU32,
-    PalletStakingPalletConfigOpPercent,
-    PalletStakingPalletConfigOpPerbill,
-    PalletStakingValidatorPrefs,
-    SpWeightsWeightV2Weight,
-    Creditcoin3RuntimeOriginCaller,
-} from '@polkadot/types/lookup';
 
 export type __AugmentedSubmittable = AugmentedSubmittable<() => unknown>;
 export type __SubmittableExtrinsic<ApiType extends ApiTypes> = SubmittableExtrinsic<ApiType>;
diff --git a/cli/src/lib/interfaces/index.ts b/cli/src/lib/interfaces/index.ts
index 58052a7..7f1de78 100644
--- a/cli/src/lib/interfaces/index.ts
+++ b/cli/src/lib/interfaces/index.ts
@@ -1,4 +1,4 @@
 // Auto-generated via `yarn polkadot-types-from-defs`, do not edit
 /* eslint-disable */
 
-// export * from './types.js';
+export * from './types.js';

build:types is defined as

        "build:types": "yarn generate:defs && yarn generate:meta",
        "generate:defs": "ts-node --skip-project node_modules/.bin/polkadot-types-from-defs --package creditcoin3/lib/interfaces --input ./src/lib/interfaces --endpoint ./creditcoin.json",
        "generate:meta": "ts-node --skip-project node_modules/.bin/polkadot-types-from-chain --package creditcoin3/lib/interfaces  --endpoint ./creditcoin.json --output ./src/lib/interfaces",

and I've identified that the removal of these imports is coming from polkadot-types-from-chain.

https://github.com/gluwa/creditcoin3/pull/95 - bumps versions of Polkadot.js packages and just rebuilds the type definitions and fails because of the error above - see the failing CI job javascript-typecheck / cli for example

https://github.com/gluwa/creditcoin3/pull/99 - adds automatic regeneration of type definitions (will push an automatic commit to GitHub if successfull) and fails because of this very same issue - see the failing CI job integration-test-cli for example (actual diff in CI logs).

[smohan-dw]

The error occurs only with Node versions 20 and above.

[atodorov]

The error occurs only with Node versions 20 and above.

This statement doesn't appear to be true!

Locally I am seeing the same problem with Node v18.18.2. Here we see the same problem with v18.18.2. Actual diff & node --version output is in the logs (search for DEBUG).

@smohan-dw do you have a Node version which you know to be working so that I can give it a try, instead of checking every possible version ?

[smohan-dw]

@smohan-dw do you have a Node version which you know to be working so that I can give it a try, instead of checking every possible version ?

Node v18.18.2 is working fine for me. You can take a look at https://github.com/smohan-dw/cord.js/tree/sm-updates-0x3

[Muhammad-Altabba]

The lookupTypes are missed from the list used to extract imports for the generated types. And the PR #5769 should fix this. Thanks,