Custom kwargs not passed when extending PretrainedConfig class

[cceyda]

System Info

• transformers version: 4.26.1
• Platform: macOS-13.2.1-arm64-arm-64bit
• Python version: 3.9.12
• Huggingface_hub version: 0.12.0
• PyTorch version (GPU?): 1.13.0.dev20220925 (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help?

@sgugger @Narsil

Information

• [ ] The official example scripts
• [X] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [X] My own task or dataset (give details below)

Reproduction

I want to load existing model's configs with .from_pretrained but also want to pass my own kwargs moo & boo. I'm extending PretrainedConfig like below:

from transformers import PretrainedConfig

class MyClassificationConfig(PretrainedConfig):
    def __init__(self, 
                moo='poo',
                boo=5,
                **kwargs):
        print(boo) # prints 5 because boo doesn't get passed
        print(kwargs)
        # do custom calculations and set some custom config values here
        super().__init__(**kwargs)

MyClassificationConfig.from_pretrained('google/canine-s',# example model, any other is same
id2label={1:"g",2:"b"}, 
moo="poo", 
boo="hoo")

only predefined id2label,label2id,num_classes values get updated in the config. Happens here.

the custom moo and poo param doesn't get passed to MyClassificationConfig. Because kwargs don't get passed here Results in moo & boo argument values not changing from the default. Since kwargs are optional can result in silent errors where you are actually using default values while thinking you are passing values!

I think this is a bug. But if it is intentional maybe nice to warn the user so there are no silent errors

Expected behavior

Should be able to extend PretrainedConfig class allowing custom kwargs.

[sgugger]

This is intended. Hadding the kwargs is done here but to filter out the value that have nothing to do in the config, we detect whether they are attribute of the config or not. So you should change your code like this:

class MyClassificationConfig(PretrainedConfig):
    def __init__(self, 
                moo='poo',
                boo=5,
                **kwargs):
        print(boo) # prints 5 because boo doesn't get passed
        print(kwargs)
        # do custom calculations and set some custom config values here
        self.moo = moo
        self.poo = poo
        super().__init__(**kwargs)

[cceyda]

this is saving the custom moo, boo to the returned config,

config = MyClassificationConfig.from_pretrained("hf-internal-testing/config-no-model",boo="hoo")

but I still can't access the kwarg value I set inside the __init__ to do some calculations. ~~Is there a post config init function I can override?~~ I guess I can do it at call.

[cceyda]

Also I'm a bit confused about the following behaviour (example):

Even though the DinatConfig class doesn't take as a param image_size and thus doesn't do self.image_size=image_size anywhere if I pass config=DinatConfig(...,image_size=128,...) this (imagined) image_size param becomes part of the config(because super() assigns them)

DinatConfig(image_size=5).image_size

which can be good if you want to save some stuff along in your config but isn't it also prone to confusion say that I confuse config key names? Also if I can just pass what ever kwarg and it becomes part of config then do I really need to extend the PretrainedConfig class if I'm just assigning passed kwargs to self 🤔 I guess just to assign default values.

[sgugger]

Sorry you actually don't need to set them as attributes, you just need to pass them along to the super method:

class MyClassificationConfig(PretrainedConfig):
    def __init__(self, 
                moo='poo',
                boo=5,
                **kwargs):
        print(boo) # prints 5 because boo doesn't get passed
        print(kwargs)
        # do custom calculations and set some custom config values here
        super().__init__(moo=moo, boo=boo, **kwargs)

will have the right value set in the config.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.