Feature request: import custom code module in generated modular input scripts.

[pmeyerson]

So I have my_custom_modular_input.py, my custom code for a modular input, which implements valiate_input and stream_events

ucc-gen creates my_custom.py, a rest handler skeleton which has stream_events, valiate_input, all the arguments defined in globalConfig.json, and the filename is already linked to the modular input definition.

Since I already have my_custom_modular_input.py, it would be great if ucc-gen could automatically import this modular when writing my_custom.py and use it in the stream_events and validate_input method calls.

I'm currently using some regex and file manipulation with additional_packaging.py which works fine but feels brittle if the format or output of ucc-gen would change.

I feel this would be a very useful feature for folks working with modular input scripts, but if I've missed a better workflow please let me know. Thanks for considering.

[ghost]

Thanks for this feature request, I agree that ucc-gen could handle this better.

I think that your approach is good enough for this use case. I can't think of anything better.

[lmnogues]

This feature could be really usefull, it could allow to use custom code without having to maintain the get_scheme() manually after some changes in the globalConfig.json... May be it can be implemented as a custom template as a first step and avoid breakage of other use cases?

[lmnogues]

Is there any plan to add this functionality ?

[artemrys]

I read the ask one more time, you are actually referring to the actual modular input code, not the rest handlers created by ucc-gen automatically. My initial comment was based on the assumption that you are referring to the rest handler custom code.

What you are referring to, may be an actually interesting feature request. I have limited capacity this quarter, I can keep this in mind during our next quarter planning.

[pmeyerson]

@Jalkar FYI to get around this issue, I create the customizations I need to the rest handler in a separate module, extending splunktaucclib. Then I let UCC create the "real" rest handler code , and just modify the handler class that is launched at the bottom of the file. My use case is I need to take extra actions when an input is removed or disabled, outside of module input script. But maybe your use case is different from mine?

The advantage here being I don't need to understand if UCC will try to overwrite my customizations or not, and don't need to manually implement get_scheme(). I let UCC do as much of the work as possible.

My opinion, if I understand @Jalkar - I'm a bit happier without UCC trying to modify code I've already written. It makes debugging easier when modules I've authored don't change during build process, but maybe others feel differently, or it decreases the scope of work for UCC PR to something that is doable. HTH

IE: custom_handler.py has my customizations

class CustomHandlerApp1(splunktaucclib.rest_handler.admin_external.AdminExternalHandler):
      def __init__(self, *args,**kwargs):
         ...

     @admin_external.build_conf_info
     def handleEdit(self, confInfo):
        ...
    
    @admin_external.build_conf_info
    def handleRemove(self, confInfo):
       ...

UCC-GEN generates TA_inputname_rh_inputname.py:

...
if __name__ == '__main__':
    ...
    admin_external.handle(endpoint, handler=AdminExternalHandler)

additional_packaging adds/modifies TA_inputname_rh_inputname.py:

from custom_handler import CustomHandlerApp1

if __name__ == '__main__':
    ...
    admin_external.handle(endpoint, **handler=CustomHandlerApp1**)

[lmnogues]

So in the end you need to change the generated file after each ucc-gen exec ?

[artemrys]

ucc-gen does not overwrite the files if they are already present in package/bin/ folder for example with the correct name.

If you have an input called demo_input.py in package/bin/ folder with your code, then ucc-gen does not override that file with the default content. The same is for the rest handlers and files.

As far as I understand, we are talking here about 2 different but similar use cases:

• custom code for the rest handlers
• custom code for the modular inputs

Both of them are valid feature requests, but should be tracked differently.

[lmnogues]

My issue is currently when i'm adding a new parameter for an existing input, I need to manually edit the "scheme" of the existing package/bin/my_mod_input.py For me the best solution would be to have a auto-generated "my_mod_input_base.py" which contains the autogenerated scheme with possibility to override with a custom "my_mode_input.py" inheriting from "my_mode_input_base"

[lmnogues]

re-reading your last comments, i just catch the diff rest handler/modular inputs :)

[pmeyerson]

FYI @Jalkar RE: "So in the end you need to change the generated file after each ucc-gen exec ?" >> YES. I added code in additional_packaging.py to open the generated rest handler file, search for the the line of code and change it with regex. Additional_packaging.py is automatically executed by ucc-gen as part of the build process - HTH @artemrys - thanks for the correction regarding ucc not-overwriting!

[lmnogues]

ok thank for the feedback (I never realised the additional_packaging was a functionnality include in UCC ^^

[artemrys]

Yeah, it was never in the documentation, I added it recently (https://splunk.github.io/addonfactory-ucc-generator/how_to_use/#additional_packagingpy-file).

I'll leave this issue open, I think this maybe interesting to implement in the future, not committing to it so far.

[artemrys]

I'll rename this issue to track the customisation for the UCC-generated modular inputs and will open a new one to track customisation for the actual REST handlers.

By the way, I updated the docs and now it has an example for the custom REST handlers (https://splunk.github.io/addonfactory-ucc-generator/custom_rest_handler/).

[VatsalJagani]

@pmeyerson - This would be huge addition to UCC I think. I also don't think this would be that hard to do.

By any chance can you provide content of additional_packaging.py that you have.

[pmeyerson]

here's what worked for me - lets me call custom modular input handler functions without having to maintain the scheme arguments when globalConfig.json is updated

1. add import statement
2. modify def stream_events() function
3. modify def validation_input() function
4. set use_single_instance value

        modinput_handler_file = 'output/{}/bin/{}'.format(ta_name, modinput_handler_file)
        repl1_append = 'import {} as input_module\n'.format(custom_modinput_handler)

        # replace default generation of stream_events() by ucc-gen
        pattern2 = r'def stream_event[\w\W]*(?:\n\n)'
        repl2 = 'def stream_events(self, inputs, ew):\n        input_module.stream_events(self, inputs, ew)\n\n\n'

        # replace default generation of validate_input() by ucc-gen
        pattern3 = r'def validate_input[\w\W]*return\n'
        repl3 = 'def validate_input(self, definition):\n        input_module.validate_input(self, definition)\n\n'

        ## set to single instance mode = false
        pattern4 = r'scheme.use_single_instance = True'
        repl4 = 'scheme.use_single_instance = False'

        with open(modinput_handler_file, 'r+b') as file1:
            print("additional_packaging:opened {} for post-processing".format(file1.name))
            
            # insert repl1_append after first line of file
            file1.seek(0)
            pattern1 = file1.readline().decode("utf-8")
            file1.seek(0)
            content = file1.read().decode("utf-8")

            repl1 = pattern1 + repl1_append
            
            if not re.search(pattern1, content): print("additional_packaging:pattern1 not found")
            if not re.search(pattern2, content): print("additional_packaging:pattern2 not found")
            if not re.search(pattern3, content): print("additional_packaging:pattern3 not found")
            if not re.search(pattern4, content): print("additional_packaging:pattern4 not found")
            content = re.sub(pattern1, repl1, content)
            content = re.sub(pattern2, repl2, content)
            content = re.sub(pattern3, repl3, content)
            content = re.sub(pattern4, repl4, content)
            content = content.encode("utf-8")

            file1.seek(0)
            file1.write(content)
            file1.truncate()
            file1.close()

        print("done modifying file {}".format(file1.name))

HTH

[pmeyerson]

for reference, here's stub from my custom modular input handler file:

...
def validate_input(self, definition):
  """ get all parameters and raise ValueError if something isn't right 
       NOTE: getting the paramters must be manually maintained if new fields are added to modular input definition"""
       
   param1 = definition.parameters.get("param1")
   param2 = definition.parameters.get("param2")
   ...
   # raise ValueError on any validation failure
   ...
   ...
   
def stream_events(self, inputs, ew):
   """ entry point that executes when an input parameter is saved in the UI, and on interval or splunkd restart """
   
   for input_name, input in iteritems(inputs.__dict__["inputs"]):
      input["name"] = input_name
      ...
      ...
      # do stuff

[VatsalJagani]

@pmeyerson - Much appreciated. This was very helpful!!!