Create copy of a LightningWork object from an already running Work

[aniketmaurya]

🚀 Feature

It would be a great feature if we can copy work when we need a replica of something that already exists. This would save the reinstallation of dependency time.

def __init__(self):
    work1: LightningWork = SomeWork()

def run():
   work1.run()

   if work1.is_successful and SOME_CONDITION:
     # no need to reinstall dependencies 
      new_work = work1.copy()

Motivation

While deploying Models with LightningWork I am autoscaling based on a number of requests and the upscale requires me to create a new Work this is a time taking stuff. If we can just clone an already running instance it will save time.

def upscale():
   if num_requests > ALLOWED_NUM_REQUESTS:
       self.work[work_count + 1] = Work()
      self.work[work_count + 1].run()
     work_count +=1

Pitch

Alternatives

Additional context

---

If you enjoy Lightning, check out our other projects! ⚡

• Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

• Lite: enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.

• Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.

• Bolts: Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.

• Lightning Transformers: Flexible interface for high-performance research using SOTA Transformers leveraging PyTorch Lightning, Transformers, and Hydra.