gpu-operator icon indicating copy to clipboard operation
gpu-operator copied to clipboard

gpu-operator creates ci using mig

Open asskss opened this issue 2 years ago • 0 comments

mig-config.yaml

    mig-configs:
      custom-config: 
        - devices: [0]
          mig-enabled: false
        - devices: [1]     
          mig-enabled: true   
          mig-devices:
            "7g.80gb": 1    
        - devices: [2]
          mig-enabled: true
          mig-devices:
            "2g.20gb": 3
        - devices: [3]      
          mig-enabled: true 
          mig-devices:
            "3g.40gb": 1
            "4g.40gb": 1    
        - devices: [4]      
          mig-enabled: true
          mig-devices:
            "3g.40gb": 1
            "4g.40gb": 1    
        - devices: [5]      
          mig-enabled: true
          mig-devices:
            "3g.40gb": 1
            "4g.40gb": 1
        - devices: [6]
          mig-enabled: true
          mig-devices:
            "3g.40gb": 1
            "4g.40gb": 1 
        - devices: [7]
          mig-enabled: true
          mig-devices:
            "1g.10gb": 1
            "2g.20gb": 1
            "4g.40gb": 1 

nvidia-smi mig -lcip -gi 0

+--------------------------------------------------------------------------------------+
| Compute instance profiles:                                                           |
| GPU     GPU       Name             Profile  Instances   Exclusive       Shared       |
|       Instance                       ID     Free/Total     SM       DEC   ENC   OFA  |
|         ID                                                          CE    JPEG       |
|======================================================================================|
|   0      0       MIG 1c.7g.80gb       0      0/7           14        5     0     1   |
|                                                                      7     1         |
+--------------------------------------------------------------------------------------+
|   0      0       MIG 2c.7g.80gb       1      0/3           28        5     0     1   |
|                                                                      7     1         |
+--------------------------------------------------------------------------------------+
|   0      0       MIG 3c.7g.80gb       2      0/2           42        5     0     1   |
|                                                                      7     1         |
+--------------------------------------------------------------------------------------+
|   0      0       MIG 4c.7g.80gb       3      0/1           56        5     0     1   |
|                                                                      7     1         |
+--------------------------------------------------------------------------------------+
|   0      0       MIG 7g.80gb          4*     0/1           98        5     0     1   |
|                                                                      7     1         |
+--------------------------------------------------------------------------------------+
|   1      0       MIG 1c.7g.80gb       0      0/7           14        5     0     1   |
|                                                                      7     1         |
+--------------------------------------------------------------------------------------+
|   1      0       MIG 2c.7g.80gb       1      0/3           28        5     0     1   |
|                                                                      7     1         |
+--------------------------------------------------------------------------------------+
|   1      0       MIG 3c.7g.80gb       2      0/2           42        5     0     1   |
|                                                                      7     1         |
+--------------------------------------------------------------------------------------+
|   1      0       MIG 4c.7g.80gb       3      0/1           56        5     0     1   |
|                                                                      7     1         |
+--------------------------------------------------------------------------------------+
|   1      0       MIG 7g.80gb          4*     0/1           98        5     0     1   |
|                                                                      7     1         |
+--------------------------------------------------------------------------------------+

nvidia-smi mig -lci -gi 0

+--------------------------------------------------------------------+
| Compute instances:                                                 |
| GPU     GPU       Name             Profile   Instance   Placement  |
|       Instance                       ID        ID       Start:Size |
|         ID                                                         |
|====================================================================|
|   0      0       MIG 7g.80gb          4         0          0:7     |
+--------------------------------------------------------------------+
|   1      0       MIG 7g.80gb          4         0          0:7     |
+--------------------------------------------------------------------+

nvidia-smi mig -cci 2 -gi 0

Unable to create a compute instance on GPU  0 GPU instance ID  0 using profile 2: Insufficient Resources
Failed to create compute instances: Insufficient Resources

I want to create a MIG 3c.7g/80gb specification prompt for Independent Resources.How to solve it.

asskss avatar Nov 29 '23 09:11 asskss