fastllm
fastllm copied to clipboard
Published
20 hours ago •
ztxz16
ztxz16
pyfastllm 内存泄露
测试脚本：
from typing import List
import json
import time
import os
import sys
cur_dir = os.path.dirname(os.path.abspath(__file__))
fastllm_dir = os.path.join(cur_dir, "fastllm")
sys.path.append(fastllm_dir)
import pyfastllm
glm2_model_path = os.path.join(fastllm_dir, "chatglm2-6b-fp16.flm")
glm2_model = pyfastllm.create_llm(glm2_model_path)
def _receive(idx: int, content_list: List[bytes]):
#print(idx)
pass
def gen_llm_config(llm_config):
_gen_config = pyfastllm.GenerationConfig()
_gen_config.enable_hash_id = True
_gen_config.max_length = llm_config["cfg_max_length"]
_gen_config.top_k = llm_config["cfg_top_k"]
_gen_config.top_p = llm_config["cfg_top_p"]
_gen_config.temperature = llm_config["cfg_temperature"]
_gen_config.repeat_penalty = llm_config["cfg_repeat_penalty"]
_gen_config.last_n = llm_config["cfg_last_n"]
return _gen_config
def response(model, prompt_input, config):
gmask_token_id = 130001
bos_token_id = 130004
input_ids = model.weight.tokenizer.encode(prompt_input)
input_ids = input_ids.to_list()
input_ids.extend([gmask_token_id, bos_token_id])
input_ids = [int(v) for v in input_ids]
# print(input_ids)
handle = model.launch_response(input_ids, config)
continue_token = True
ret_byte = b""
ret_str = ""
while continue_token:
resp_token = model.fetch_response(handle)
continue_token = (resp_token != -1)
content = model.weight.tokenizer.decode_byte([resp_token])
ret_byte += content
ret_str = ret_byte.decode(errors='ignore')
return ret_str
def main():
with open("data.json", "r") as f:
d = json.load(f)
with open("params.json", "r") as f:
d2 = json.load(f)
prompt = d["data"][0]
config = d["config"]
llm_config = gen_llm_config(config)
for i in range(0, 100):
glm2_model.response(prompt, _receive, llm_config)
#time.sleep(1)
print("done 1")
"""
for i in range(0, 100):
ret = response(glm2_model, d2["query"], llm_config)
print(ret)
#time.sleep(1)
print("done 2")
"""
time.sleep(10000000)
if __name__ == "__main__":
main()
使用google-perftools定位FillLLMInputs一直在分配内存但是没有释放，如下图分配30多G：
请问是我的使用方式不对吗
附带data.json的内容如下：
{
"data": [
"[Round 0]\n\n\u95ee\uff1a\u4ee5\u4e0b\u7528\u4e09\u4e2a\u5f15\u53f7\u5305\u542b\u7684\u5185\u5bb9\u662f\u6587\u7ae0\u5185\u5bb9:\n\"\"\" Sample International Contract for Sale of Goods, pursuant to the United NationsConvention on Contracts for the International Sale of Goods TERAMATE,Ltd. with its principal office West Road Drive27, Hopson Chart, Briston, AN44FL,UKrepresented by Matt Wattson, on the basis of Power of Attorney from 23 June 2008(hereinafter referred to as the ,,Seller\" on the first side) and AGFH,a.S. ID:78333998 having its principal office at: Palachova 152, Prague 2, Zip Code: 12000registered in the Commercial Register,Section B, Entry No. 4127 maintained by theMunicipal Court, Prague acting by: Ing. Karel Nekola, Chairman of the Board of Directors(hereinafter referred to as the ,,Buyer\" on the second side) (Seller and Buyer referred to also as the \"Contracting Parties\"or separately each the\"Contracting Party\") have entered on the day, month and year as bellow, pursuant to the United NationsConvention on Contracts for the International Sale of Goods (hereinafter referred to as\"Convention\"), into the following CONTRACT FOR SALE OF GOODS I.Subject-matter of the Contract The Subject-matter of this Contract is particularly the obligation of the Seller to deliver goodsspecified in the Exhibit No. I hereto to the Buyer and to transfer the property in goods to theBuyer under the terms and conditions herein and the obligation of the Buyer to accept thedelivered goods from the Seller and to pay the agreed purchase price. II.Sale of Goods 1.The Seller hereby agrees to deliver the Buyer goods (movables) specified in Exhibit No.1hereto(hereinafter referred to as the ,,Goods\") and in the time,quality and quantity specifiedin Exhibit No. I hereto. The Buyer shall collect the Goods and pay Seller for Goods thepurchase price specified in the Article II. hereof.2. The Seller fulfils his obligation to deliver the Goods when the Goods have been madeavailable to the Buyer at the place of business of the Seller. The Parties have agreed that theBuyer shall arrange for carriage of the Goods from the place of business of the Seller througha carrier the name of which Buyer shall notify Seller. The Seller shall arrange the loading ofGoods, and the Goods shall be packed in the manner set forth in Exhibit No. 2. Unless\n otherwise expressly provided herein, the Goods shall be packed in manner adequate to protectthe Goods. 3.The Seller shall dcliver the Goods to Buyer's carrier on 15 December 2008 during regularworking hours (08.00 to 16.00 hours). Seller shall notify Buyer regarding the delivery ofGoods to carrier by fax message sent to phone No 185555501394.The title in the Goods shall pass to Buyer immediately upon delivery of Goods to theBuyer's carrier. Risk of damage to or loss of the Goods shall pass to the Buyer at the time ofdelivery.5. The Buyer hereby declares he received all information regarding the Goods necessary toarrange insurance coverage.6.Seller shall send the Buyer documents related to the Goods within 10 days after delivery ofGoods and at the Buyer's address set out in herein. III.Purchase Price 1.The Buyer shall pay the Seller the purchase price of the goods amounting EUR 50000(hereinafterreferredto asthe,,Purchase Price\").2. The Purchase Price shall be due upon the invoice issued and sent by the Seller not later than10 days from delivery and collection of Goods by the Buyer. The invoice shall be payable notlater than 21 days from the issue of the invoice by Seller.3. If the Buyer fails to pay the purchase price, the Seller shall have the right to default interestat the rate of 0,1 % of outstanding amount for each day of default without prejudice to anyclaims for damage pursuant to the Article 74 of the Convention. IV.Product Liability 1.The Seller shall be liable for any lack of conformity in Goods which exists at the time whenthe risk passes to the Buyer and which occurs within 24 months from the date of delivery ofGoods by the Buyer's carrier. The Seller declares that the Goods during a period of 24 monthsfrom the date of collection by the Buyer's carrier will remain fit for the purposes for whichthe Goods would ordinarily be used or during this period will retain specified qualities(hereinafter referred to as the ,,Warranty Period\").2.The Seller shall not be responsible for the defects arising out of the failure to followoperation instructions, for the defects caused by improper storage after the Goods weredelivered or for the defects caused by circumstances that were beyond the reasonable control.3.The Buyer shall, immediately upon delivery of the Goods by the carrier, duly examine theGoods and if the defects of Goods were apparent upon the collection of Goods, the Buyershall promptly give notice on this to the Seller.4. Should the Buyer discover any defects during the Warranty Period, the Buyer shall givewritten notice of the defect to the Seller and not later than within 15 days after such defect hadbeen detected. In a written notice specifying the defects he shall have the following options:-replace of defective Goods by delivery of non-defective Goods;demand to- repair the defective Goods if the defects are repairable;demand- appropriate Purchase Price reduction; or\n to withdraw from the- Contract.5.The Seller, upon receipt a notice from the Buyer stating the defect, promptly shall give awritten statement and reply whether he accepts the claim for defects or not. VI.Exclusion of Liability 1.A party is not liable for a failure to perform any of his obligations if he proves that thefailure was due to an impediment beyond his control and that he could reasonably be expectedto have taken the impediment into account at the time of the conclusion of the Contract or tohave ayoided or overcome it or its consequences. The exemption provided by this Article haseffect for the period during which the impediment exists.2. The non-pefforning party shall give prompt written notice to the other party of the reasonfor its failure to perform and the extent and duration of its inability to perform. VII. Arbitration Clause All the disputes resulting from this agreement or in conjunction with it, will be decided finallyin the arbitration procedure before one arbitrator Mgr. Marie Pluharova, registered in the listof arbitrators of Czech Arbitration Centre s.r.o., ID 28163427, Klatovska 515/169,32100Plzen-Litice and registered in the list of arbitrators of Ministry of Justice Czech Republicreg.nr.69 and appointed in accordance with Act No. 216/1994 coll. of Laws, on ArbitrationProcedure and Execution of Arbitration Awards, and with the Proceeding Rules of CzechArbitration Centre announced at it's websites www.arbitrators.cz. The parties authorize thearbitrator to settle the dispute based on the principles of natural equity. Compensation forarbitration costs (including the expenses of the contractual parties) will be awarded by thearbitrator based on the principle of success in the dispute. VIII.Final Provisions 1.This Contract shall enter into force and shall take effect on the day when it is executed.2.The Contracting Parties hereby agree that entering into this Contract and performing dutiesunder this Contract have been duly approved by the relevant company bodies of theContracting Parties in a compliance with legal regulations, by-laws and other internalregulations of the Contracting Parties; and no other approval or consent shall be required.3.The Contracting Parties agrees to respect the legitimate interests of the other Party, shallconduct in accordance with the purpose of this Contract and shall not counteract such purposeand they shall perform all legal and other actions that may prove necessary to reach thepurpose of this Contract. 4. All documents in writing shall be mailed at the address of the Contracting Parties set forth\n in the heading of this Contract unless either of the Contracting Partics shall give a writtennotice to the other Party on changing its address. Whatever papers the delivery of which isrequired, assumed or is made available by this Contract and regardless of any other availableway allowed by the legal regulations to prove such a delivery, shall be deemed to have beenserved if such had been delivered to the other Contracting Party at the address set forth in theheading of this Contract or at the address noticed in written form by either Contracting Partyto the other Party.5. Any changes and amendments to this Contract shall require a written form.6.If any provision of this Contract is determined to be invalid or unenforceable,the validityor enforceability of the other provisions either of this Contract as neither a whole nor otherprovisions will be affected unless such an invalid or unenforceable provision is severable.Contracting Parties herby agrees to supersede such an invalid or unenforceable provision by anew valid and forceable provision that most closely matches the intent and the purpose of theoriginal provision.7.This Contract and the relations arising from shall be govermed by the Law of the CzechRepublic,particularly by the United Nations Convention on Contracts for the IntermationalSale of Goods.8.This Contract had been made in two duplicates whereby each Contracting Party shall retainone copy each. e TERAMATE, Ltd.Karel NekolaDone in Prague on 13 June 2019 ml AGFH,a.S.Matt Wattson Done in Prague on 13 June 2019\n\"\"\"\n\u7528150\u5b57\u4ee5\u5185\u7684\u4e2d\u6587\u603b\u7ed3\u4ee5\u4e0a\u6587\u7ae0\u5185\u5bb9:\n\n\u7b54\uff1atime_stamp:169762201245954754816808106421951820652289425540685"
],
"config": {
"cfg_max_length": 4096,
"cfg_top_k": 1,
"cfg_top_p": 0.8,
"cfg_temperature": 0.1,
"cfg_repeat_penalty": 1.0,
"cfg_last_n": 64
}
}
我也遇到这个问题了， gpu版的， 跑完以后就报错了， 释放GPU内存的代码似乎有问题 Error: CUDA error when release memory! CUDA error = 4, cudaErrorCudartUnloading at /fastllm/src/devices/cuda/fastllm-cuda.cu:1493 'driver shutting down'
