Pickle.jl
Pickle.jl copied to clipboard
An experimental package for loading and saving object in Python Pickle format.
Pickle.jl
An experimental package for loading and saving object in Python Pickle and Torch Pickle format.
Supports
Load
- basic builtin types. e.g. `Integer`, `String`, `Tuple`, `Dict`, `Vector`, `Set` ...
julia> Pickle.load(open("test.pkl"))
Dict{Any,Any} with 6 entries:
"int" => 0
"str" => "Julia!"
"bytes" => UInt8[0x31, 0x32, 0x33, 0x34]
"tuple" => (1, 2.0, "3", UInt8[0x34])
"bool" => true
"float" => 3.14159
-
some basic support of loading
numpy.array
andscipy.sparse.csr_matrix
withPickle.npyload("data.pkl")
. -
framework for easily analyze and implement required translation methods. For those data which are not able to restore directly will be stored in a
Defer
object which is similar toExpr
but is mutable.
julia> load("test/test_pkl/test-np.pkl")
Defer(:build, Defer(:reduce, Defer(:numpy.core.multiarray._reconstruct), Defer(:numpy.ndarray), (0,), UInt8[0x62]),
(1, (3, 5), Defer(:build, Defer(:reduce, Defer(:numpy.dtype), f8, 0, 1), (3, "<", nothing, nothing, nothing, -1, -
1, 0)), false, UInt8[0x29, 0x0f, 0x9c, 0x3b, 0x11, 0x5a, 0xbf, 0x3f, 0x29, 0xa6 … 0xb8, 0x3f, 0x6b, 0x4a, 0x31, 0
xab, 0x2b, 0xc5, 0xf2, 0x3f]))
julia> dump(ans)
Pickle.Defer
head: Symbol build
args: Array{Any}((2,))
1: Pickle.Defer
head: Symbol reduce
args: Array{Any}((4,))
1: Pickle.Defer
head: Symbol numpy.core.multiarray._reconstruct
args: Array{Any}((0,))
2: Pickle.Defer
head: Symbol numpy.ndarray
args: Array{Any}((0,))
3: Tuple{Int64}
1: Int64 0
4: Array{UInt8}((1,)) UInt8[0x62]
2: Tuple{Int64,Tuple{Int64,Int64},Pickle.Defer,Bool,Array{UInt8,1}}
1: Int64 1
2: Tuple{Int64,Int64}
1: Int64 3
2: Int64 5
3: Pickle.Defer
head: Symbol build
args: Array{Any}((2,))
1: Pickle.Defer
head: Symbol reduce
args: Array{Any}((4,))
1: Pickle.Defer
head: Symbol numpy.dtype
args: Array{Any}((0,))
2: String "f8"
3: Int64 0
4: Int64 1
2: Tuple{Int64,String,Nothing,Nothing,Nothing,Int64,Int64,Int64}
1: Int64 3
2: String "<"
3: Nothing nothing
4: Nothing nothing
5: Nothing nothing
6: Int64 -1
7: Int64 -1
8: Int64 0
4: Bool false
5: Array{UInt8}((120,)) UInt8[0x29, 0x0f, 0x9c, 0x3b, 0x11, 0x5a, 0xbf, 0x3f, 0x29, 0xa6 … 0xb8, 0x3f, 0x6b
, 0x4a, 0x31, 0xab, 0x2b, 0xc5, 0xf2, 0x3f]
Store
- basic builtin types. e.g. `Integer`, `String`, `Tuple`, `Dict`, `Vector`, `Set` ...
julia> x
Dict{Any,Any} with 9 entries:
"int" => 42
"list" => Any["February", 14, 2012]
"str" => "Julia!"
"set" => Set(Any[2, 3, 21, 12, 1])
"bigint" => 1234567890987654321012345678909876543210
"bytes" => UInt8[0x31, 0x32, 0x33, 0x34]
"tuple" => (1, 2.0, "3", UInt8[0x34])
"bool" => Dict{Any,Any}(false=>true,true=>false)
"float" => 3.14159
julia> store("./test.pkl", x)
julia> stores(x)
"\x80\x04}\x94(\x8c\x03int\x94K*\x8c\x04list\x94]\x94(\x8c\bFebruary\x94K\x0eM\xdc\ae\x8c\x03str\x94\x8c\x06Julia!\x94\x8c\x03set\x94\x8f\x94(K\x02K\x03K\x15K\fK\x01\x90\x8c\x06bigint\x94\x8a\x11\xea\x1e\xd9Z7\xff\xad9[e;\xa9\x80 ɠ\x03\x8c\x05bytes\x94C\x041234\x8c\x05tuple\x94(K\x01G@\0\0\0\0\0\0\0\x8c\x013\x94C\x014t\x94\x8c\x04bool\x94}\x94(\x89\x88\x88\x89u\x8c\x05float\x94G@\t!\xfbM\x12\xd8Ju."
julia> load("./test.pkl")
Dict{Any,Any} with 9 entries:
"int" => 42
"list" => Any["February", 14, 2012]
"str" => "Julia!"
"set" => Set(Any[2, 3, 21, 12, 1])
"bigint" => 1234567890987654321012345678909876543210
"bytes" => UInt8[0x31, 0x32, 0x33, 0x34]
"tuple" => (1, 2.0, "3", UInt8[0x34])
"bool" => Dict{Any,Any}(false=>true,true=>false)
"float" => 3.14159
Pickle.Torch
We also support loading/saving the tensor data from/for pytorch.
julia> Pickle.Torch.THsave("mydata.bin", [randn(3,5), randn(5)])
julia> Pickle.Torch.THload("mydata.bin")
2-element Array{Any,1}:
[1.5106877710095366 -1.1454729135625932 … 2.06558662039955 -1.5367586535984377; 0.039481538567394656 -0.32939192495490544 … 1.3092722093574312 -2.008938993198881; -1.208358021687811 1.207098188115399 … 0.40730876859947734 1.6270781822957923]
[-0.5909715360681883, -0.0948081699846433, -0.17734064360419854, 0.43085740457102734, -0.48091537835876497]