QI JUN
QI JUN
In Python, the [range](https://www.w3schools.com/python/ref_func_range.asp) function returns a sequence of numbers, starting from 0 by default, and increments by 1 (by default), and stops before a specified number. For example: ```...
# What does this PR do ? Polish the batch matmul code, make it more pytorchnic
#279 Python provides [cProfile](https://docs.python.org/2/library/profile.html#module-cProfile) module to help us profiling python codes. A common usage can be: `python -m cProfile -o train.prof train.py` Since train.prof is a binary file, we have...
- [x] Revise and polish elasticdl local tutorial, set up a mnist training experiment with minikube quickly #1912 - [ ] Add a deployment tutorial, save model when training, and...
## 背景介绍 PyTorch的 [Dataset class](https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class)定义 我们可以发现，PyTorch要求Dataset必须提供 `__len__` 接口和 `__getitem__`接口，这就要求 数据集是已知长度的，并且是可以被随机访问的。 这里与TensorFlow不同，TensorFlow的Dataset是可以从一个generator创建的，generator只要求用户实现 `__next__`接口即可，并不要求 `__len__` 接口和 `__getitem__` 接口。 因此，我们需要提出一种新的思路。 ## 简单的做法 1. worker从master那里拿到一个task 2. worker 使用 recordio_reader提供的接口，把该task包含的record都读到内存中 3. records是一个即知道长度，又可以随机访问的数组，我们可以从这个数组中创建一个 RecordDataset 4. RecordDataset中，每一个record都是string类型的，用户需要提供一个feed函数把string类型转换为数值类型。我们发现，这个feed函数实际上就是...
We generate a checkpoint using this script https://github.com/sql-machine-learning/elasticdl/blob/develop/scripts/travis/gen_mnist_checkpoint.py. Then we load the checkpoint to a job with two PS pods. Let's verify the consistency between Python PS and Go PS.
```python CategoryCrossing(PreprocessingLayer): """This layer transforms multiple categorical inputs to categorical outputs by Cartesian product, and hash the output if necessary. If any of the inputs is sparse, then all outputs...
I find that there is some keras preprocessing layer in [census_wide_deep_model](https://github.com/sql-machine-learning/elasticdl/blob/develop/model_zoo/census_wide_deep_model/keras_process_layer.py). Could we move these preprocessing layer into elasticdl preprocessing directory? Or if the preprocessing layer has been implemented already,...
In TensorFlow 1.x, or similar computation graph based deep learning framework, we could serialize the model into a protobuf file. And then, we can split the file into two parts,...
Step 1: - [ ] embedding table Python data structure - [ ] tensor proto message and Python data structure - [ ] a new PS, which binds to a...