Rebuilding the kernel to enable swap

[craic]

I've just run into this problem...

"Note: L4T 28.1 does not have the swap file option selected in the stock kernel, so a custom kernel must be used with swap enabled. The option is 'Support for paging of anonymous memory (swap)'. The kernel configuration symbols are CONFIG_SWAP and SWAP."

Any advice on rebuilding the kernel? I've not attempted that before...

My kernel version is 4.4.38-tegra

The various resources on the web are a bit daunting/confusing

It's annoying that NVidia say they will add this option... over a year ago...

thanks!

[jetsonhacks]

There are a couple of ways to rebuild the kernel. The official NVIDIA recommendation is to use a cross compiler on the host PC, as covered in their documentation.

Personally, I just build it on the Jetson itself, much like this article: http://www.jetsonhacks.com/2017/08/07/build-kernel-ttyacm-module-nvidia-jetson-tx1/

I will note that actually building the kernel and compiling TensorFlow can be rather challenging.

[kaisark]

I added a USB driver module for the SPE and rebuilt the kernel on the TX1 according to your instructions. The build worked well. Will we see a similar style blog post for adding CONFIG_SWAP and rebuilding the kernel? BTW, I have installed Tensorflow on the Raspberry pi 3 (Jessie 8) and that took some time and swap was required...

Linux tegra-ubuntu 4.4.38-jetsonbot-v0.1 SMP PREEMPT Sun Sep 17 23:12:25 EDT 2017 aarch64 aarch64 aarch64 GNU/Linux

https://github.com/samjabrahams/tensorflow-on-raspberry-pi/blob/master/GUIDE.md#2-install-a-memory-drive-as-swap-for-compiling

Disregard. I see it now. http://www.jetsonhacks.com/2017/09/18/build-tensorflow-on-nvidia-jetson-tx1-development-kit/

[kaisark]

All set now. TF installed after I rebuilt the kernel with swap. I then used a 8 GB USB drive and followed the instructions below for adding swap:

https://askubuntu.com/questions/173676/how-to-make-a-usb-stick-swap-disk

nvidia@tegra-ubuntu:~$ grep Swap /proc/meminfo SwapCached: 63172 kB SwapTotal: 7973884 kB SwapFree: 7575092 kB

nvidia@tegra-ubuntu:~$ cat /proc/swaps Filename Type Size Used Priority /dev/sda1 partition 7973884 398792 5

1. Make sure your USB stick is connected. In a terminal type:

sudo blkid

It will list all your connected drives. Search for your USB stick you want to use as swap and copy the UUID (everything inside these quotes UUID="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX").

This represents the individual name of your device.

Also note the /dev/XXX point of your device.

2. Now unmount your device by typing:

sudo umount /dev/XXX (where XXX represents your device name)

3. Format your USB stick as swap, e.g. by terminal

sudo mkswap /dev/XXX (<-- be sure to use the correct device name here or you'll probably end up formatting the wrong drive!)

or simply use GParted.

4. Now you have to edit your /etc/fstab file, so type

gksudo gedit /etc/fstab

and enter the following

UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX none swap sw,pri=5 0 0

(for all the Xs, use the UUID number you got by typing sudo blkid)

5. Now type

sudo swapon -a

That's it, this should already work.

[jetsonhacks]

@kaisark - Glad you got it to work!

[CoderCoderCoder]

@kaisark @jetsonhacks Hello, I am trying to rebuild the kernel for enabling the swap as well. Huge thanks for your great articles and information in this regard! The official Nvidia "manual" is an absolute piece of crap, but with your information it seems to be doable. Just one question - "make xconfig" shows me 3 options in regard to swap - the first one is the main switch (CONFIG_SWAP), but there are two more - CONFIG_ZSWAP and CONFIG_FRONTSWAP - is it OK to enable them on Jetson TX1? I read through the docs and it seems that they could make swapping significantly faster, but both are quite new and CONFIG_ZSWAP is even "experimental". I would hugely appreciate any help, as it's my first time ever that I need to recompile/rebuild a Linux kernel =).

[jetsonhacks]

You should build the kernel that suits your needs. As for zswap and frontswap, I'm not sure why you would want to trade 20% of available memory in a memory constrained, shared video memory system for swap space. But if that's your need, go for it! Just make sure your kernel sources match the kernel being used.

[CoderCoderCoder]

@jetsonhacks Many thanks for your reply! As I want to train quite complex neural networks on TX1 (CNNs, RNNs, GANs) with a solid amount of data in some cases (and avoid using costly cloud computing for personal experimental stuff), I also need a lot of RAM (which is not present in TX1). So my plan is to make a 8+ GB swap file on a SSD drive and make the swapping as efficient and fast as possible. Does it make sense to enable zswap and frontswap in such case? Again - I have absolutely zero experience with kernel compilation, it was just based on some basic research of these options and what I assumed they would do. Any suggestions and recommendations are more than welcome, thank you!

[jetsonhacks]

If you are training large, complex networks the TX1 is a poor choice. The TX1 is a poor alternative to a desktop or laptop, more or less a cloud of GPUs. As for swap and memory cache, you should already know the answers to the questions you are asking, or run experiments to determine the suitable configuration. Swap memory and memory pressure is a complex subject matter, and usually you need empirical data to determine its effectiveness for any given application.

[kaisark]

The TX1 was not designed with Deep Learning training in mind. The TX1 is better thought of as a low powered IOT ("edge") device for running deep learning Inference (run-time) like object detection in the field. However, the TX1 only has 4 GB of memory and that is not enough to run a lot of the bigger deep learning networks in real time. With the TX1, everything turns into an optimization problem it seems. I think if budget is not an issue, the TX2 is a good choice (though pricier) for running deep learning at the edge. I have seen a lot of startups get funding while using the TX2 as an IOT solution.

When it comes to DL training, Cloud GPU/TPU providers like AWS and Google Cloud still reign supreme.

Concerning DL laptops/desktops, I have started similar research. It would be nice to carry around a deep learning machine to the neighborhood cafe? Another option (less mobile) is to connect an EGPU to a Macbook via thunderbolt connection.

example: laptop https://www.amazon.com/dp/B01NBOE95P/ref=nav_timeline_asin?_encoding=UTF8&psc=1

example: egpu https://www.amazon.com/Sonnet-Breakaway-Thunderbolt-Expansion-GPU-350W-TB3Z/dp/B072L9QTZT/ref=sr_1_1?s=electronics&ie=UTF8&qid=1518502846&sr=1-1&keywords=egpu+thunderbolt+3&dpID=41mJWRJWe-L&preST=SX300_QL70&dpSrc=srch