How to deal with references to ~/emacs.d?

[tardigradus]

After following the installation instructions and running emacs on the command line, I get the error

File is missing: Opening directory, No such file or directory, /home/tardigradis/.emacs.d/lisp

This is because I have

(let ((default-directory  "~/.emacs.d/lisp/"))
  (normal-top-level-add-subdirs-to-load-path))

in my init.el. How should such references be dealt with?