[wip] QLoRA with bias + Llama 3.2 Vision QLoRA configs

[ebsmothers]

After opening https://github.com/pytorch/ao/issues/979 on torchao, it was pointed out to me that I was overcomplicating things.. we can just keep the bias in bf16, which is apparently a pretty standard thing to do (ref).

So this PR does exactly that.. just let bias stay in the higher precision for both our FrozenNF4Linear and our LoRALinear when we set quantize_base=True. I am not tackling DoRA rn as that's a bit more effort.

Big TODO here: add an actual unit test with bias=True for both of these classes.

But can confirm that

tune run lora_finetune_single_device --config llama3_2_vision/11B_qlora_single_device

and

tune run --nproc_per_node 2 lora_finetune_distributed --config llama3_2_vision/11B_qlora

both run.

[pytorch-bot[bot]]

:link: Helpful Links

:test_tube: See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1726

• :page_facing_up: Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

:white_check_mark: No Failures

As of commit 09491e61c7bf022a443ef8dc4942bf53f9b81906 with merge base 17ba37d281548e5d60215f741888066717ad5b3e (): :green_heart: Looks good so far! There are no failures yet. :green_heart:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 92.63158% with 7 lines in your changes missing coverage. Please review.

Project coverage is 67.89%. Comparing base (73aa126) to head (09491e6). Report is 399 commits behind head on main.

Files with missing lines	Patch %	Lines
...tune/models/llama3_2_vision/_component_builders.py	46.15%	7 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1726      +/-   ##
==========================================
- Coverage   70.25%   67.89%   -2.36%     
==========================================
  Files         309      308       -1     
  Lines       16285    16301      +16     
==========================================
- Hits        11441    11068     -373     
- Misses       4844     5233     +389     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.