[Fine-tuninig] bitsandbytes Error (for LoRA fine tuning)

Windows 환경에서 LoRA fine-tuning을 위해 bitsandbytes 패키지를 설치하려고 했으나, ‘bitsandbytes’ library는 원래 Windows를 지원하지 않는다고 한다..

 

그래도 방법이 있지 않을까??

직접여러 솔루션들을 수행해본 결과 3시간 만에 찾아냈다!!

---

Solution1 (Fail)

 

출처: CUDA error in Windows · Issue #689 · TimDettmers/bitsandbytes

CUDA error in Windows · Issue #689 · TimDettmers/bitsandbytes

 

 

1. Download bitsandbytes-0.39.1-py3-none-win_amd64.whl
2. cd [Download Directory]
3. pip install bitsandbytes-0.39.1-py3-none-win_amd64.whl
4. remove this code in C:\Users\UserName\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes_main_.py

    if 'CONDA_PREFIX' in os.environ:
        paths = find_file_recursive(os.environ['CONDA_PREFIX'], f'*cuda*{SHARED_LIB_EXTENSION}')
        print_header("ANACONDA CUDA PATHS")
        print(paths)
        print('')
    if 'CUDA_HOME' in os.environ:
        paths = find_file_recursive(os.environ['CONDA_PREFIX'], f'*cuda*{SHARED_LIB_EXTENSION}')
        print_header("CUDA_HOME CUDA PATHS")
        print(paths)
        print('')
    elif 'CUDA_PATH' in os.environ:
        paths = find_file_recursive(os.environ['CONDA_PREFIX'], f'*cuda*{SHARED_LIB_EXTENSION}')
        print_header("CUDA_PATH CUDA PATHS")
        print(paths)
        print('')

 

 

python -m bitsandbytes

C:\Users\Master>python -m bitsandbytes

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll
CUDA SETUP: CUDA runtime path found: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\cudart64_110.dll
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 117
CUDA SETUP: Loading binary C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll...
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

+++++++++++++++ WORKING DIRECTORY CUDA PATHS +++++++++++++++
C:\Users\Master\anaconda3\envs\stylegan3\bin\cudart32_110.dll
C:\Users\Master\anaconda3\envs\stylegan3\bin\cudart64_110.dll
C:\Users\Master\anaconda3\envs\stylegan3\lib\site-packages\torch\lib\c10_cuda.dll
C:\Users\Master\anaconda3\envs\stylegan3\lib\site-packages\torch\lib\torch_cuda.dll
C:\Users\Master\anaconda3\envs\stylegan3\Library\bin\cudart64_110.dll
C:\Users\Master\anaconda3\pkgs\cuda-cudart-dev-11.8.89-0\bin\cudart32_110.dll
C:\Users\Master\anaconda3\pkgs\cuda-cudart-dev-11.8.89-0\bin\cudart64_110.dll
C:\Users\Master\anaconda3\pkgs\cudatoolkit-11.1.1-hb074779_11\Library\bin\cudart64_110.dll
C:\Users\Master\anaconda3\pkgs\cudatoolkit-11.8.0-h09e9e62_11\Library\bin\cudart64_110.dll
C:\Users\Master\anaconda3\pkgs\pytorch-2.0.1-py3.9_cuda11.8_cudnn8_0\Lib\site-packages\torch\lib\c10_cuda.dll
C:\Users\Master\anaconda3\pkgs\pytorch-2.0.1-py3.9_cuda11.8_cudnn8_0\Lib\site-packages\torch\lib\torch_cuda.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda111.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda111_nocublaslt.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda112.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda112_nocublaslt.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda113.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda113_nocublaslt.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda114.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda114_nocublaslt.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda115.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda115_nocublaslt.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda116.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda116_nocublaslt.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda117_nocublaslt.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda118_nocublaslt.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda120.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda120_nocublaslt.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda121.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda121_nocublaslt.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda122.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda122_nocublaslt.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\cuda_setup\libbitsandbytes_cuda116.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\lib\c10_cuda.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\lib\cudart64_110.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\lib\torch_cuda.dll
C:\Users\Master\AppData\Local\Programs\Python\Python311\Lib\site-packages\torchvision\cudart64_110.dll
C:\Users\Master\AppData\Local\Programs\Python\Python37\Lib\site-packages\torch\lib\c10_cuda.dll
C:\Users\Master\AppData\Local\Programs\Python\Python37\Lib\site-packages\torch\lib\cudart64_110.dll
C:\Users\Master\AppData\Local\Programs\Python\Python37\Lib\site-packages\torch\lib\torch_cuda.dll
C:\Users\Master\AppData\Local\Programs\Python\Python37\Lib\site-packages\torch\lib\torch_cuda_cpp.dll
C:\Users\Master\AppData\Local\Programs\Python\Python37\Lib\site-packages\torch\lib\torch_cuda_cu.dll
C:\Users\Master\AppData\Local\Programs\Python\Python37\Lib\site-packages\torchvision\cudart64_110.dll
C:\Users\Master\Downloads\libbitsandbytes_cuda116.dll

++++++++++++++++++ LD_LIBRARY CUDA PATHS +++++++++++++++++++
 C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\lib\x64 CUDA PATHS
Could not read LD_LIBRARY_PATH: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\lib\x64

++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = True
COMPUTE_CAPABILITIES_PER_GPU = ['8.6']
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Running a quick check that:
    + library is importable
    + CUDA function is callable

WARNING: Please be sure to sanitize sensible info from any such env vars!

SUCCESS!
Installation was successful!

---

 

Solution2 (Fail..)

 

pip install bitsandbytes-windows

 

 

Windows를 위한  bitsandbytes-windows 패키지이다. 하지만 나 같은 경우에는 정상적으로 동작하지 않았다.

---

 

Solution3 (Sovled!)

 

1. windwos 지원하는 bitsandbytes 설치

 

python -m pip install bitsandbytes --prefer-binary --extra-index-url=https://jllllll.github.io/bitsandbytes-windows-webui

 

출처: GitHub - jllllll/bitsandbytes-windows-webui: Windows compile of bitsandbytes for use in text-generation-webui.

GitHub - jllllll/bitsandbytes-windows-webui: Windows compile of bitsandbytes for use in text-generation-webui.

 

 

2. package 편집

 

 

in bitandbytes/ main .py

#  def _decode()
    def _decode(subprocess_err_out_tuple):
        return tuple(
            to_decode.decode("UTF-8").strip()
            for to_decode in subprocess_err_out_tuple
        )
## change
    def _decode(subprocess_err_out_tuple: Tuple[bytes, bytes]) -> Tuple[str, str]:
        std_out, std_err = subprocess_err_out_tuple
        try:
            out = std_out.decode("UTF-8").strip()
        except UnicodeDecodeError:
            out = std_out.decode("ISO-8859-1").strip()
        try:
            err = std_err.decode("UTF-8").strip()
        except UnicodeDecodeError:
            err = std_err.decode("ISO-8859-1").strip()
        return out, err

# def find_file_recursive()
def find_file_recursive(folder, filename):
    cmd = f'find {folder} -name {filename}' if not IS_WINDOWS_PLATFORM else f'where /R "{folder}" "{filename}"'
    out, err = execute_and_return(cmd)
    if len(err) > 0:
        raise RuntimeError('Something when wrong when trying to find file.')

    return out

## change
def find_file_recursive(folder, filename):
    cmd = f'find {folder} -name {filename}' if not IS_WINDOWS_PLATFORM else f'where /R "{folder}" "{filename}"'
    out, err = execute_and_return(cmd)
    if len(err) > 0:
        if 'Á¤º¸: Á¦°øµÈ ÆÐÅÏ¿¡ ÇØ´çµÇ´Â ÆÄÀÏÀ» ãÁö ¸øÇß½À´Ï´Ù.' in err:
            print(f"No files matching the pattern {filename} were found in {folder}.")
            return ''
        else:
            print(f"Error occurred while executing command: {cmd}")
            print(f"Error message: {err}")
            raise RuntimeError('Something went wrong when trying to find file.')
    return out

 

in bitandbytes/ cextension.py

change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path))

 

in cuda_setup/main.py:

def evaluate_cuda_setup():
    cuda_setup = CUDASetup.get_instance()
    if 'BITSANDBYTES_NOWELCOME' not in os.environ or str(os.environ['BITSANDBYTES_NOWELCOME']) == '0':
        cuda_setup.add_log_entry('')
        cuda_setup.add_log_entry('='*35 + 'BUG REPORT' + '='*35)
        cuda_setup.add_log_entry(('Welcome to bitsandbytes. For bug reports, please run\n\npython -m bitsandbytes\n\n'),
              ('and submit this information together with your error trace to: https://github.com/jllllll/bitsandbytes/issues'))
        cuda_setup.add_log_entry('='*80)
    if not torch.cuda.is_available(): return "libbitsandbytes_cuda116.dll", None, None, None

 

3. python -m bitsandbytes

 

터미널에서:

python -m bitsandbytes