Please remove triton dependency for Windows users

[FurkanGozukara]

I am able to use V1 perfectly fine on Windows 10

But V2 depends on Triton

Even though I have installed pre-compiled Triton package still i am getting below error

Error: unsupported operand type(s) for +: 'NoneType' and 'str'

Traceback: Traceback (most recent call last):
  File "G:\CogVLM2_v1\Gradio_app_CogVLM_v2.py", line 182, in post
    outputs = model.generate(**inputs, **gen_kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\transformers\generation\utils.py", line 1989, in generate
    result = self._sample(
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\transformers\generation\utils.py", line 2932, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\accelerate\hooks.py", line 169, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "R:\HF_Models\modules\transformers_modules\THUDM\cogvlm2-llama3-chat-19B\2bf7de6892877eb50142395af14847519ba95998\modeling_cogvlm.py", line 649, in forward
    outputs = self.model(
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\accelerate\hooks.py", line 169, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "R:\HF_Models\modules\transformers_modules\THUDM\cogvlm2-llama3-chat-19B\2bf7de6892877eb50142395af14847519ba95998\modeling_cogvlm.py", line 403, in forward
    return self.llm_forward(
  File "R:\HF_Models\modules\transformers_modules\THUDM\cogvlm2-llama3-chat-19B\2bf7de6892877eb50142395af14847519ba95998\modeling_cogvlm.py", line 520, in llm_forward
    layer_outputs = checkpoint(custom(idx),
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\_compile.py", line 24, in inner
    return torch._dynamo.disable(fn, recursive)(*args, **kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\_dynamo\eval_frame.py", line 489, in _fn
    return fn(*args, **kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\_dynamo\external_utils.py", line 17, in inner
    return fn(*args, **kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\utils\checkpoint.py", line 489, in checkpoint
    ret = function(*args, **kwargs)
  File "R:\HF_Models\modules\transformers_modules\THUDM\cogvlm2-llama3-chat-19B\2bf7de6892877eb50142395af14847519ba95998\modeling_cogvlm.py", line 499, in custom_forward
    outputs = layer(
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\accelerate\hooks.py", line 169, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "R:\HF_Models\modules\transformers_modules\THUDM\cogvlm2-llama3-chat-19B\2bf7de6892877eb50142395af14847519ba95998\modeling_cogvlm.py", line 262, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\accelerate\hooks.py", line 169, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "R:\HF_Models\modules\transformers_modules\THUDM\cogvlm2-llama3-chat-19B\2bf7de6892877eb50142395af14847519ba95998\modeling_cogvlm.py", line 205, in forward
    query_states, key_states = self.rotary_emb(query_states, key_states, position_ids=position_ids, max_seqlen=position_ids.max() + 1)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\accelerate\hooks.py", line 169, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "R:\HF_Models\modules\transformers_modules\THUDM\cogvlm2-llama3-chat-19B\2bf7de6892877eb50142395af14847519ba95998\util.py", line 469, in forward
    q = apply_rotary_emb_func(
  File "R:\HF_Models\modules\transformers_modules\THUDM\cogvlm2-llama3-chat-19B\2bf7de6892877eb50142395af14847519ba95998\util.py", line 329, in apply_rotary_emb
    return ApplyRotaryEmb.apply(
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\torch\autograd\function.py", line 553, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "R:\HF_Models\modules\transformers_modules\THUDM\cogvlm2-llama3-chat-19B\2bf7de6892877eb50142395af14847519ba95998\util.py", line 255, in forward
    out = apply_rotary(
  File "R:\HF_Models\modules\transformers_modules\THUDM\cogvlm2-llama3-chat-19B\2bf7de6892877eb50142395af14847519ba95998\util.py", line 212, in apply_rotary
    rotary_kernel[grid](
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\triton\runtime\jit.py", line 166, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\triton\runtime\jit.py", line 348, in run
    device = driver.get_current_device()
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\triton\runtime\driver.py", line 230, in __getattr__
    self._initialize_obj()
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\triton\runtime\driver.py", line 227, in _initialize_obj
    self._obj = self._init_fn()
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\triton\runtime\driver.py", line 260, in initialize_driver
    return CudaDriver()
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\triton\runtime\driver.py", line 122, in __init__
    self.utils = CudaUtils()
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\triton\runtime\driver.py", line 69, in __init__
    so = _build("cuda_utils", src_path, tmpdir)
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\triton\common\build.py", line 89, in _build
    cuda_lib_dirs = libcuda_dirs()
  File "G:\CogVLM2_v1\CogVLM2\venv\lib\site-packages\triton\common\build.py", line 26, in libcuda_dirs
    return [os.environ.get("CUDA_PATH") + "\\lib\\x64"]
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

and here below is my pip freeze

(venv) G:\CogVLM2_v1\CogVLM2\venv\Scripts>pip freeze
accelerate==0.33.0
aiofiles==23.2.1
aiohttp==3.9.5
aiosignal==1.3.1
altair==5.3.0
annotated-types==0.7.0
anyio==3.7.1
anykeystore==0.2
apex==0.9.10.dev0
async-timeout==4.0.3
asyncer==0.0.2
attrs==23.2.0
bidict==0.23.1
bitsandbytes==0.43.2
blinker==1.8.2
blis==0.7.11
boto3==1.34.149
botocore==1.34.149
braceexpand==0.1.7
cachetools==5.4.0
catalogue==2.0.10
certifi==2024.7.4
chainlit==1.1.306
charset-normalizer==3.3.2
chevron==0.14.0
click==8.1.7
cloudpathlib==0.18.1
colorama==0.4.6
confection==0.1.5
contourpy==1.2.1
cpm-kernels==1.0.11
cryptacular==1.6.2
cycler==0.12.1
cymem==2.0.8
dataclasses-json==0.5.14
datasets==2.20.0
deepspeed @ https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/deepspeed-0.11.2_cuda121-cp310-cp310-win_amd64.whl
defusedxml==0.7.1
Deprecated==1.2.14
dill==0.3.8
distro==1.9.0
dnspython==2.6.1
einops==0.8.0
email_validator==2.2.0
exceptiongroup==1.2.2
fastapi==0.110.3
fastapi-cli==0.0.4
ffmpy==0.3.2
filelock==3.13.1
filetype==1.2.0
fonttools==4.53.1
frozenlist==1.4.1
fsspec==2024.2.0
gitdb==4.0.11
GitPython==3.1.43
googleapis-common-protos==1.63.2
gradio==4.39.0
gradio_client==1.1.1
greenlet==3.0.3
grpcio==1.65.1
h11==0.14.0
hjson==3.1.0
httpcore==1.0.5
httptools==0.6.1
httpx==0.27.0
huggingface-hub==0.24.2
hupper==1.12.1
idna==3.7
importlib_metadata==8.0.0
importlib_resources==6.4.0
Jinja2==3.1.3
jmespath==1.0.1
jsonlines==4.0.0
jsonschema==4.23.0
jsonschema-specifications==2023.12.1
kiwisolver==1.4.5
langcodes==3.4.0
language_data==1.2.0
Lazify==0.4.0
literalai==0.0.607
loguru==0.7.2
marisa-trie==1.2.0
markdown-it-py==3.0.0
MarkupSafe==2.1.5
marshmallow==3.21.3
matplotlib==3.9.1
mdurl==0.1.2
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.16
murmurhash==1.0.10
mypy-extensions==1.0.0
nest-asyncio==1.6.0
networkx==3.2.1
ninja==1.11.1.1
numpy==1.26.3
oauthlib==3.2.2
openai==1.37.0
opentelemetry-api==1.26.0
opentelemetry-exporter-otlp==1.26.0
opentelemetry-exporter-otlp-proto-common==1.26.0
opentelemetry-exporter-otlp-proto-grpc==1.26.0
opentelemetry-exporter-otlp-proto-http==1.26.0
opentelemetry-instrumentation==0.47b0
opentelemetry-proto==1.26.0
opentelemetry-sdk==1.26.0
opentelemetry-semantic-conventions==0.47b0
orjson==3.10.6
packaging==23.2
pandas==2.2.2
PasteDeploy==3.1.0
pbkdf2==1.3
pillow==10.2.0
plaster==1.1.2
plaster-pastedeploy==1.0.1
preshed==3.0.9
protobuf==4.25.4
psutil==6.0.0
py-cpuinfo==9.0.0
pyarrow==17.0.0
pyarrow-hotfix==0.6
pydantic==2.8.2
pydantic_core==2.20.1
pydeck==0.9.1
pydub==0.25.1
Pygments==2.18.0
PyJWT==2.8.0
pynvml==11.5.3
pyparsing==3.1.2
pyramid==2.0.2
pyramid-mailer==0.15.1
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-engineio==4.9.1
python-multipart==0.0.9
python-socketio==5.11.3
python3-openid==3.2.0
pytz==2024.1
PyYAML==6.0.1
referencing==0.35.1
regex==2024.7.24
repoze.sendmail==4.4.1
requests==2.32.3
requests-oauthlib==2.0.0
rich==13.7.1
rpds-py==0.19.1
ruff==0.5.5
s3transfer==0.10.2
safetensors==0.4.3
seaborn==0.13.2
semantic-version==2.10.0
sentencepiece==0.2.0
shellingham==1.5.4
simple-websocket==1.0.0
six==1.16.0
smart-open==7.0.4
smmap==5.0.1
sniffio==1.3.1
spacy==3.7.5
spacy-legacy==3.0.12
spacy-loggers==1.0.5
SQLAlchemy==2.0.31
srsly==2.4.8
sse-starlette==2.1.2
starlette==0.37.2
streamlit==1.37.0
SwissArmyTransformer==0.4.11
sympy==1.12
syncer==2.0.3
tenacity==8.5.0
tensorboardX==2.6.2.2
thinc==8.2.5
timm==1.0.7
tokenizers==0.19.1
toml==0.10.2
tomli==2.0.1
tomlkit==0.12.0
toolz==0.12.1
torch==2.2.0+cu121
torchaudio==2.2.0+cu121
torchvision==0.17.0+cu121
tornado==6.4.1
tqdm==4.66.4
transaction==4.0
transformers==4.43.2
translationstring==1.4
triton @ https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/triton-2.1.0-cp310-cp310-win_amd64.whl
typer==0.12.3
typing-inspect==0.9.0
typing_extensions==4.9.0
tzdata==2024.1
uptrace==1.26.0
urllib3==2.2.2
uvicorn==0.25.0
velruse==1.1.1
venusian==3.1.0
wasabi==1.1.3
watchdog==4.0.1
watchfiles==0.20.0
weasel==0.4.1
webdataset==0.2.86
WebOb==1.8.7
websockets==11.0.3
win32-setctime==1.1.0
wrapt==1.16.0
wsproto==1.2.0
WTForms==3.1.2
wtforms-recaptcha==0.3.2
xformers==0.0.24
xxhash==3.4.1
yarl==1.9.4
zipp==3.19.2
zope.deprecation==5.0
zope.interface==6.4.post2
zope.sqlalchemy==3.1

[FurkanGozukara]

I hope you can help @zRzRzRzRzRzRzR @Sleepychord @huangshiyu13

[FurkanGozukara]

by the way with same libraries (python 3.10 venv with cuda 11.8) i installed on ubuntu and works perfect. so error is related to windows

[sipie800]

There are PaliGemma or Florence-v2 which work well on win.

[FurkanGozukara]

There are PaliGemma or Florence-v2 which work well on win.

we made cogvlm work on windows as well but very very slow due to some error we made i think

#172

[tarunabhdutta]

Yes please look into this matter, I think it is very much possible to get Cog VLM 2 working fine on windows