how to save 4bit/2bit models?

[xiguadong]

Ask a Question

Question

I have a QAT quantized model , but the type of weight is 4bit/2bit. Could I save it to onnx format? I found that onnx.proto only support type as belows:

onnx/onnx/onnx.proto

Line 484 in 5b1346e

message TensorProto {

[jcwchen]

Hi @xiguadong,
You are right -- as you can see in the proto definition, currently 4 bit/2 bit models are not supported.

[xiguadong]

so the next question is:
how do I support the 4/2bit? could you tell me the steps?
thk u very much

[jcwchen]

If you are looking for a future official support, this proposal will need to be reviewed by onnx/sig-archinfra and onnx/sig-operators since it's a fundamental change in IR. Then the frontend converter and backend runtime will start to support it. The process will take quite a while. May I understand more about your use case?

To sort of workaround it, you can try to modify ONNX locally and build it from source. An example PR for adding bfloat16: #1421. However, please note that probably most runtimes for ONNX have not supported 2 bit/4 bit models yet for now since such a type is not in the IR.

[xiguadong]

I work for a npu chip company, we have some 2bit/4bit models runs on our chip. We decide to use onnx format save the 2bit/4bit cause the format we used before performs bad. But onnx is very beautiful and easy to use! And we can test the new spec soon if official support it.
It seems like 2bit/4bit has been a trend. I noticed that openvino/cuda/torch has support it. And , I guess onnxruntime also support 4bit quantization although save it with 8bit https://github.com/microsoft/onnxruntime/blob/207ad7eef9f91f88312cc28b2528bdc6362eaadb/onnxruntime/python/tools/quantization/onnx_quantizer.py#L51
Btw, we also plan to push our engine to github this week, this is our repo（ :）its empty now). And we could send you some interesting demo board in the future.

Thanks !

[elephantpanda]

Yes please support 4bit, 2bit and 1bit.

[maziyarpanahi]

I am not sure about the 2bit or less as I've never used them. But I can see at least in NLP the 4-bit is not that far from the 16-bit when it comes to perplexity (in some use cases this trade-off between speed and accuracy might be very interesting):

ggml-org/llama.cpp#270 (comment)

[elephantpanda]

I tried to convert the 4bit Llama model to Onnx. It started with a 3GB files and the resulting onnx was 0.5GB so obviously it didn't work. The way the 4bit llama works is that they added a custom CUDA command in the GPTQ quantization. Which I don't think is supported by onnxruntime.

Presumably the actual onnx would just need to be saved in 8bit, since I think the CUDA cores take the 8bit and apply things to simulate 4bit instructions without actually using 4bit instructions.

Anyway, if someone can get 4bit Llama model running with ONNX that would be a good advance in AI.

I can reiterate the point is to save VRAM by 2x which can save cost by 2x. Which would be a major advance.

Since LLAMA is made my Microsoft you should have access to the model. (I'm surprised a Microsoft model converted to 4bit cannot be run with Microsoft software!)

See also: microsoft/onnxruntime#14997

[nenkoru]

I tried to convert the 4bit Llama model to Onnx. It started with a 3GB files and the resulting onnx was 0.5GB so obviously it didn't work. The way the 4bit llama works is that they added a custom CUDA command in the GPTQ quantization. Which I don't think is supported by onnxruntime.

Presumably the actual onnx would just need to be saved in 8bit, since I think the CUDA cores take the 8bit and apply things to simulate 4bit instructions without actually using 4bit instructions.

Anyway, if someone can get 4bit Llama model running with ONNX that would be a good advance in AI.

I can reiterate the point is to save VRAM by 2x which can save cost by 2x. Which would be a major advance.

Since LLAMA is made my Microsoft you should have access to the model. (I'm surprised a Microsoft model converted to 4bit cannot be run with Microsoft software!)

See also: microsoft/onnxruntime#14997

Did you manage to run 8bit quantized version using GPTQ method though?

[elephantpanda]

I tried to convert the 4bit Llama model to Onnx. It started with a 3GB files and the resulting onnx was 0.5GB so obviously it didn't work. The way the 4bit llama works is that they added a custom CUDA command in the GPTQ quantization. Which I don't think is supported by onnxruntime.
Presumably the actual onnx would just need to be saved in 8bit, since I think the CUDA cores take the 8bit and apply things to simulate 4bit instructions without actually using 4bit instructions.
Anyway, if someone can get 4bit Llama model running with ONNX that would be a good advance in AI.
I can reiterate the point is to save VRAM by 2x which can save cost by 2x. Which would be a major advance.
Since LLAMA is made my Microsoft you should have access to the model. (I'm surprised a Microsoft model converted to 4bit cannot be run with Microsoft software!)
See also: microsoft/onnxruntime#14997

Did you manage to run 8bit quantized version using GPTQ method though?

Yes, I believe so. However, as I recall, I don't think my computer had enough RAM to convert it to onnx format. Well whatever the case I haven't so far got an int8 llama model running with onnx. If anyone achieves this, let me know!

[nenkoru]

I tried to convert the 4bit Llama model to Onnx. It started with a 3GB files and the resulting onnx was 0.5GB so obviously it didn't work. The way the 4bit llama works is that they added a custom CUDA command in the GPTQ quantization. Which I don't think is supported by onnxruntime.

Presumably the actual onnx would just need to be saved in 8bit, since I think the CUDA cores take the 8bit and apply things to simulate 4bit instructions without actually using 4bit instructions.

Anyway, if someone can get 4bit Llama model running with ONNX that would be a good advance in AI.

I can reiterate the point is to save VRAM by 2x which can save cost by 2x. Which would be a major advance.

Since LLAMA is made my Microsoft you should have access to the model. (I'm surprised a Microsoft model converted to 4bit cannot be run with Microsoft software!)

See also: microsoft/onnxruntime#14997

Did you manage to run 8bit quantized version using GPTQ method though?

Yes, I believe so. However, as I recall, I don't think my computer had enough RAM to convert it to onnx format. Well whatever the case I haven't so far got an int8 llama model running with onnx. If anyone achieves this, let me know!

You might actually want to take a look into two things:

huggingface/optimum#922

https://huggingface.co/nenkoru/alpaca-lora-7b-fp16-onnx

I managed to convert alpaca-lora-7b to a ONNX format. I haven't tried quantizing yet.

[changtimwu]

some supporting data
https://arxiv.org/abs/2301.12017

[xiguadong]

We can try quantize llama.onnx with 4bit , but how to save it?

https://github.com/tpoisonooo/llama.onnx