Tensor Parallel demo

[KimmiShi]

Hi, I have run SDXL with Tensor Parallel as well as sequence parallel. Below is my PR, and may it help those who need it.

The Motivation:
Just trying to avoid using grad checkpointing to get higher throughput when inputs have higher resolution like 720p.

However, tensor parallel comes at a cost, and I have not gained throughput by TP. (Tested with 720*1080 on A100, batchsize=16 and amp).

Just in case someone have the same idea or try to run tensor prarallel with more blocks, below is my code changes:

PR: support tensor parallel for sdxl

[sayakpaul]

Thanks for your efforts here.

However, tensor parallel comes at a cost, and I have not gained throughput by TP. (Tested with 720*1080 on A100, batchsize=16 and amp).

However, this tells me tensor parallelism wasn't of much use here?

[KimmiShi]

Thanks for your efforts here.

However, tensor parallel comes at a cost, and I have not gained throughput by TP. (Tested with 720*1080 on A100, batchsize=16 and amp).

However, this tells me tensor parallelism wasn't of much use here?

Yes. Only useful when model scales up or image with bigger resolution. I will close this.

[forechoandlook]

Hello, I need train sdxl with lora in 1080TI, which should use tensor parallel since the limited mem. Do you have the train demo? Very thanks.

[KimmiShi]

Hello, I need train sdxl with lora in 1080TI, which should use tensor parallel since the limited mem. Do you have the train demo? Very thanks.

The PR posted above can be used, but without lora.

[KimmiShi]

Hello, I need train sdxl with lora in 1080TI, which should use tensor parallel since the limited mem. Do you have the train demo? Very thanks.

@forechoandlook maybe you need qlora.

[forechoandlook]

thank you , i will try.