Qwen/Qwen-VL · Hugging Face

Models Text Generation
1

Qwen-VL

1573×442 56.2 KB

Qwen-VL :robot: | :hugs: ｜ Qwen-VL-Chat :robot: | :hugs: ｜ Qwen-VL-Chat-Int4 :hugs:
WeChat | Discord | Demo ｜ Report

Qwen-VL (Qwen Large Vision Language Model) is the visual multimodal version of the large model series, Qwen (abbr. Tongyi Qianwen), proposed by Alibaba Cloud. Qwen-VL accepts image, text, and bounding box as inputs, outputs text and bounding box. The features of Qwen-VL include:
We release Qwen-VL and Qwen-VL-Chat, which are pretrained model and Chat model respectively. For more details about Qwen-VL, please refer to our technical memo. This repo is the one for Qwen-VL.
Requirements

  • python 3.8 and above
  • pytorch 1.12 and above, 2.0 and above are recommended
  • CUDA 11.4 and above are recommended (this is for GPU user

Quickstart
Below, we provide simple examples to show how to use Qwen-VL with :hugs: Transformers.

Before running the code, make sure you have setup the environment and installed the required packages. Make sure you meet the above requirements, and then install the dependent libraries.

pip install -r requirements.txt

Now you can start with Transformers. More usage aboue vision encoder, please refer to tutorial.

:hugs: Transformers

To use Qwen-VL for the inference, all you need to do is to input a few lines of codes as demonstrated below. However, please make sure that you are using the latest code.

from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import GenerationConfig
import torch
torch.manual_seed(1234)

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-VL", trust_remote_code=True)

# use bf16
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-VL", device_map="auto", trust_remote_code=True, bf16=True).eval()
# use fp16
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-VL", device_map="auto", trust_remote_code=True, fp16=True).eval()
# use cpu only
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-VL", device_map="cpu", trust_remote_code=True).eval()
# use cuda device
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-VL", device_map="cuda", trust_remote_code=True).eval()

# Specify hyperparameters for generation (No need to do this if you are using transformers>=4.32.0)
# model.generation_config = GenerationConfig.from_pretrained("Qwen/Qwen-VL", trust_remote_code=True)

query = tokenizer.from_list_format([
    {'image': 'https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg'},
    {'text': 'Generate the caption in English with grounding:'},
])
inputs = tokenizer(query, return_tensors='pt')
inputs = inputs.to(model.device)
pred = model.generate(**inputs)
response = tokenizer.decode(pred.cpu()[0], skip_special_tokens=False)
print(response)
# <img>https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg</img>Generate the caption in English with grounding:<ref> Woman</ref><box>(451,379),(731,806)</box> and<ref> her dog</ref><box>(219,424),(576,896)</box> playing on the beach<|endoftext|>
image = tokenizer.draw_bbox_on_latest_picture(response)
if image:
  image.save('2.jpg')
else:
  print("no box")

2048×1365 235 KB

We evaluated the model’s ability from two perspectives:

  1. Standard Benchmarks: We evaluate the model’s basic task capabilities on four major categories of multimodal tasks:
  • Zero-shot Caption: Evaluate model’s zero-shot image captioning ability on unseen datasets;
  • General VQA: Evaluate the general question-answering ability of pictures, such as the judgment, color, number, category, etc;
  • Text-based VQA: Evaluate the model’s ability to recognize text in pictures, such as document QA, chart QA, etc;
  • Referring Expression Comprehension: Evaluate the ability to localize a target object in an image described by a referring expression.
  1. TouchStone: To evaluate the overall text-image dialogue capability and alignment level with humans, we have constructed a benchmark called TouchStone, which is based on scoring with GPT4 to evaluate the LVLM model.
  • The TouchStone benchmark covers a total of 300+ images, 800+ questions, and 27 categories. Such as attribute-based Q&A, celebrity recognition, writing poetry, summarizing multiple images, product comparison, math problem solving, etc;
  • In order to break the current limitation of GPT4 in terms of direct image input, TouchStone provides fine-grained image annotations by human labeling. These detailed annotations, along with the questions and the model’s output, are then presented to GPT4 for scoring.
  • The benchmark includes both English and Chinese versions.

The results of the evaluation are as follows:

Qwen-VL outperforms current SOTA generalist models on multiple VL tasks and has a more comprehensive coverage in terms of capability range.

717×495 65.5 KB

Zero-shot Captioning & General VQA)

Model type Model Zero-shot Captioning General VQA
NoCaps Flickr30K VQAv2dev OK-VQA GQA SciQA-Img
(0-shot) VizWiz
(0-shot)
Generalist
Models Flamingo-9B - 61.5 51.8 44.7 - - 28.8
Flamingo-80B - 67.2 56.3 50.6 - - 31.6
Unified-IO-XL 100.0 - 77.9 54.0 - - -
Kosmos-1 - 67.1 51.0 - - - 29.2
Kosmos-2 - 66.7 45.6 - - - -
BLIP-2 (Vicuna-13B) 103.9 71.6 65.0 45.9 32.3 61.0 19.6
InstructBLIP (Vicuna-13B) 121.9 82.8 - - 49.5 63.1 33.4
Shikra (Vicuna-13B) - 73.9 77.36 47.16 - - -
Qwen-VL (Qwen-7B) 121.4 85.8 78.8 58.6 59.3 67.1 35.2
Qwen-VL-Chat 120.2 81.0 78.2 56.6 57.5 68.2 38.9
Previous SOTA
(Per Task Fine-tuning) - 127.0
(PALI-17B) 84.5
(InstructBLIP
-FlanT5-XL) 86.1
(PALI-X
-55B) 66.1
(PALI-X
-55B) 72.1
(CFR) 92.53
(LLaVa+
GPT-4) 70.9
(PALI-X
-55B)

  • 在 Zero-shot Caption 中，Qwen-VL 在 Flickr30K 数据集上取得了 SOTA 的结果，并在 Nocaps 数据集上取得了和 InstructBlip 可竞争的结果。
  • 在 General VQA 中，Qwen-VL 取得了 LVLM 模型同等量级和设定下 SOTA 的结果。
  • For zero-shot image captioning, Qwen-VL achieves the SOTA on Flickr30K and competitive results on Nocaps with InstructBlip.
  • For general VQA, Qwen-VL achieves the SOTA under the same generalist LVLM scale settings.

文本导向的视觉问答 (Text-oriented VQA)

Model type Model TextVQA DocVQA ChartQA AI2D OCR-VQA
Generalist Models BLIP-2 (Vicuna-13B) 42.4 - - - -
InstructBLIP (Vicuna-13B) 50.7 - - - -
mPLUG-DocOwl (LLaMA-7B) 52.6 62.2 57.4 - -
Pic2Struct-Large (1.3B) - 76.6 58.6 42.1 71.3
Qwen-VL (Qwen-7B) 63.8 65.1 65.7 62.3 75.7
Specialist SOTAs
(Specialist/Finetuned) PALI-X-55B (Single-task FT)
(Without OCR Pipeline) 71.44 80.0 70.0 81.2 75.0
  • In text-related recognition/QA evaluation, Qwen-VL achieves the SOTA under the generalist LVLM scale settings.
  • Resolution is important for several above evaluations. While most open-source LVLM models with 224 resolution are incapable of these evaluations or can only solve these by cutting images, Qwen-VL scales the resolution to 448 so that it can be evaluated end-to-end. Qwen-VL even outperforms Pic2Struct-Large models of 1024 resolution on some tasks.

Referring Expression Comprehension)

Model type Model RefCOCO RefCOCO+ RefCOCOg GRIT
val test-A test-B val test-A test-B val-u test-u refexp
Generalist Models GPV-2 - - - - - - - - 51.50
OFA-L* 79.96 83.67 76.39 68.29 76.00 61.75 67.57 67.58 61.70
Unified-IO - - - - - - - - 78.61
VisionLLM-H 86.70 - - - - - - -
Shikra-7B 87.01 90.61 80.24 81.60 87.36 72.12 82.27 82.19 69.34
Shikra-13B 87.83 91.11 81.81 82.89 87.79 74.41 82.64 83.16 69.03
Qwen-VL-7B 89.36 92.26 85.34 83.12 88.25 77.21 85.58 85.48 78.22
Qwen-VL-7B-Chat 88.55 92.27 84.51 82.82 88.59 76.79 85.96 86.32 -
Specialist SOTAs
(Specialist/Finetuned) G-DINO-L 90.56 93.19 88.24 82.75 88.95 75.92 86.13 87.02 -
UNINEXT-H 92.64 94.33 91.46 85.24 89.63 79.79 88.73 89.37 -
ONE-PEACE 92.58 94.18 89.26 88.77 92.21 83.23 89.22 89.27 -

  • Qwen-VL achieves the SOTA in all above referring expression comprehension benchmarks.
  • Qwen-VL has not been trained on any Chinese grounding data, but it can still generalize to the Chinese Grounding tasks in a zero-shot way by training Chinese Caption data and English Grounding data.

We provide all of the above evaluation scripts for reproducing our experimental results. Please read eval/EVALUATION.md for more information.

Chat Evaluation)

TouchStone is a benchmark based on scoring with GPT4 to evaluate the abilities of the LVLM model on text-image dialogue and alignment levels with humans. It covers a total of 300+ images, 800+ questions, and 27 categories, such as attribute-based Q&A, celebrity recognition, writing poetry, summarizing multiple images, product comparison, math problem solving, etc. Please read touchstone/README_CN.md for more information.

English)

Model Score
PandaGPT 488.5
MiniGPT4 531.7
InstructBLIP 552.4
LLaMA-AdapterV2 590.1
mPLUG-Owl 605.4
LLaVA 602.7
Qwen-VL-Chat 645.2

中文 (Chinese)

Model Score
VisualGLM 247.1
Qwen-VL-Chat 401.2
Qwen-VL-Chat has achieved the best results in both Chinese and English alignment evaluation.
If you meet problems, please refer to FAQ and the issues first to search a solution before you launch a new issue.
Researchers and developers are free to use the codes and model weights of both Qwen-VL and Qwen-VL-Chat. We also allow their commercial use. Check our license at LICENSE for more details.
If you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil: :slight_smile: 
@article{Qwen-VL,
  title={Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities},
  author={Bai, Jinze and Bai, Shuai and Yang, Shusheng and Wang, Shijie and Tan, Sinan and Wang, Peng and Lin, Junyang and Zhou, Chang and Zhou, Jingren},
  journal={arXiv preprint arXiv:2308.12966},
  year={2023}
}

If you are interested to leave a message to either our research team or product team, feel free to send an email to [email protected].