Video-Text-to-Text
Transformers
Safetensors
sam2
English
vica_qwen
text-generation
multimodal
vision-language
video understanding
visuospatial cognition
spatial reasoning
vlm
llava
qwen
siglip
hiera
dual-encoder
Instructions to use nkkbr/ViCA2-init with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use nkkbr/ViCA2-init with Transformers:
# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("nkkbr/ViCA2-init", device_map="auto") - sam2
How to use nkkbr/ViCA2-init with sam2:
# Use SAM2 with images import torch from sam2.sam2_image_predictor import SAM2ImagePredictor predictor = SAM2ImagePredictor.from_pretrained(nkkbr/ViCA2-init) with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): predictor.set_image(<your_image>) masks, _, _ = predictor.predict(<input_prompts>)# Use SAM2 with videos import torch from sam2.sam2_video_predictor import SAM2VideoPredictor predictor = SAM2VideoPredictor.from_pretrained(nkkbr/ViCA2-init) with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): state = predictor.init_state(<your_video>) # add new prompts and instantly get the output on the same frame frame_idx, object_ids, masks = predictor.add_new_points(state, <your_prompts>): # propagate the prompts to get masklets throughout the video for frame_idx, object_ids, masks in predictor.propagate_in_video(state): ... - Notebooks
- Google Colab
- Kaggle
File size: 747 Bytes
2b77ece | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | ---
license: apache-2.0
tags:
- multimodal
- vision-language
- video understanding
- visuospatial cognition
- spatial reasoning
- vlm
- llava
- qwen
- siglip
- hiera
- sam2
- dual-encoder
language:
- en
library_name: transformers
pipeline_tag: video-text-to-text
model_name: ViCA2-7B-Init
---
## Usage and Full Documentation
For detailed model description, training setup, datasets, evaluation results, and inference code, **please refer to the following links**:
[](https://github.com/nkkbr/ViCA)
[](https://huggingface.co/nkkbr/ViCA2) |