tongflow

TongFlow: The Open-Source Modality-First GenAI Platform

https://github.com/user-attachments/assets/946592d3-2e48-4948-aaa4-4b6ab3156209

English · 简体中文 · 日本語

Different modalities. One workflow.

Text, images, audio, video and 3D are your materials. Decide how they transform and combine, then choose the model behind each step. Every result is material you can use again.

Open-source core · Your models · Self-hostable · tongflow.com · Open the canvas

Sponsors

MiniMax H3 video generation API — Metaso (秘塔科技) Metaso runs MiniMax H3 video generation at ¥0.09/second for 768P and ¥0.15/second for 2K. Native 2K, synced audio and picture, an OpenAI-compatible API, and ComfyUI support — no GPU to deploy yourself. 🎁 Sign up through the TongFlow link to claim bonus credits and a partner discount. One key for the whole canvas — Infistar (无限星河AI) An OpenAI-compatible gateway: one key and one balance cover the canvas's text, image, video and transcription nodes — GPT-6, Claude Opus 5, Gemini 3 Pro, Grok and DeepSeek alongside the broadest Chinese lineup (Qwen, GLM, Doubao, MiniMax, Kimi, StepFun), plus GPT Image 2.5, Seedream 5 and Qwen-Image for stills and Wan / Seedance for video. Images from ¥0.06 each. Use it through the tongflow-router-infistar plugin. 🎁 Sign up through the TongFlow link for $5 of trial credit.

Demo Examples

Workflow · Result :--: · :--: Basic — Type text (Add), generate images (Transform), then blend them into one (Combine). · Intermediate — (Add topic → write script → generate speech) + (character description → generate image) → lip-synced video = talking-head avatar. · Advanced — Generate lyrics + song + characters + scenes + storyboard → produce a music video. ·

Every output stays on the canvas as a material: branch from a generated image, feed a transcript into the next step, or swap the model behind one node and keep the rest of the workflow.

How To Start

There are three ways to use TongFlow, all sharing the same open-source core: TongFlow Cloud (the desktop app or app.tongflow.com in a browser), self-host (from source or with Docker), or build with an agent (the tongflow npm package or TongFlow inside Claude Code, any MCP client or dsh).

The TongFlow desktop app is a lightweight (~10 MB) shell around the cloud studio at app.tongflow.com — install it, sign in, and start creating. The cloud studio also runs in any modern browser.

Step 1 — Install the desktop app

Download the installer for your platform, install it, and open it.

• macOS (Universal — Apple Silicon & Intel): TongFlow-mac-universal.dmg
• Windows: TongFlow-win-x64.msi

All builds are on the Releases page.

macOS: the builds are not yet notarized with Apple, so Gatekeeper will block the first launch ("TongFlow is damaged and can't be opened"). After moving the app to Applications, clear the quarantine flag once and it opens normally:

```bash

xattr -cr /Applications/TongFlow.app

`

Download from this page directly — installers passed through chat apps (e.g. WeChat) may be renamed or re-flagged.

Step 2 — Sign in and create

Sign in with Google or WeChat and start creating — the cloud studio manages plugins and execution for you.

Prefer a fully local, account-free TongFlow? That's what self-hosting is for — see Run from source or Run with Docker, then follow Self-host setup. (The desktop app up to v0.1.13 bundled this local runtime; those installers remain on the Releases page.)

Modality-First

A modality is a form of information: text, image, audio, video, 3D, document, URL. TongFlow makes these the building blocks of a workflow. Decide what you want to do with them, then choose the model for the job. Every step is three separate decisions:

· Question · Examples :-- · :-- · :-- Material · What do you have? · A document, a photo, a recording, an idea. Capability · What could it become? · Understand, generate, transform, combine or split. Implementation · What should run it? · A compatible plugin and the model it provides.

• Every result is a new beginning. Materials have their own nodes. An uploaded image and a generated image can both feed the next compatible operation. Branch out from a result instead of starting over.
• Choose a capability, then a model. A workflow records what a step does separately from the plugin that runs it. Switch compatible implementations while keeping the surrounding structure.
• Four operations. Add, transform, combine, split & batch. Understanding, generating and processing fit into the same system; generation is one kind of operation.
• A shared structure for people and agents. The canvas, exporter and agent tools use the same node registry. Build visually or through the tongflow package; inside Claude Code, an MCP client or dsh, an agent creates workflows you can open, edit and run again.
• Open ecosystem. The ABI defines each capability's input/output contract independent of who implements it. Any platform can publish plugins the same way, and the official plugins below cover nearly every node in the matrix.

Read more: Why TongFlow starts with modalities.

Capabilities

Every node below is a capability from the ABI, grouped by the four operations. Materials (text, image, audio, video, 3D, document, URL) are the nodes that connect them.

✅ = available out of the box with an official plugin · ⬜ = node exists in the canvas but has no official plugin yet (planned).

Add

• ✅ Text input: type text and add a text node.
• ✅ Add image: pick a local file and add an image node.
• ✅ Add photo: capture with the device camera and add an image node.
• ✅ Add sketch: draw on the canvas and add an image node.
• ✅ Add audio: pick a local audio file and add an audio node.
• ✅ Record audio: record with the mic and add an audio node.
• ✅ Add video: pick a local video file and add a video node.
• ✅ Record video: record with the camera and add a video node.
• ✅ Add document: pick a local file and add a document node.
• ✅ Add URL: fetch a page from a link and add text, image, audio, or video nodes.
• ✅ Add 3D model: choose a local model file and add a 3D model node.

Transform

Text

• ✅ Generate / rewrite: create or edit copy from a prompt.

Image

• ✅ Image generation: images from text.
• ✅ Image editing: inpaint, edit, or redraw with instructions.
• ✅ Image understanding: captions, Q&A, or descriptions from an image.
• ✅ Image upscaling: enlarge for sharper detail.
• ✅ Pose detection: 308-keypoint whole-body skeleton overlay (body, hands, face).
• ✅ Body-part segmentation: 29-class human parsing overlay.
• ✅ Surface normals: per-pixel normal map — human-centric or full scene.
• ✅ Matting: cut the human or salient foreground out as a transparent PNG.

Video

• ✅ Video generation: video from text.
• ✅ Image-to-video: animate a still into motion.
• ✅ First/last-frame video: two key images to interpolate a clip.
• ✅ Images → video: multi-image reference fusion — several reference images plus text into a new video.
• ✅ Omni-reference video: mix image, video, and audio references (plus text) into one video with native stereo audio.
• ✅ Video understanding: summaries or descriptions from video.
• ✅ Video upscaling: higher-resolution output.
• ✅ Extract first / last frame: grab a frame as an image.
• ✅ Video editing: edit a video from a text instruction.
• ✅ Subtitle removal: clean subtitles from a video.
• ✅ Watermark removal: remove watermarks from a video.

Audio

• ✅ Music generation: music from text, with optional reference-audio conditioning.
• ✅ Audio generation: one prompt → speech, sound effects, and ambience in a single pass, with optional reference voices.
• ✅ Audio understanding: describe a clip (music, speech, or ambient sound) in text.
• ✅ Music repaint: regenerate a chosen time range of a song.
• ✅ Music cover: restyle a song via a caption and/or a reference track.
• ✅ Add track / complete arrangement: generate one new stem over a mix, or fill in missing tracks.
• ✅ Music brief: one-sentence idea → lyrics, style tags, BPM, key, and duration.
• ✅ Speech synthesis: text-to-speech — preset style, voice clone (reference audio), or instruction-driven.
• ✅ Speech recognition: transcribe speech from audio or video.
• ✅ Noise reduction: denoise audio.
• ✅ Speaker diarization: separate audio by speaker.
• ✅ Voice / timbre replacement: re-voice speech or singing in the timbre of a reference clip.
• ✅ Multi-track / vocal-accompaniment separation: isolate vocals, drums, bass, guitar, and 8 more stems.
• ✅ Open-vocabulary sound separation: describe any sound in words ("dog barking") and split the audio into that sound and everything else.

3D, documents & links

• ✅ Image → 3D: single-view 3D model from an image.
• ✅ Video → motion capture: monocular video to skeletal animation (body + fingers + face channels, GLB).
• ✅ Document → text: extract plain text from documents.
• ✅ Link → text: turn page content into text.

Combine

• ✅ Image fusion: blend or edit multiple references into one image.
• ✅ Lip sync: audio + video → video (lip-sync); also audio + image → video and audio + text → video variants.
• ✅ Emotive speech: text + voice reference → speech in that voice, with emotion control.
• ✅ Character swap: video + reference (scene blend / character replacement).
• ✅ Motion transfer: character image + driving video, end-to-end (no pose extraction).
• ✅ Combine text: merge multiple text nodes into one.
• ✅ Concatenate clips: join multiple videos end to end.
• ✅ Mux audio + video: merge into one file.

Split & batch

• ✅ Split by shots: cut a long video into segments by scene.
• ✅ Split video & audio: demux a video into separate video and audio tracks.
• ✅ Extract audio track: pull audio into its own asset.
• ✅ Split long text: break a long passage into chunks.
• ✅ Merge / tidy text blocks: combine segments (use the auto-merge option).
• ✅ Filter or drop clips: drop unwanted clips by rule or selection.
• ✅ Arrange & batch groups: group and arrange text/clip batches for downstream processing.

Official plugins

The official GPU/CPU plugins currently run on Modal — up to $30/month of free GPU compute (H100/A100, etc.). See Self-host setup for the MODAL_TOKEN_* setup. Any other platform can publish its own plugins the same way.

API plugins

First-party providers (a lab's own models):

• tongflow-api-gemini — Google Gemini with a per-node model picker: text, vision, image (Nano Banana / Imagen 4), Veo video, TTS and transcription
• tongflow-api-openai — OpenAI with a per-node model picker: gen_text, image gen/edit/fusion (gpt-image-2), vision, document OCR, Whisper transcription and TTS
• tongflow-api-deepseek — DeepSeek with a per-node model picker (V4.1 Flash / V4 Pro, streaming thinking bubble): gen_text / text tools, plus image understanding on Flash
• tongflow-api-bytedance — ByteDance Volcengine Ark with a per-node model picker: Doubao text & vision, Seedream image gen/edit/fusion, Seedance text/image/audio → video
• tongflow-api-xai — xAI Grok with a per-node model picker: gen_text (Grok 4.x), image understanding, and Grok Imagine text-to-image
• tongflow-api-runway — Runway Dev unified API with a per-node model picker: video (Gen-4.5, Gen-4 Turbo, Aleph edit, Act-Two, Seedance, Veo), image (GPT Image 2, Seedream 5, Gemini image 3) and ElevenLabs TTS
• tongflow-api-metaso — MiniMax H3 video generation hosted by Metaso: text / image → video at ¥0.09/s (768P) or ¥0.15/s (2K), through an OpenAI-compatible Videos API
• tongflow-api-qwen-audio — Alibaba Qwen-Audio-3.1 on Model Studio: TTS-Next renders speech + sound effects + ambience in one pass (also powers voice clone, voice-from-description and emotive speech), TTS-Flash preset voices, and ASR-Flash transcription with timestamps and speaker labels

Router plugins

Aggregators — one key, many third-party models across labs:

• tongflow-router-openrouter — OpenRouter with a per-node model picker: gen_text (free by default, plus GPT-5.5 / Claude / Gemini / Grok / DeepSeek), vision / audio understanding, image gen/edit and transcription
• tongflow-router-cometapi — CometAPI gateway with a per-node model picker that also lists CometAPI's live catalog: gen_text / text tools (GPT-5.5, Claude, Gemini, DeepSeek, Grok, Qwen, Kimi), image / video / audio understanding, image gen/edit/fusion (GPT Image 2, Seedream), text / image(s) → video (Sora 2, Veo 3.1, Seedance, Wan, MiniMax, HappyHorse, Vidu), video edit (Omni), TTS and Whisper transcription
• tongflow-router-toapis — ToAPIs gateway with a per-node model picker: gen_text / text tools (GPT-5.6, Claude, Gemini, DeepSeek, Qwen, GLM, Kimi, MiniMax), image understanding, image gen/edit/fusion (GPT Image 2, Seedream 5, Gemini Image, Flux 2, Grok), text / image(s) / first-last-frame / multimodal refs → video (Sora 2, Veo 3.1, Seedance 2, Kling, MiniMax H3, Wan, HappyHorse, Vidu), video edit (HappyHorse); the picker also lists the live per-key model list
• tongflow-router-apimart — APIMart gateway with a per-node model picker: image gen/edit (Z-Image, Seedream, Nano Banana, GPT-Image), text/image → video (Kling, VEO3, Sora2, Seedance), gen_text (GPT-5, Claude, Gemini), Whisper transcription and TTS
• tongflow-router-replicate — Replicate with a per-node model picker across the catalog: text, vision, image gen/edit/fusion/upscale/matting, text/image → video, transcription, TTS / voice clone, music, and image → 3D (FLUX, Seedream, Veo, Kling, Whisper, Hunyuan3D…)
• tongflow-router-fal — fal.ai with a per-node model picker: image (gen/edit/fusion/upscale/matting/pose/normal/seg), video (text/image → video, first-last frame, talking-head, lip-sync, upscale), audio (transcription, TTS, voice clone, music, source separation) and image → 3D
• tongflow-router-infistar — Infistar (无限星河AI) gateway with a per-node model picker that also lists the key's own model list: gen_text / text tools (GPT-5.6, Claude, Gemini, Qwen, DeepSeek, GLM, Kimi, Doubao, MiniMax, Grok), image understanding, image gen/edit/fusion (GPT Image 2, Qwen-Image, Seedream, Wan), text / image → video (Wan 3.0, Seedance, MiniMax H3) and transcription

GPU/CPU plugins

• tongflow-modal-ffmpeg — transcoding, muxing, media pipelines
• tongflow-modal-pyscenedetect — shot-boundary detection for splitting clips
• tongflow-modal-z-image — Z-Image text-to-image
• tongflow-modal-ernie-image — ERNIE Image text-to-image (alternative)
• tongflow-modal-krea2 — Krea 2 Turbo text-to-image (open-weights 12B, 8-step, up to 2K)
• tongflow-modal-moss-image — MOSS-Image (OpenMOSS) 5B text-to-image: dense Chinese/English text, prompts up to 4K tokens, JSON layout prompts that place text and objects by box (alternative)
• tongflow-modal-flux2-klein9b — FLUX.2 Klein 9B multi-reference fusion / image editing
• tongflow-modal-qwen-image-edit — Qwen-Image 2.1: text-to-image, instruction editing, multi-image fusion (up to 10 references) and transparent background removal (headless ComfyUI, int8; weights Qwen Research Licence, non-commercial)
• tongflow-modal-boogu — Boogu-Image-0.1 (fp8) text-to-image (dense bilingual text) & single-reference image editing
• tongflow-modal-infinitetalk — InfiniteTalk audio-driven lip-sync (audio + image / video → talking-head video)
• tongflow-modal-longcat-avatar — LongCat-Video-Avatar 1.5 (Meituan) audio-driven talking video: portrait + speech → lip-synced video, Whisper audio encoder, 8-step distilled, up to 60 s (alternative)
• tongflow-modal-wan-animate — Wan-Animate-2 end-to-end character animation (character image + driving video, no pose extraction)
• tongflow-modal-scail2 — SCAIL-2 controlled character animation (image + driving video), and character swap
• tongflow-modal-minimax-h3 — MiniMax-H3 33B video generation with native stereo audio (text / first- & last-frame / multi-image / omni-reference)
• tongflow-modal-bernini — Bernini-R 1.3B unified video renderer (text/image → image/video, video editing, subtitle / watermark removal)
• tongflow-modal-sam3 — SAM 3 / SAM 3.1 text-guided matting: cut every instance of a described concept out of an image (transparent PNG) or track it through a video (green-screen matte)
• tongflow-modal-triposplat — TripoSplat single image → 3D Gaussian splat
• tongflow-modal-sam-3d-objects — SAM 3D Objects single image → 3D Gaussian splat of the foreground object (auto mask, robust to occlusion/clutter; alternative)
• tongflow-modal-sam-3d-body — SAM 3D Body single image → full-body 3D human mesh GLB (multi-person, MHR rig; alternative), and video motion capture (per-frame MHR regression → animated character GLB; alternative)
• tongflow-modal-sapiens2 — Sapiens2 (Meta) human suite: pose detection, body-part segmentation, surface normals, human matting, image → 3D point cloud, and video motion capture (geometric engine: keypoints + pointmap → animated MHR character GLB)
• tongflow-modal-sensenova-vision — SenseNova-Vision (SenseTime) unified vision model: image understanding / visual QA, detection & OCR structured text, full-scene surface normals, salient-object matting, and human pose overlay (alternative)
• tongflow-modal-seedvr2 — SeedVR2 image / video super-resolution
• tongflow-modal-gemma4 — Gemma-4 multimodal text (image / video understanding)
• tongflow-modal-qwen38 — Qwen3.8-27B multimodal text (text generation, image / video understanding; alternative)
• tongflow-modal-qwen3asr — Qwen3 speech recognition
• tongflow-modal-qwen3tts — Qwen3 text-to-speech
• tongflow-modal-indextts2 — IndexTTS-2.5 emotive text-to-speech: zero-shot voice cloning (alternative) and emotion-controlled speech from a voice reference
• tongflow-modal-seed-vc — Seed-VC