Live UE Scene Bridge

Problem

Cinev Studio had character and camera workflows, but controls for lighting, asset placement, weather, and time of day were still limited. I built a PoC for controlling these through natural language and passing the scene into image and video generation services.

Solution

I connected natural-language scene control and external generative services using the plugin’s existing Python–C++ JSON bridge. Cesium sets the location and UDS controls time and weather; an Unreal viewport capture then feeds into image and video generation. My focus was defining processing states across C++, Python, and TypeScript and communicating progress to users.

3-Layer Architecture

Web UI / CLI → HTTP:34115 → FastAPI + NLP (Claude/Gemini)
                                    ↓
              ┌──────────────┬──────────────┬──────────────┐
              │ UE C++ Plugin│ Nano Banana  │ Veo-3        │
              │ TCP:55557    │ Image Gen    │ Video Gen    │
              └──────────────┴──────────────┴──────────────┘

Core — Cesium + UDS Integration

Cesium and Ultra Dynamic Sky set the location, time, weather, and mood; Nano Banana turns the scene into a concept image. I focused on defining the status passed between services and showing users which step was currently running.

Style transformation result — "Give me a post-apocalyptic mood"

i2i pipeline in Web UI

Roblox Integration — Business Demo

I imported FBX assets through the Roblox API into UE, placed characters on generated backgrounds, and produced images and videos with Nano Banana and Veo-3. This internal demo showed the CEO and planning team how the workflow could be used.

Roblox character placed on generated background

Processing Status Feedback

A request to give the current Unreal scene the atmosphere of another era passes through capture, transfer, generation, and waiting for a response. I defined status information across C++, Python, and the TypeScript UI so users could understand what they were waiting for.

Refining LLM Inputs and Calls

After the workflow was in place, I refined LLM inputs and call structure. I centralized LLM processing in Python to reduce duplicate calls and simplified parameter-validation instructions, decision branches, examples, and injected context. For image transformation, I compared prompt length with clarity while considering instructions needed to preserve composition and control transformation strength.

For CJK asset-name translation, I also applied batching, deduplication of shared names, and concise input/output formatting.

Deciding what to adopt next

Lighting and asset-placement controls were planned as follow-up features to fill gaps in the product. Character–environment interaction was also a later task. In a separate snow and puddle interaction experiment, results varied between runs; I deferred adoption because consistent behavior could not be secured within the remaining schedule.

Result