Research

Dense layout maps are an expensive way to control a generative model: every pixel needs a label before you can ask for anything. TerraDiT replaces them with sparse point queries, each pairing a 2D image coordinate with a free-form OpenStreetMap text tag, which makes spatial control both semantically richer and far cheaper to annotate.

I contributed to the Adaptive Local Attention block, whose MetaRBF module predicts per-concept spatial extents that parameterise 2D radial basis kernels, modulating cross-attention between latent image tokens and point prompts. That change alone improved FID from 31.15 to 28.34 over vanilla cross-attention.

The full system was trained across a three-stage, 1M-step pipeline over a 2M-image OSM-augmented corpus on four NVIDIA H100s, with DINOv3 representation alignment and RANGE geolocation conditioning, reaching 12.01 FID against 25.93 for the prior state of the art while keeping the lowest single-image generation latency among the models evaluated.

Cross-view pose estimation normally handles one ground image at a time, which throws away information: autonomous vehicle sequences, street mapping services and crowd-sourced collections routinely capture several overlapping views of the same place. This work formulates the multi-ground-view version, jointly recovering the 3-DoF pose (yaw, ground-plane translation and scale) of a set of nearby street-level images inside one aerial reference frame.

We released the first public dataset for it: over 500,000 Mapillary ground images registered against more than 100,000 NAIP aerial maps, spanning urban, suburban and rural regions under strictly disjoint geographic splits.

The model extends the VGGT multi-view reconstruction transformer with a dedicated aerial branch holding independent weights and its own global coordinate space, freezing most ground-branch weights so the pretrained geometry survives a much sparser pose-only supervision signal.

The scaling obstacle is attention: VGGT’s global attention layers cost memory quadratic in the number of input images. We replace each one with a cross-aerial attention layer followed by a sparse global attention layer, reusing the ground-to-aerial attention map to bin ground patches and select which ones each token may attend to. That holds memory at roughly the cost of frame self-attention rather than letting it grow with image count. On the non-gridded dataset the model reached 7.01 m mean localisation error and 4.46° mean orientation error.

Given a video as a left-view, estimate depth, convert it to disparity, and warp the latents to synthesise a temporally consistent right-view: no training, on the grounds that the video domain is too compute-hungry to do otherwise. The pipeline integrates Wan2.1 for generation and Video-Depth-Anything for depth, with trilinear interpolation of the depth map before the shift.

The interesting result was a negative one. Applying the stereo warp on a schedule of every ten timesteps (the approach that works for StableDiffusion on still images) produced a right view nearly identical to the left, with no disparity shift at all. The reason is the scheduler: Wan2.1 uses UniPC/DPM, high-order solvers for diffusion ODEs, where StableDiffusion’s is first-order and Markovian. The high-order solver carries momentum from the left-view latents across steps, so an intermittent warp gets averaged away before it can take effect. Shifting at every timestep starting from step zero destroys that momentum and the warp survives.

This is honest work-in-progress rather than a finished result. The current output still merges left and right views into a single frame, most likely because timestep and prompt are injected once across the batch rather than per view, and a DDPM scheduler is not currently exposed through Wan. Both are the immediate next steps.

Public health assessment of a neighbourhood traditionally means sending officials out to look at it. Street-view imagery plus a vision-language model turns that into a query. The system pairs a Google Maps interface with VLM inference over street-view panoramas, scoring census tracts for greenness, walkability, inhabitance, and the presence of speed limit and stop signs.

The architecture changed mid-project, which is the part worth talking about. The first design ran InternVL3-38B offline to generate dense captions for every location, then served fast answers at runtime with Phi-4-mini over those captions: good for latency and a natural fit for later RAG, but expensive to precompute and stale the moment the imagery updates. The second design runs a much smaller InternVL-2B directly on images at request time, trading some caption richness for the ability to answer about anywhere immediately. Benchmarks across roughly 1,000 and 8,000 locations landed near 0.12 and 0.14 seconds per location respectively.

Evaluation used an LVLM-as-a-judge setup, comparing the production VLM’s scores against a larger judge model across each metric. Agreement was strong on greenness and weaker on the binary sign-detection tasks, which is roughly what you would expect: a small model can tell you a street is leafy more reliably than it can find a stop sign in a panorama.