Joseph Zhu

I build AI agents and robots. Right now I’m a Member of Technical Staff at Jump Trading in Singapore, where I was the 3rd AI hire in the core strategy pod and run all the AI agent stuff for derivative arbitrage. Before that I was a founding researcher at AGI Inc, training computer-use agents (we got #1 on OS-World). And before that, I spent 3+ years as cofounder and the only software engineer at Tepan, writing every line of software for a stovetop bimanual robot that autonomously takes care of everything from ingredient prepping, cooking, seasoning, to self-cleaning.

Prior to founding Tepan, I quit on PhD from Berkeley AI(BAIR) and briefly joined Stanford. I did some research under the supervision of Jiajun Wu in Stanford Vision and Learning Lab(directed by Fei-Fei Li), and with Sanmi Koyejo in SNAP.

My tech stack

Click each one to expand:

• AI Agents
  This is what I spend most of my days on now. At Jump, I maintain the internal context for Jump’s own version of Claude and Codex that all ~300 people in the trading pods use. The fun part is that “is the context good?” is pretty vague, so I designed benchmarks on how well the AI uses our internal tools, and then optimized the context against those benchmarks using ralph loops. I also built agents that do real work: a pipeline that rolls out arb strategies for newly issued Chinese ETFs (the AI handles sim, alpha setup, strategy configs, and execution params), a post-trade analysis pipeline that computes perfect PnL and breaks down why we missed PnL (executor down, alpha timed out, risk limit hit, exchange/broker rejects, etc.) and then sends agents off to investigate each one with a pre-written handbook, and agents sitting in Slack alert channels that notice execution engine failures, open a PR to fix them, and ask a human to review. I still do things by hand sometimes too, e.g. I set up a strategy arbing Korea ETFs against index futures that makes around $5M a year at a Sharpe of 8.
  
  Before that at AGI Inc, I was on the other side, training the agents themselves. I built the REAL benchmark, which clones real-life websites (shopping, flight/hotel booking, restaurant booking, calling an Uber…) with state snapshots so we could generate thousands of computer-use tasks with programmable success criteria. I also wrote the VLM agent harness from scratch (clicking, dragging, scrolling, memory, etc).
  
• RL & LLM Post-training
  At AGI Inc I trained computer-use VLM agents with on-policy distillation and RL. I modified VERL’s PPO training loop to support multi-turn agent interactions with turn-wise reward assignment using GAE, and designed benchmarks specifically for clicking coordinate accuracy, then pushed that up with GRPO. On the SFT side I managed dozens of data collectors producing tens of thousands of computer-use trajectories, and trained Qwen3-235B and Qwen3.5-35B with unsloth. RL rollouts are expensive, so I wrote custom rollout infra on Kubernetes + Ray that runs on spot VMs, detects preempted ones by their dead heartbeats, and spawns new Ray workers to replace them. With 3 teammates we got #1 on OS-World (it even made the NYTimes), and the models we shipped ran on multiple OSes for a B2B contract with Lenovo, including a CES demo.
  
• Neural Networks
  I get down to pretty basic levels of this. Back in undergrad when working on audio there weren’t many standardized architectures, so I usually wrote my own neural networks(e.g. convolutional networks to generate attention masks, implementing transformers using cupy, using Fourier transforms to implement convolutions, recursive RNNs, inference for WaveNets). At Tepan, I made a custom promptable segmentation model for food ingredients based on ViT Adapter in Detectron2, and training 3D networks that use the heat diffusion process to propagate features on meshes.
  
• Information theory/Generative Models
  I’m pretty familiar with the theory in general, getting into the nitty gritty details of the whole derivation for diffusion models, flow matching, consistency models, and VAE, by re-deriving everything from scratch. I wrote a paper called HiFA for my side project that broke down the score distillation process of a diffusion model into a pseudo ground-truth generating process. This paper has been cited 100+ times within its first year.
  
  I also love reading general probability theory related papers. My favorite one is GAIL, but I haven’t 100% understood the proof yet.
  
  That said, I am ashamed to admit that “A Mathematical Theory of Communication” has been on my reading list for 5 years now and I still haven’t read it.
  
• AI Infrastructure
  I was forced to figure this out since I was the only software engineer at Tepan. I managed all the clusters we had on GCP and burned through $250k of compute credits in a year. That includes fine-tuning a DINOv2 using around 64 L4 GPUs for 10 days, while writing custom fine tuning code. Other than that, I delved deep into FSDP model sharding and figured out nitty gritty details like “why can’t BatchNorm be FSDP’ed?”, checkpointing shards vs whole models, etc, in order to train a Maskformer more efficiently with a bigger batch size. I also wrote a torchserve wrapper for this custom promptable maskformer for efficient serving. I also wrote some TensorRT back in its early days, but it didnt turn out too well because back then there was no GPT and very little support for pytorch ONNX.
  
• Hardware Supply Chain and Data Labelling
  Supply chain is quite important when running a hardware startup, especially if in China. That requires playing tug-of-war with suppliers, getting information and product samples, cross checking the prices, picking the right one that’s both cheap enough and good enough. I’ve built quite some familiarity with the supply chain for chips, basic brands, and a good sense of the rough range of prices for different products.
  
  I also worked extensively with data labellers, from picking a data labelling agency to writing data manuals, and developing ways to monitor the efficiency of data labellers. I curated a dataset of 5M food images of food ingredients, with 1M of them filtered and coarsely annotated, and 50k of them annotated with segmentation masks, all with a budget of 10k USD.
  
• 3D modelling and graphics
  I am familiar with NeRFs and differentiable rendering pipelines, in particular NVDiffRast and Pytorch3D. I’ve invented custom losses to make NeRFs sharper by regularizing the distribution of density functions along a ray, and impelemented GGX microfacet BRDFs for NeRFs with point lights(althought that didn’t work very well because it needs an environmental map, and that was computationally infeasible). Other than that, I played quite a lot with Poisson reconstruction, and explored using Poisson Recon to obtain a mesh that comes with albedo, roughness, and metallic textures, before throwing that into NVDiffRast for further optimization. I also learned GLFW to render a robot along with point clouds into a VR headset with high frame rates for teleoperation.
• Depth Cameras, Camera Calibration, and 3D Reconstruction
  I have done a lot of stuff for depth maps and depth cameras. I’ve worked with structured-light and active stereo cameras coming straight from suppliers like Orbbec and Deptrum, often with lacking functionalities that I have to implement myself, such as efficiently aligning depth streams and RGB streams of different resolutions using relative lens poses(they are actually misaligned since their camera holes are not the same, one is IR but the other is RGB), obtaining colored point clouds from those(surprisingly, some suppliers don’t even have that function built into their cameras SDKs…). I wrote a python binding for Deptrum SDK because they didn’t have one.
  
  I also have a deep understanding of camera parameters and the calibration process. I had two depth cameras that I had intrinsics of but I needed to figure out their exact relative poses in order to merge their point clouds. I bought an AprilGrid tab from Taobao and realized that only Kalibr can could give me the corners accurately, but Kalibr doesnt let me directly calibrate the extrinsics given the intrinsics, so I had to handwrite Direct Linear Transform(DLT) + some ICP stuff + some RANSAC plane detection stuff to get the job done. I’m familiar with hand-eye calibration methods such as Tsai and Lenz as well.
  
• Embedded Robotics Software
  The entire Tepan robot was built from scratch, the maximum level of packaged-ness we got was a custom servo for some joints, but we had to take apart even that. Therefore, I wrote PID controllers for quite a few different types of motors for STM and Arduino boards(back when I was building in my garage). Theres a lot of low-level engineering involved in this, here are a few examples. Back in 2022 I modified the arduino AccelStepper library to compute stepper pulse intervals in real time based on high-frequency waypoint inputs on a mere Arduino nano(where I won’t be able to use square root/division if I don’t want to miss a pulse) by extending this paper. Later on at Tepan (this is probably the biggest engineering NIGHTMARE I’ve ever faced!), we had a custom board with which tried to run FOC on a stepper motor. There was 3 months of struggling here, where the major problems are (i) the UART communication would randomly break down, in the end there was an issue with a cyclic buffer. (ii) there is a thing called “zero electric angle” in FOC, and that was changing all the time even when it was supposed to be consistent, and it turned out that the position sensor wasn’t giving absolute but incremental angles from when it was powered on. (iii) The motor just gets stuck randomly when there was a little bit of resistance. This took the longest, initially we thought there were issues with the PWM or some backEMF stuff causing the motor to produce enough force, but in the end after countless testing it turned out that the position sensor magnet wasn’t big enough and it was causing small errors (like 0.5 degrees) in position readings, and it just so happened that stepper motor FOC was REALLY sensitive to those errors.
  
• Robotics Fundamentals
  Tepan has a really weirdly shaped robot, with two 5 DOF arms, where one branches off from the other. I wrote custom forward kinematic with Jacobian computation using sympy, and wrote a custom null space inverse kinematic solvers(there was a teleop requirement so that I had to use euler angles to parameterize the robot, so the angular component of the Jacobian was for euler angles instead of the usual angular velocity, I had to do some tricky derivations to avoid gimbal lock). I also implemented gravity compensation from scratch.
  
  I have designed a lot of adaptive motions for the robot. For example, there would be a pre-defined chopping motion for round things that need to be sliced, but adapted onto food items of different positions, sizes, and distortions. To this end I also did a project called DenseMatcher, which finds correspondences between objects in order to map one’s contact points to another.
  
• VR/Spatial Computing
  At Tepan I wrote a program that uses two oculus VR controllers to program a robot. The key challenge is mapping the controller poses to robot poses, which was kinematically constrained, and also figuring out what’s the most intuitive way to couple the two. As an intuitive example, if the user wants to pause the robot for a bit, they wouldn’t want the robot to suddenly snap when it resumes if the controller moved in between. Another example would be, should I model the controller as rotating around an origin centered at itself, or rotating around an origin centered at some external coordinate frame, and what about the robot? If users want the robot to rotate at half of the speed of the controller, how would I model this division in rotation? A lot of fun ways to manipulate SE(3) matrices under different coordinate frames. Also some tricky transforms between different coordinate conventions(Bullet vs OpenGL, COM frame vs URDF link frame, etc).
  
• Signal Processing
  I worked on computational speech for quite some time and have a lot of in depth understanding of signal processing math, including convolutions Short Time Fourier Transforms, Wavelets, time varying IIR filters, as well as image signal processing stuff. Aside from published research, I worked on an AI audio codec in Tencent where I turned Linear Predictive Coding(a type of residual coding for compressing speech) into a differentiable layer in the neural network. Back at Tepan I also worked on functional maps, which is basically signal processing for meshes. It turns out that you can apply something similar to Fourier Transform to meshes, call Laplace-Beltrami eigenfunctions.
  
• Physics of Electricity & Mechanics
  I did my undergrad in EE, so I understand how analog & digital circuits work on a fundamental level, and how radio/electromagnetic waves work. (Quick question: what is the difference between radiowave and laser in terms of electromagnetic field?) Thanks to the coaching of Prof. Jont Allen, I got to learn about dynamical system, and how neurons work on the electrical level. I also dabbled in fluid mechanics(Navier-Stokes for liquid, Webster Horn for gas and sound), and structural mechanics, although I’m not an expert.