Kulbir Singh Ahluwalia with a mobile manipulator in a field

Hello, I'm Kulbir Singh Ahluwalia

I’m Kulbir Singh Ahluwalia, a Ph.D. student in Computer Science at the University of Illinois Urbana-Champaign (UIUC), advised by Prof. Girish Chowdhary and Prof. Julia Hockenmaier. My research is on language-grounded waypoint generation and navigation for agricultural and outdoor robots (WaypointGen), and on running these models efficiently on edge AI hardware. From June to August 2026 I was a Compute Chipset Product Management Intern at Qualcomm in San Diego, and in Summer 2025 I was an AI Intern at EarthSense, Inc., where I contributed to a natural language-conditioned waypoint generation pipeline for outdoor robots. In Fall 2026 I am a Teaching Assistant for CS 440/ECE 448: Artificial Intelligence with Prof. Margaret Fleck, and in Fall 2025 I co-developed the CS 498 GC Mobile Robotics course with Prof. Chowdhary. Before UIUC, I earned a Master of Engineering in Robotics from the University of Maryland, College Park. My long-term goal is scaling up Physical AI for advancing humanity.

GitHub X LinkedIn Scholar CV Resume ORCID

News

[Aug 2026] Joined the course staff of CS 440/ECE 448: Artificial Intelligence as a Teaching Assistant with Prof. Margaret Fleck.

[Jun 2026] Joined Qualcomm (San Diego, CA) as a Compute Chipset Product Management Intern, summer 2026.

[Feb 2026] Our poster WaypointGen: Learning Grounded Waypoint based Navigation for Mobile Manipulators was accepted at the CSL Student Conference (CSL SC 2026) at UIUC. [poster PDF]

[Oct 2025] Co-developed SLAM-ing Mars, a mobile manipulator exploration and SLAM assignment for CS498GC: Mobile Robotics (Fall 2025), with Prof. Girish Chowdhary.

[May 2025] Our paper Active Semantic Mapping with Mobile Manipulator in Horticultural Environments was presented at ICRA 2025. IEEE · arXiv:2412.10515 · project page

[May 2025] Joined Earthsense Inc. (Urbana, IL) as an AI Intern, summer 2025.

[Fall 2023] Presented the poster Plant Placement using Natural Language Grounding at the AIFARMS Annual Conference. [poster PDF]

[Jun 2022] Started my Ph.D. in Computer Science at UIUC, advised by Prof. Girish Chowdhary (DASLAB) and Prof. Julia Hockenmaier (Hockenmaier Lab).

Publications

View: show all / show selected / show by topic

Research Topics: Mobile Robotics / Computer Vision / Deep Learning / Machine Learning / NLP / Path Planning / Decision Making / 3D Vision

TaP: Trajectory as Policies for Spatial Reasoning using Hierarchical Abstractions & Grounded Constraint Phrases for Mobile Manipulator Navigation

Kulbir Ahluwalia, Chahit Jain, Gurmanat Singh Kahlon, Jose Roberto Cuaran Valenzuela, Svetlana Lazebnik, Girish Chowdhary, Julia Hockenmaier

IEEE Transactions on Robotics (T-RO), under review Journal Paper (Under Review)

Map-free, language-conditioned planner for agricultural mobile manipulators that writes constraint-aware waypoints in 2D bird’s-eye-view image coordinates and back-projects them to 3D with depth. On a 200-pair citrus benchmark: 99.0% structural pass and 37.5% geometric fidelity (waypoints inside the ground-truth region).

WaypointGen: Compositional Trajectory Generation by Grounding Hierarchical Language Abstractions and Constraint Phrases for Outdoor Navigation

Kulbir Ahluwalia, et al.

CoRL 2026 Workshop on Compositional Generalization for Real-World Robot Learning Workshop Paper (Under Review, Non-Archival)

Modular pipeline for language-commanded navigation of agricultural mobile manipulators: a local bird’s-eye view from RGB-D, open-vocabulary target and avoidance masks, code-based command checks, and a vision-language model that picks among MPPI trajectory candidates. Offline studies on 349 palm-plantation targets score the candidates, not the model’s choice.

WaypointGen: Learning Grounded Waypoint based Navigation for Mobile Manipulators

Kulbir Ahluwalia*, Chahit Jain*, Jose Roberto Cuaran Valenzuela, Shreya Gummadi, Michael McGuire, Arun Sivakumar, Julia Hockenmaier, Girish Chowdhary

Coordinated Science Laboratory Student Conference (CSL SC 2026) Conference Poster

We present WaypointGen, a map-free, language-conditioned 2D path planner that grounds natural language navigation commands to bird’s-eye-view (BEV) waypoints. It parses region-of-interest constraints from NL commands via open-vocabulary grounding (DINO-X detection, SAM3 instance masks, and Describe Anything Model captions), then uses a Qwen3-MoE-30B-A3B VLM to select target ROIs and choose among Model Predictive Path Integral (MPPI) trajectory candidates in BEV space, with a Qwen3-235B VLM as an LLM judge for weak supervision. A semi-supervised labeling pipeline seeded with a small set of human annotations and in-context learning generates the dataset across agricultural environments (palm, solar, greenhouse, corn/soybean, citrus, and TreeScope).

PDF

Active Semantic Mapping with Mobile Manipulator in Horticultural Environments

Jose Roberto Cuaran Valenzuela, Kulbir Ahluwalia, Kendall Koe, Naveen Kumar Uppalapati, Girish Chowdhary

International Conference on Robotics and Automation (ICRA 2025) Conference paper

We introduce an efficient active semantic mapping approach for horticultural robotics, using a mobile manipulator with an RGB-D camera. Probabilistic semantic octomaps are used to detect target regions of interest such as fruits, generate candidate viewpoints, and compute information gain for next-best-view planning. An efficient ray-casting strategy and a novel information gain function accounting for semantics and occlusions is introduced for efficient target-focused map exploration.

Project Page Paper arXiv Code

Plant Placement using Natural Language Grounding

Kulbir Ahluwalia, Michael Vilsoet, Rajarshi Haldar, Peixin Chang, Julia Hockenmaier, Girish Chowdhary

AIFARMS Conference (Fall 2023) Conference Poster

This poster presents a text-enabled FarmBot system that enables users to control a robotic gardening FarmBot system via natural language. Using a custom Python wrapper built on the FarmBot REST API, natural language commands are grounded using real-time robot state and translated into executable code with a fine-tuned CodeT5 model. The system generates valid plant placement configurations that satisfy natural language defined spatial constraints.

Poster PDF

DeepPaSTL: Spatio-Temporal Deep Learning Methods for Predicting Long-Term Pasture Terrains Using Synthetic Datasets

Murtaza Rangwala, Jun Liu, Kulbir Ahluwalia, Shayan Ghajar, Harnaik Singh Dhami, Benjamin Tracy, Pratap Tokekar, Ryan Williams

Published in Agronomy 2021 (Special Issue AI and Agricultural Robots) Journal Paper

DeepPaSTL aims to accurately forecast long-term pasture growth, tackling the challenge of estimating pasture biomass without relying on extensive site-specific data or frequent field measurements. This approach enables predicting pasture evolution without monitoring fields regularly, using past observed pasture heights as input. DeepPaSTL introduces a bi-directional ConvLSTM encoder–decoder to learn the spatio-temporal pasture growth dynamics purely from spatial height measurements.

Paper PDF

Intermittent Deployment for Large-Scale Multi-Robot Forage Perception: Data Synthesis, Prediction, and Planning

Jun Liu, Murtaza Rangwala, Kulbir Ahluwalia, Shayan Ghajar, Harnaik Singh Dhami, Pratap Tokekar, Benjamin Tracy, Ryan Williams

IEEE Transactions on Automation Science and Engineering, vol. 21, no. 1, pp. 27-47, 2024 Journal Paper

Targets large-scale pasture monitoring for precision agriculture, deploying a team of robots to track grassland growth for optimal rotational grazing and land productivity, addressing the lack of timely growth data in current practice. Proposes an integrated pipeline combining synthetic data generation, deep neural network-based spatiotemporal prediction, and an intermittent multi-robot deployment strategy to periodically survey evolving pastureland at low cost.

IEEE PDF arXiv

Active Semantic Mapping with Mobile Manipulator in Horticultural Environments

Jose Roberto Cuaran Valenzuela, Kulbir Ahluwalia, Kendall Koe, Girish Chowdhary

ICRA@40, 2024 Abstract

An abstract accepted at the 40th Anniversary of the IEEE Conference on Robotics and Automation (ICRA@40), 2024.

Smartphone Optical Sensors

Simarjeet S Saini, Aneesh Sridhar, Kulbir Ahluwalia

Optics and Photonics News, 2019 Featured Article

An article featuring the multispectral Fundus Eye camera prototype, as presented in Optics and Photonics News.

Paper Article PDF

Reinforcement Learning Integrated with Supervised Learning for Training of Near Infrared Spectrum Data for Non-Destructive Testing of Fruits

Yuqi Li, Kulbir Ahluwalia, Simarjeet S Saini

Sensing for Agriculture and Food Quality and Safety XII, 2020 Conference Presentation

A conference presentation on combining reinforcement and supervised learning for non-destructive testing of fruits using near infrared spectrum data.

Paper

Teaching

Teaching Assistant - CS 440/ECE 448: Artificial Intelligence

University of Illinois, Aug 2026 - Present

Instructor: Prof. Margaret Fleck
• Grading Moodle quizzes and testing the machine-problem autograders.

Fall 2026 Course Page

Teaching Assistant - CS498GC: Mobile Robotics for Computer Scientists

University of Illinois, Aug 2025 - Dec 2025

Co-developed with Prof. Girish Chowdhary
• Co-developed the course curriculum on mobile robotics, ROS2, sensor fusion, and SLAM algorithms.
• Managed coding exercises and problem sets on Extended Kalman Filtering and odometry implementation.
• Held office hours and helped students with ROS2 development and debugging.
• Built and maintained the course website; maintained the autograding infrastructure on Gradescope.
• Special Topic Fall 2025: SLAM-ing on Mars

Fall 2025 Course Website Instructor

Teaching Assistant - CS444: Deep Learning for Computer Vision

University of Illinois, Jan 2024 - May 2024, Jan 2025 - May 2025, Jan 2026 - May 2026

Instructor: Dr. Svetlana Lazebnik
• Updated and verified starter code for assignments, and answered student questions during office hours and through Campuswire.
• Assessed student submissions via SpeedGrader on Canvas, and designed multimodal quiz questions, including single-choice, multiple-choice, and matching formats.

Spring 2024 Spring 2025 Spring 2026

Teaching Assistant - CS519: Scientific Visualization

University of Illinois, May 2025 - Aug 2025

Instructor: Dr. Eric Shaffer
• Created multimodal exam questions with integrated visualizations using Python and matplotlib for assessing student understanding of scientific visualization concepts.
• Assisted students with implementation of advanced visualization algorithms including ray marching, transfer functions, and interactive widget development.

Summer 2025 Course Page Instructor

Work Experience

Product Management Intern - Qualcomm Technologies, Inc. (Compute Chipset)

San Diego, CA, USA | June 8 - Aug 21, 2026 (on-site)

Team: Compute Chipset Product Management (Snapdragon X Elite / X2 Elite AI-PC platforms)
• Conducted market and competitive analysis of on-device AI, generative AI, and vision-language model (VLM) features for AI-PC platforms.
• Synthesized feedback into product requirements and roadmap inputs.
• Evaluated VLM workloads on Snapdragon silicon for power, thermal, and latency on the Hexagon NPU.
• Worked with engineering teams on AIMET and AI Hub Models, and prepared internal briefings on edge AI trends.

Company Website

AI Intern - Earthsense Inc.

Urbana, IL, USA | May 2025 - Aug 2025

Supervisor: Michael McGuire, Lead Computer Vision Engineer
• Contributed to developing a natural language-conditioned waypoint generation pipeline for agricultural robot navigation.
• Created an automatic labeling pipeline for large outdoor robot navigation datasets using Grounded SAM2, streamlining data processing.
• Deployed and integrated open-source Visual Language Models (Molmo-7B-D, Gemma-3-27B, Qwen-2.5-VL-72B, Qwen3-30B-A3B, Llama4-Scout, Spatial-VLM; 3B-72B params) for robot reasoning in image space and open-world natural language instruction conditioned question answering for 4 wheeled skid steer outdoor robots.

Company Website

Projects

View: show all / show selected / show by topic

Topics: Mobile Robotics / Computer Vision / Deep Learning / Machine Learning / NLP / Path Planning / 3D Vision

Enhancing Stereo Depth Maps through RGBD-Conditioned Generative Models

Aug-Dec 2024

Fine-tuned Stable Diffusion V2 for RGB+D-conditioned monocular depth estimation. Achieves up to 46% Abs Rel improvement over Marigold and Depth-Anything-V2 on IRS, VKITTI, and NYUv2 benchmarks. (CS598SHW / 3D vision project)

Contributions to Active Semantic Mapping (ICRA 2025)

Jan-May 2024

Co-authored the ICRA 2025 paper with J. Cuaran (first author), K. Koe, N. Uppalapati and G. Chowdhary: probabilistic semantic octomaps for detecting target regions, generating candidate viewpoints, and computing information gain for next-best-view planning on a mobile manipulator.

Turning Zero Shot into Few Shot via Self-prompting

Aug-Dec 2023

Explored self-prompting strategies to convert zero-shot language model capabilities into effective few-shot performance without manual prompt engineering.

Implemented Linear, Logistic, Polynomial regression, SVM, Convolutional, and Attention based models

Apr-May 2023

Built a comprehensive suite of classical and deep learning models from scratch, including SVMs, CNNs, and Attention mechanisms for classification and regression tasks (CS446).

VAE and GAN implementation

March 2023

Implemented VAE & GAN for a digit generation task.

ResNet implementation

Feb 2023

Implemented ResNet for a classification task on the MNIST dataset.

Neural transition-based dependency parser

Dec 2022

Built a neural transition-based dependency parser using feed-forward networks for syntactic analysis of natural language sentences.

Machine translation with RNN and Transformers

Nov 2022

Implemented sequence-to-sequence machine translation using both RNN encoder-decoders and Transformer architectures.

Text classification with CNN and RNN

Oct 2022

Compared CNN and RNN architectures for text classification, exploring the trade-offs between local feature extraction and sequential modeling.

SLAM from 2D LiDAR data using split and merge line extraction algorithm

Dec 2021

Implemented simultaneous localization and mapping from 2D LiDAR scans using split-and-merge line extraction for feature-based SLAM.

State estimation using Extended Kalman Filter for GPS+Encoder sensor fusion

Nov 2021

Fused GPS and wheel encoder data using an Extended Kalman Filter for robust state estimation on a mobile robot.

Field-robot trajectory reconstruction from RTK-GPS, IMU, and encoders (ROS)

Sep 2021

Processed and visualized multi-sensor trajectory data from RTK-GPS, IMU, and encoders using ROS for field robot navigation.

Autonomous Vaccine Delivery Robot

May 2021

Designed an autonomous robot capable of navigating and localizing itself in a test arena using QR codes and arrows. It uses a RGB camera, IMU, optical encoders, and an ultrasonic sensor to detect, retrieve and transport user-specified blocks. Featured video, Robot videos, Featured post

Webpage

Image segmentation using superpixels

Dec 2020

Built a segmentation pipeline using SLIC superpixels with VGG16 feature extraction for superpixel classification.

Webpage

Persistent-Monitoring using Multi-Robot (UAV-UGV) Coordination

Dec 2020

Coordinated UAV-UGV teams for persistent monitoring of large-scale environments with optimized deployment schedules.

Webpage

Optimized a GestureGAN for resource constrained settings

Dec 2020

Used MobileNet to optimize cross-view image generation with a 5.7X reduction in parameters.

Webpage

Self-adjusting roadmaps

May 2020

Navigation in unknown environments using LD-PRM.

Webpage

Estimated the motion of a car using Visual Odometry

May 2020

Estimated camera ego-motion from sequential video frames using feature matching and essential matrix decomposition for visual odometry.

Webpage

Color segmentation using Gaussian mixture models & Expectation maximization

April 2020

Segmented colored buoys in underwater imagery using GMMs trained via Expectation Maximization for robust color-based detection.

Webpage

Image Classification using CNN

May 2020

Built a CNN for binary image classification (cat vs dog) with data augmentation and transfer learning.

Webpage

AR-Tag detection

March 2020

Superimposed an image and virtual cube on an AR tag.

Webpage

Tracked moving objects using Lucas-Kanade Tracker

April 2020

Implemented the Lucas-Kanade optical flow tracker for real-time object tracking across video frames.

Webpage

Baxter transporting cubes in Gazebo

April 2020

Simulated a Baxter robot transporting cubes between tables in Gazebo using ROS Kinetic. Used inverse kinematics for pick-and-place in a custom Gazebo world.

Webpage

Implemented A* algorithm for Path Planning on Turtlebot 3

May 2020

Implemented the A* algorithm in a configuration space with obstacles. The Turtlebot 3 obeys non-holonomic constraints with 8 combinations of two user-defined RPMs.

Webpage

Path planning for point and rigid robot using Dijkstra's Algorithm

April 2020

Implemented Dijkstra’s shortest path algorithm for both point and rigid robots with obstacle avoidance in 2D configuration space.

Webpage

Lane detection and Turn prediction for self driving car

March 2020

Developed lane detection using histogram-based sliding window analysis and polynomial curve fitting. Implemented bird’s-eye-view perspective transform for lane visualization.

Webpage

Agile Robotics for Industrial Automation Competition (ARIAC) 2019

May 2020

Developed an industrial system with UR 10 robotic arms, conveyor belts, and AGVs. The system picked parts from a conveyor, disposed faulty items, assembled orders, and delivered them using AGVs.

Webpage

Designed a PID controller for Turtlebot 3

April 2020

Designed and tuned a PID controller for trajectory tracking on a Turtlebot 3 in simulation.

Webpage

Modeled a UR 5 arm with Parallel Gripper in Rviz

Sep-Dec 2019

Simulated a 6 DOF UR5 arm using Moveit and Rviz. Calculated DH parameters and computed forward kinematics manually, verified via Peter Corke’s Robotics toolbox. Simulation videos

Webpage

Designed a LQR and LQG controller

Dec 2019

Developed controllers for two inverted pendulums on a moving cart.

Teleoperated gesture controlled robotic arm

Aug 2018-May 2019

Engineered a prototype to transport objects between rooms via web-based remote access with live video and gesture control. The robot featured on-board power, a custom LED light source for low-light navigation, smart device control, and a speaker for prerecorded messages.

Webpage

Pick n place transporter bot

Aug-Dec 2016

Awarded First Prize in IIT Roorkee and placed 6th out of 400 teams at IIT Bombay. Video 1 Video 2

Smart Garden

March 2017

Won First Prize in the Texas Instruments Hardware Hackathon.

Research Affiliations

I am fortunate to be mentored by distinguished faculty at the intersection of robotics, natural language processing, and agricultural technology:

DASLAB - Distributed Autonomous Systems Laboratory

Director: Prof. Girish Chowdhary

Focus: Agricultural robotics, field robots, autonomous systems, and machine learning for agriculture

Lab Website Prof. Chowdhary

Hockenmaier Lab

Director: Prof. Julia Hockenmaier

Focus: Natural language processing, computational linguistics, vision and language, semantic parsing

Lab Website Prof. Hockenmaier

Research Focus: Natural Language Grounding for Agricultural Robots - Advancing Physical AI for Humanity

Field Deployment Portfolio

In-the-wild perception on a real mobile-manipulator platform, deployed in outdoor fields and greenhouses with Jose R. Cuaran (DASLAB, UIUC). These visuals back the co-authored ICRA 2025 paper "Active Semantic Mapping with Mobile Manipulator in Horticultural Environments".

ICRA 2025 poster: Active Semantic Mapping with Mobile Manipulator in Horticultural Environments (Cuaran + Ahluwalia et al., UIUC)

ICRA 2025 - Active Semantic Mapping

Co-authored with Jose R. Cuaran. Probabilistic semantic octomaps from RGB-D, candidate viewpoint generation, and information-gain next-best-view planning. arXiv:2412.10515.

Greenhouse Data Collection

Mobile manipulator platform operating through greenhouse rows during active semantic mapping data collection.

Jose Cuaran operating the robot during a field deployment

Team Field Operations

Jose R. Cuaran (DASLAB, UIUC) operating the mobile manipulator during an outdoor deployment.

Equipment transport and field-day logistics

Field Logistics

Equipment transport for multi-day outdoor data-collection sessions with the mobile manipulator.

Miscellaneous

First prize in B.Tech. Major Project, 2019

Kulbir Ahluwalia, Akash Sharma, Garvit Periwal

Punjab Engineering College

First prize in final year MAJOR PROJECT in the B.Tech. Examination of Electrical Engineering, 2015-19 titled Teleoperated Gesture controlled Robotic arm.

Webpage Code

Certificate of Appreciation, (2017,2018)

IEEE, Punjab Engineering College

Received certificate of appreciation for contributions to IEEE PEC (2017,2018).

Webpage

Talk is cheap. Show me the code. Show me the results.

Contact

The quickest way to reach me is email: ksa5@illinois.edu. You can also find me on LinkedIn.

Email CV