Hello, I'm Kulbir Singh Ahluwalia

I’m Kulbir Singh Ahluwalia, a Ph.D. candidate in Computer Science at the University of Illinois, Urbana-Champaign (UIUC), focusing on natural language grounding for agricultural robots to advance this critical field. I am fortunate to be mentored by Prof. Girish Chowdhary and Prof. Julia Hockenmaier. My academic foundation includes a Master of Engineering in Robotics from the University of Maryland, where I gained expertise in robotic systems and physics simulations. Recently, during my summer internship at EarthSense, Inc., I contributed to developing a natural language-conditioned waypoint generation pipeline. My technical skills include Robot Operating System (ROS) and implementing SOTA NLP and CV pipelines for Mobile Manipulators. I also co-developed the CS-498-GC Mobile Robotics course with Prof. Chowdhary. My long-term life goal is scaling up Physical AI for advancing humanity.

GitHub X LinkedIn Scholar CV Resume ORCID

News

[May 2026] Upcoming Compute Chipset Product Management Intern at Qualcomm. I will be in San Diego, California this summer!

[Apr 2026] Our poster WaypointGen: Natural Language to 2D Navigation Waypoints Generation using VLMs was accepted at the CSL Student Conference (CSL SC 2026) at UIUC. [poster PDF]

[Mar 2026] Began preparing WaypointGen++ for submission to IEEE Robotics and Automation Letters (RA-L), extending our VLM-based waypoint generation pipeline to outdoor mobile manipulators.

[Oct 2025] Released SLAM-ing Mars, a two-part Mobile Manipulator Exploration & SLAM challenge for CS498GC Fall 2025: students operated a Husky + UR3 + Robotiq gripper in a Mars Gazebo world. Part 1 (25 pts) covered robot setup and navigation; Part 2 (75 pts) extended to full SLAM with MoveIt 2 + RViz 2 waypoint following, 3D and 4D reconstruction of Thanksgiving-themed dynamic turkeys, and an optional Mars dust-storm bonus.

[May 2025] Our paper Active Semantic Mapping with Mobile Manipulator in Horticultural Environments was presented at ICRA 2025. arXiv:2412.10515 · project page

[May 2025] Joined Earthsense Inc. as an AI Intern for Summer 2025 in Urbana, IL, deploying VLMs (Molmo-7B, Gemma-3-27B, Qwen-2.5-VL-72B, Llama4-Scout, SpatialVLM) for natural-language-conditioned navigation on outdoor agricultural robots.

[Fall 2023] Presented the poster Plant Placement using Natural Language Grounding at the AIFARMS Annual Conference. [poster PDF]

[Aug 2022] Started my Ph.D. in Computer Science at UIUC, advised by Prof. Girish Chowdhary (DASLAB) and Prof. Julia Hockenmaier (HMR Lab).

Publications

View: show all / show selected / show by topic

Research Topics: Mobile Robotics / Computer Vision / Deep Learning / Machine Learning / NLP / Path Planning / Decision Making / 3D Vision

WaypointGen++

Kulbir Ahluwalia*, Chahit Jain*, Jose Roberto Cuaran Valenzuela, Shreya Gummadi, Michael McGuire, Arun Sivakumar, Julia Hockenmaier, Girish Chowdhary

In Preparation for IEEE RAL In Preparation for IEEE RAL

Follow-up to WaypointGen. Extending VLM-based navigation waypoint generation for mobile manipulators in outdoor environments. (In Preparation for IEEE RAL)

WaypointGen: Natural Language to 2D Navigation Waypoints Generation using VLMs

Kulbir Ahluwalia*, Chahit Jain*, Jose Roberto Cuaran Valenzuela, Shreya Gummadi, Michael McGuire, Arun Sivakumar, Julia Hockenmaier, Girish Chowdhary

Coordinated Science Laboratory Student Conference (CSL SC 2026) Conference Poster

We introduce WaypointGen, a 14-step pipeline that grounds natural language instructions to 2D navigation waypoints. We utilize a QWEN 3 VLM-based filtering approach with pre-defined templates to extract relevant geometric constraints. The method employs SLIC in Birds-Eye-View (BEV) and Model Predictive Path Integral (MPPI) for trajectory selection, demonstrating enhanced navigation capabilities for mobile manipulators in dynamic environments.

PDF

Active Semantic Mapping with Mobile Manipulator in Horticultural Environments

Jose Roberto Cuaran Valenzuela, Kulbir Ahluwalia, Kendall Koe, Naveen Kumar Uppalapati, Girish Chowdhary

International Conference on Robotics and Automation (ICRA 2025) Conference paper

We introduce an efficient active semantic mapping approach for horticultural robotics, using a mobile manipulator with an RGB-D camera. Probabilistic semantic octomaps are used to detect target regions of interest such as fruits, generate candidate viewpoints, and compute information gain for next-best-view planning. An efficient ray-casting strategy and a novel information gain function accounting for semantics and occlusions is introduced for efficient target-focused map exploration.

Project Page arXiv Code

Plant Placement using Natural Language Grounding

Kulbir Ahluwalia, Michael Vilsoet, Rajarshi Haldar, Peixin Chang, Julia Hockenmaier, Girish Chowdhary

AIFARMS Conference (Fall 2023) Conference Poster

This poster presents a text-enabled FarmBot system that enables users to control a robotic gardening FarmBot system via natural language. Using a custom Python wrapper built on the FarmBot REST API, natural language commands are grounded using real-time robot state and translated into executable code with a fine-tuned CodeT5 model. The system generates valid plant placement configurations that satisfy natural language defined spatial constraints.

Poster PDF

DeepPaSTL: Spatio-Temporal Deep Learning Methods for Predicting Long-Term Pasture Terrains Using Synthetic Datasets

Murtaza Rangwala, Jun Liu, Kulbir Ahluwalia, Shayan Ghajar, Harnaik Singh Dhami, Benjamin Tracy, Pratap Tokekar, Ryan Williams

Published in Agronomy 2021 (Special Issue AI and Agricultural Robots) Conference Paper

DeepPaSTL aims to accurately forecast long-term pasture growth, tackling the challenge of estimating pasture biomass without relying on extensive site-specific data or frequent field measurements. This approach enables predicting pasture evolution without monitoring fields regularly, using past observed pasture heights as input. DeepPaSTL introduces a bi-directional ConvLSTM encoder–decoder to learn the spatio-temporal pasture growth dynamics purely from spatial height measurements.

Paper PDF

Intermittent Deployment for Large-Scale Multi-Robot Forage Perception: Data Synthesis, Prediction, and Planning

Jun Liu, Murtaza Rangwala, Kulbir Ahluwalia, Shayan Ghajar, Harnaik Singh Dhami, Benjamin Tracy, Pratap Tokekar, Ryan Williams

IEEE Transactions on Automation Science and Engineering, 2021 Journal Paper

Targets large-scale pasture monitoring for precision agriculture, deploying a team of robots to track grassland growth for optimal rotational grazing and land productivity, addressing the lack of timely growth data in current practice. Proposes an integrated pipeline combining synthetic data generation, deep neural network-based spatiotemporal prediction, and an intermittent multi-robot deployment strategy to periodically survey evolving pastureland at low cost.

IEEE PDF arXiv

Active Semantic Mapping with Mobile Manipulator in Horticultural Environments

Jose Roberto Cuaran Valenzuela, Kulbir Ahluwalia, Kendall Koe, Girish Chowdhary

ICRA@40, 2024 Abstract

An abstract accepted at the 40th Anniversary of the IEEE Conference on Robotics and Automation (ICRA@40), 2024.

Smartphone Optical Sensors

Simarjeet S Saini, Aneesh Sridhar, Kulbir Ahluwalia

Optics and Photonics News, 2019 Featured Article

An article featuring the multispectral Fundus Eye camera prototype, as presented in Optics and Photonics News.

Paper Article PDF

Reinforcement Learning Integrated with Supervised Learning for Training of Near Infrared Spectrum Data for Non-Destructive Testing of Fruits

Yuqi Li, Kulbir Ahluwalia, Simarjeet S Saini

Sensing for Agriculture and Food Quality and Safety XII, 2020 Conference Presentation

A conference presentation on combining reinforcement and supervised learning for non-destructive testing of fruits using near infrared spectrum data.

Paper

Teaching

Teaching Assistant - CS498GC: Mobile Robotics for CS

University of Illinois, Aug 2025 - Present

Co-developed with Prof. Girish Chowdhary
• Co-developed course curriculum focusing on mobile robotics, ROS2, sensor fusion, and SLAM algorithms.
• Managing coding exercises and problem sets involving Extended Kalman Filtering and odometry implementation.
• Conducting office hours and helping students with ROS2 development and debugging.
• Maintaining course website and autograding infrastructure on Gradescope.
• Special Topic Fall 2025: SLAM-ing on Mars

Fall 2025 Course Website Instructor

Teaching Assistant - CS444: Deep Learning for Computer Vision

University of Illinois, Jan 2024 - May 2024, Jan 2025 - May 2025

Instructor: Dr. Svetlana Lazebnik
• Updated and verified starter code for assignments, and answered student questions during office hours and through Campuswire.
• Assessed student submissions via SpeedGrader on Canvas, and designed multimodal quiz questions, including single- choice, multiple-choice, and matching formats.

Spring 2024 Spring 2025

Teaching Assistant - CS519: Scientific Visualization

University of Illinois, May 2025 - Aug 2025

Instructor: Dr. Eric Shaffer
• Created multimodal exam questions with integrated visualizations using Python and matplotlib for assessing student understanding of scientific visualization concepts.
• Assisted students with implementation of advanced visualization algorithms including ray marching, transfer functions, and interactive widget development.

Course Page Instructor

Work Experience

AI Intern - Earthsense Inc.

Urbana, IL, USA | May 2025 - Aug 2025

Supervisor: Michael McGuire, Lead Computer Vision Engineer
• Key Achievement: Contributed to developing a natural language-conditioned waypoint generation pipeline for agricultural robot navigation.
• Implemented state-of-the-art NLP and CV pipelines for Mobile Manipulators, enabling natural language instruction following.
• Created an automatic labeling pipeline for large outdoor robot navigation datasets using Grounded SAM2, streamlining data processing.
• Deployed and integrated open-source Visual Language Models (Molmo-7B-demo, Gemma-3-27B, Qwen-2.5-VL-72B, Qwen3-30B, Llama4-Scout, Spatial-VLM) for robot reasoning in image space and open-world natural language instruction conditioned question answering for 4 wheeled skid steer outdoor robots.
• Enhanced ROS-based systems for real-world agricultural applications, directly supporting the advancement of Physical AI.

Company Website

Projects

View: show all / show selected / show by topic

Topics: Mobile Robotics / Computer Vision / Deep Learning / Machine Learning / NLP / Path Planning / 3D Vision

Enhancing Stereo Depth Maps through RGBD-Conditioned Generative Models

Aug-Dec 2024

Fine-tuned Stable Diffusion V2 for monocular depth estimation, achieving 46% improvement over Marigold and Depth-Anything-V2 benchmarks.

Active Semantic Mapping with Mobile Manipulator for Precision Agriculture

Jan-May 2024

Probabilistic semantic octomaps for detecting target regions, generating candidate viewpoints, and computing information gain for next-best-view planning on a mobile manipulator.

Turning Zero Shot into Few Shot via Self-prompting

Aug-Dec 2023

Explored self-prompting strategies to convert zero-shot language model capabilities into effective few-shot performance without manual prompt engineering.

Implemented Linear, Logistic, Polynomial regression, SVM, Convolutional, and Attention based models

Apr-May 2023

Built a comprehensive suite of classical and deep learning models from scratch, including SVMs, CNNs, and Attention mechanisms for classification and regression tasks.

VAE and GAN implementation

March 2023

Implemented VAE & GAN for digit generation task.

RESNet implementation

Feb 2023

Implemented RESNet for classification task on MNIST dataset.

Neural transition-based dependency parser

Dec 2022

Built a neural transition-based dependency parser using feed-forward networks for syntactic analysis of natural language sentences.

Webpage

Machine translation with RNN and Transformers

Nov 2022

Implemented sequence-to-sequence machine translation using both RNN encoder-decoders and Transformer architectures.

Webpage

Text classification with CNN and RNN

Oct 2022

Compared CNN and RNN architectures for text classification, exploring the trade-offs between local feature extraction and sequential modeling.

Webpage

SLAM from 2D LiDAR data using split and merge line extraction algorithm

Dec 2021

Implemented simultaneous localization and mapping from 2D LiDAR scans using split-and-merge line extraction for feature-based SLAM.

Webpage

State estimation using Extended Kalman Filter for GPS+Encoder sensor fusion

Nov 2021

Fused GPS and wheel encoder data using an Extended Kalman Filter for robust state estimation on a mobile robot.

Webpage

Processed data from RTK-GPS, IMU and encoders to plot trajectory of a field robot

Sep 2021

Processed and visualized multi-sensor trajectory data from RTK-GPS, IMU, and encoders using ROS for field robot navigation.

Webpage

Autonomous Vaccine Delivery Robot

May 2021

Designed an autonomous robot capable of navigating and localizing itself in a test arena using QR codes and arrows. It uses a RGB camera, IMU, optical encoders, and an ultrasonic sensor to detect, retrieve and transport user-specified blocks. Featured video, Robot videos, Featured post

Webpage

Image segmentation using superpixels

Dec 2020

Built a segmentation pipeline using SLIC superpixels with VGG16 feature extraction for superpixel classification.

Webpage

Persistent-Monitoring using Multi-Robot (UAV-UGV) Coordination

Dec 2020

Coordinated UAV-UGV teams for persistent monitoring of large-scale environments with optimized deployment schedules.

Webpage

Optimized a GestureGAN for resource constrained settings

Dec 2020

Used MobileNet to optimize cross-view image generation with a 5.7X reduction in parameters.

Webpage

Self-adjusting roadmaps

May 2020

Navigation in unknown environments using LD-PRM.

Webpage

Estimated the motion of a car using Visual odometry

May 2020

Estimated camera ego-motion from sequential video frames using feature matching and essential matrix decomposition for visual odometry.

Webpage

Color segmentation using Gaussian mixture models & Expectation maximization

April 2020

Segmented colored buoys in underwater imagery using GMMs trained via Expectation Maximization for robust color-based detection.

Webpage

Image Classification using CNN

May 2020

Built a CNN for binary image classification (cat vs dog) with data augmentation and transfer learning.

Webpage

AR-Tag detection

March 2020

Superimposed an image and virtual cube on an AR tag.

Webpage

Tracked moving objects using Lucas-Kanade Tracker

April 2020

Implemented the Lucas-Kanade optical flow tracker for real-time object tracking across video frames.

Webpage

Baxter transporting cubes in Gazebo

April 2020

Simulated a Baxter robot transporting cubes between tables in Gazebo using ROS Kinetic. Used inverse kinematics for pick-and-place in a custom Gazebo world.

Webpage

Implemented A star algorithm for Path Planning on Turtlebot 3

May 2020

Implemented the A* algorithm in a configuration space with obstacles. The Turtlebot 3 obeys non-holonomic constraints with 8 combinations of two user-defined RPMs.

Webpage

Path planning for point and rigid robot using Djikstra's Algorithm

April 2020

Implemented Dijkstra’s shortest path algorithm for both point and rigid robots with obstacle avoidance in 2D configuration space.

Webpage

Lane detection and Turn prediction for self driving car

March 2020

Developed lane detection using histogram-based sliding window analysis and polynomial curve fitting. Implemented bird’s-eye-view perspective transform for lane visualization.

Webpage

Agile Robotics for Industrial Automation Competition (ARIAC) 2019

May 2020

Developed an industrial system with UR 10 robotic arms, conveyor belts, and AGVs. The system picked parts from a conveyor, disposed faulty items, assembled orders, and delivered them using AGVs.

Webpage

Designed a PID controller for Turtlebot 3

April 2020

Designed and tuned a PID controller for trajectory tracking on a Turtlebot 3 in simulation.

Webpage

Modelled a UR 5 arm with Parallel Gripper in Rviz

Sep-Dec 2019

Simulated a 6 DOF UR5 arm using Moveit and Rviz. Calculated DH parameters and computed forward kinematics manually, verified via Peter Corke’s Robotics toolbox. Simulation videos

Webpage

Designed a LQR and LQG controller

Dec 2019

Developed controllers for two inverted pendulums on a moving cart.

Teleoperated gesture controlled robotic arm

Aug 2018-May 2019

Engineered a prototype to transport objects between rooms via web-based remote access with live video and gesture control. The robot featured on-board power, a custom LED light source for low-light navigation, smart device control, and a speaker for prerecorded messages.

Webpage

Pick n place transporter bot

Aug-Dec 2016

Awarded First Prize in IIT Roorkee and placed 6th out of 400 teams at IIT Bombay. Video 1 Video 2

Smart Garden

March 2017

Won First Prize in the Texas Instruments Hardware Hackathon.

Research Affiliations

I am fortunate to be mentored by distinguished faculty at the intersection of robotics, natural language processing, and agricultural technology:

DASLAB - Distributed Autonomous Systems Laboratory

Director: Prof. Girish Chowdhary

Focus: Agricultural robotics, field robots, autonomous systems, and machine learning for agriculture

Lab Website Prof. Chowdhary

HMR Lab - Hockenmaier Research Group

Director: Prof. Julia Hockenmaier

Focus: Natural language processing, computational linguistics, vision and language, semantic parsing

Lab Website Prof. Hockenmaier

Research Focus: Natural Language Grounding for Agricultural Robots - Advancing Physical AI for Humanity

Field Deployment Portfolio

In-the-wild perception on a real mobile-manipulator platform, deployed in outdoor fields and greenhouses with Jose R. Cuaran (DASLAB, UIUC). These visuals back the co-authored ICRA 2025 paper "Active Semantic Mapping with Mobile Manipulator in Horticultural Environments".

ICRA 2025 poster: Active Semantic Mapping with Mobile Manipulator in Horticultural Environments (Cuaran + Ahluwalia et al., UIUC)

ICRA 2025 - Active Semantic Mapping

Co-authored with Jose R. Cuaran. Probabilistic semantic octomaps from RGB-D, candidate viewpoint generation, and information-gain next-best-view planning. arXiv:2412.10515.

Greenhouse Data Collection

Mobile manipulator platform operating through greenhouse rows during active semantic mapping data collection.

Jose Cuaran operating the robot during a field deployment

Team Field Operations

Jose R. Cuaran (DASLAB, UIUC) operating the mobile manipulator during an outdoor deployment.

Equipment transport and field-day logistics

Field Logistics

Equipment transport for multi-day outdoor data-collection sessions with the mobile manipulator.

Miscellaneous

First prize in B.Tech. Major Project, 2019

Kulbir Ahluwalia, Akash Sharma, Garvit Periwal

Punjab Engineering College

First prize in final year MAJOR PROJECT in the B.Tech. Examination of Electrical Engineering, 2015-19 titled Teleoperated Gesture controlled Robotic arm.

Video Webpage

Certificate of Appreciation, (2017,2018)

IEEE, Punjab Engineering College

Received certificate of appreciation for contributions to IEEE PEC (2017,2018).

Webpage

Webpage

Talk is cheap. Show me the code. Show me the results.