Hello, I'm Kulbir Singh Ahluwalia

I’m Kulbir Singh Ahluwalia, a Ph.D. candidate in Computer Science at the University of Illinois, Urbana-Champaign (UIUC), focusing on natural language grounding for agricultural robots to advance this critical field. I am fortunate to be mentored by Prof. Girish Chowdhary and Prof. Julia Hockenmaier. My academic foundation includes a Master of Engineering in Robotics from the University of Maryland, where I gained expertise in robotic systems and physics simulations. Recently, during my summer internship at EarthSense, Inc., I contributed to developing a natural language-conditioned waypoint generation pipeline. My technical skills include Robot Operating System (ROS) and implementing SOTA NLP and CV pipelines for Mobile Manipulators. I also co-developed the CS-498-GC Mobile Robotics course with Prof. Chowdhary. My long-term life goal is scaling up Physical AI for advancing humanity.

GitHub X LinkedIn Scholar CV Resume ORCID

Publications

View: show all / show selected / show by topic

Research Topics: Mobile Robotics / Computer Vision / Deep Learning / Machine Learning / NLP / Path Planning / Decision Making / 3D Vision

WaypointGen++

Kulbir Ahluwalia, Chahit Jain

Under Review Ongoing Project

Follow-up to WaypointGen. Ongoing project extending VLM-based navigation waypoint generation. (Under Review)

WaypointGen: Natural Language to 2D Navigation Waypoints Generation using VLMs

Kulbir Ahluwalia, Chahit Jain

Coordinated Science Laboratory Student Conference (CSL SC 2026) Accepted Poster

We introduce WaypointGen, a 14-step pipeline that grounds natural language instructions to 2D navigation waypoints. We utilize a QWEN 3 VLM-based filtering approach with pre-defined templates to extract relevant geometric constraints. The method employs SLIC in Birds-Eye-View (BEV) and Model Predictive Path Integral (MPPI) for trajectory selection, demonstrating enhanced navigation capabilities for mobile manipulators in dynamic environments.

PDF

Active Semantic Mapping with Mobile Manipulator in Horticultural Environments

Jose Roberto Cuaran Valenzuela, Kulbir Ahluwalia, Kendall Koe, Naveen Kumar Uppalapati, Girish Chowdhary

International Conference on Robotics and Automation (ICRA 2025) Conference paper

We introduce an efficient active semantic mapping approach for horticultural robotics, using a mobile manipulator with an RGB-D camera. Probabilistic semantic octomaps are used to detect target regions of interest such as fruits, generate candidate viewpoints, and compute information gain for next-best-view planning. An efficient ray-casting strategy and a novel information gain function accounting for semantics and occlusions is introduced for efficient target-focused map exploration.

Project Page arXiv Code

Plant Placement using Natural Language Grounding

Kulbir Ahluwalia, Michael Vilsoet, Rajarshi Haldar, Peixin Chang, Julia Hockenmaier, Girish Chowdhary

AIFARMS Conference (Fall 2023) Conference Poster

This poster presents a text-enabled FarmBot system that enables users to control a robotic gardening FarmBot system via natural language. Using a custom Python wrapper built on the FarmBot REST API, natural language commands are grounded using real-time robot state and translated into executable code with a fine-tuned CodeT5 model. The system generates valid plant placement configurations that satisfy natural language defined spatial constraints.

Poster PDF

DeepPaSTL: Spatio-Temporal Deep Learning Methods for Predicting Long-Term Pasture Terrains Using Synthetic Datasets

Murtaza Rangwala, Jun Liu, Kulbir Ahluwalia, Shayan Ghajar, Harnaik Singh Dhami, Benjamin Tracy, Pratap Tokekar, Ryan Williams

Published in Agronomy 2021 (Special Issue AI and Agricultural Robots) Conference Paper

DeepPaSTL aims to accurately forecast long-term pasture growth, tackling the challenge of estimating pasture biomass without relying on extensive site-specific data or frequent field measurements. This approach enables predicting pasture evolution without monitoring fields regularly, using past observed pasture heights as input. DeepPaSTL introduces a bi-directional ConvLSTM encoder–decoder to learn the spatio-temporal pasture growth dynamics purely from spatial height measurements.

Paper PDF

Intermittent Deployment for Large-Scale Multi-Robot Forage Perception: Data Synthesis, Prediction, and Planning

Jun Liu, Murtaza Rangwala, Kulbir Ahluwalia, Shayan Ghajar, Harnaik Singh Dhami, Benjamin Tracy, Pratap Tokekar, Ryan Williams

IEEE Transactions on Automation Science and Engineering, 2021 Journal Paper

Targets large-scale pasture monitoring for precision agriculture, deploying a team of robots to track grassland growth for optimal rotational grazing and land productivity, addressing the lack of timely growth data in current practice. Proposes an integrated pipeline combining synthetic data generation, deep neural network-based spatiotemporal prediction, and an intermittent multi-robot deployment strategy to periodically survey evolving pastureland at low cost.

IEEE PDF arXiv

Active Semantic Mapping with Mobile Manipulator in Horticultural Environments

Jose Roberto Cuaran Valenzuela, Kulbir Ahluwalia, Kendall Koe, Girish Chowdhary

ICRA@40, 2024 Abstract

An abstract accepted at the 40th Anniversary of the IEEE Conference on Robotics and Automation (ICRA@40), 2024.

Smartphone Optical Sensors

Simarjeet S Saini, Aneesh Sridhar, Kulbir Ahluwalia

Optics and Photonics News, 2019 Featured Article

An article featuring the multispectral Fundus Eye camera prototype, as presented in Optics and Photonics News.

Paper Article PDF

Reinforcement Learning Integrated with Supervised Learning for Training of Near Infrared Spectrum Data for Non-Destructive Testing of Fruits

Yuqi Li, Kulbir Ahluwalia, Simarjeet S Saini

Sensing for Agriculture and Food Quality and Safety XII, 2020 Conference Presentation

A conference presentation on combining reinforcement and supervised learning for non-destructive testing of fruits using near infrared spectrum data.

Paper

Teaching

Teaching Assistant - CS498GC: Mobile Robotics for CS

University of Illinois, Aug 2025 - Present

Co-developed with Prof. Girish Chowdhary
• Co-developed course curriculum focusing on mobile robotics, ROS2, sensor fusion, and SLAM algorithms.
• Managing coding exercises and problem sets involving Extended Kalman Filtering and odometry implementation.
• Conducting office hours and helping students with ROS2 development and debugging.
• Maintaining course website and autograding infrastructure on Gradescope.
• Special Topic Fall 2025: SLAM-ing on Mars

Fall 2025 Course Website Instructor

Teaching Assistant - CS444: Deep Learning for Computer Vision

University of Illinois, Jan 2024 - May 2024, Jan 2025 - May 2025

Instructor: Dr. Svetlana Lazebnik
• Updated and verified starter code for assignments, and answered student questions during office hours and through Campuswire.
• Assessed student submissions via SpeedGrader on Canvas, and designed multimodal quiz questions, including single- choice, multiple-choice, and matching formats.

Spring 2024 Spring 2025

Teaching Assistant - CS519: Scientific Visualization

University of Illinois, May 2025 - Aug 2025

Instructor: Dr. Eric Shaffer
• Created multimodal exam questions with integrated visualizations using Python and matplotlib for assessing student understanding of scientific visualization concepts.
• Assisted students with implementation of advanced visualization algorithms including ray marching, transfer functions, and interactive widget development.

Course Page Instructor

Work Experience

AI Intern - Earthsense Inc.

Urbana, IL, USA | May 2025 - Aug 2025

Supervisor: Michael McGuire, Lead Computer Vision Engineer
• Key Achievement: Contributed to developing a natural language-conditioned waypoint generation pipeline for agricultural robot navigation.
• Implemented state-of-the-art NLP and CV pipelines for Mobile Manipulators, enabling natural language instruction following.
• Created an automatic labeling pipeline for large outdoor robot navigation datasets using Grounded SAM2, streamlining data processing.
• Deployed and integrated open-source Visual Language Models (Molmo-7B-demo, Gemma-3-27B, Qwen-2.5-VL-72B, Qwen3-30B, Llama4-Scout, Spatial-VLM) for robot reasoning in image space and open-world natural language instruction conditioned question answering for 4 wheeled skid steer outdoor robots.
• Enhanced ROS-based systems for real-world agricultural applications, directly supporting the advancement of Physical AI.

Company Website

Projects

View: show all / show selected / show by topic

Topics: Mobile Robotics / Computer Vision / Deep Learning / Machine Learning / NLP / Path Planning / 3D Vision

Enhancing Stereo Depth Maps through RGBD-Conditioned Generative Models

Aug-Dec 2024

Active Semantic Mapping with Mobile Manipulator for Precision Agriculture

Jan-May 2024

Turning Zero Shot into Few Shot via Self-prompting

Aug-Dec 2023

Implemented Linear, Logistic, Polynomial regression, SVM, Convolutional, and Attention based models

Apr-May 2023

VAE and GAN implementation

March 2023

Implemented VAE & GAN for digit generation task.

RESNet implementation

Feb 2023

Implemented RESNet for classification task on MNIST dataset.

Neural transition-based dependency parser

Dec 2022

Webpage

Machine translation with RNN and Transformers

Nov 2022

Webpage

Text classification with CNN and RNN

Oct 2022

Webpage

SLAM from 2D LiDAR data using split and merge line extraction algorithm

Dec 2021

Webpage

State estimation using Extended Kalman Filter for GPS+IMU+Encoder sensor fusion

Nov 2021

Webpage

Processed data from RTK-GPS, IMU and encoders to plot trajectory of a field robot

Sep 2021

Webpage

Autonomous Vaccine Delivery Robot

May 2021

Designed an autonomous robot capable of navigating and localizing itself in a test arena using QR codes and arrows. It uses a RGB camera, IMU, optical encoders, and an ultrasonic sensor to detect, retrieve and transport user-specified blocks. Featured video, Robot videos, Featured post

Webpage

Image segmentation using superpixels

Dec 2020

Built a segmentation network using SLIC superpixels as input. A pretrained VGG16 network had its last layers replaced by fully connected layers to classify superpixels. (Accuracy 98%)

Webpage

Persistent-Monitoring using Multi-Robot (UAV-UGV) Coordination

Dec 2020

Webpage

Optimized a GestureGAN for resource constrained settings

Dec 2020

Used MobileNet to optimize cross-view image generation with a 5.7X reduction in parameters.

Webpage

Self-adjusting roadmaps

May 2020

Navigation in unknown environments using LD-PRM.

Webpage

Estimated the motion of a car using Visual odometry

May 2020

Webpage

Color segmentation using Gaussian mixture models & Expectation maximization

April 2020

Webpage

Image Classification using CNN

May 2020

Webpage

AR-Tag detection

March 2020

Superimposed an image and virtual cube on an AR tag.

Webpage

Tracked moving objects using Lucas-Kanade Tracker

April 2020

Webpage

Baxter transporting cubes in Gazebo

April 2020

Simulated a Baxter robot transporting cubes between tables in Gazebo using ROS Kinetic. Waypoints were generated using Rviz and obstacles were avoided in the custom-designed Gazebo world.

Webpage

Implemented A star algorithm for Path Planning on Turtlebot 3

May 2020

Implemented the A* algorithm in a configuration space with obstacles. The Turtlebot 3 obeys non-holonomic constraints with 8 combinations of two user-defined RPMs.

Webpage

Path planning for point and rigid robot using Djikstra's Algorithm

April 2020

Webpage

Lane detection and Turn prediction for self driving car

March 2020

Developed an algorithm using hough transform and histogram of lanes. Also implemented homography and warp perspective functions from scratch for overlays.

Webpage

Agile Robotics for Industrial Automation Competition (ARIAC) 2019

May 2020

Developed an industrial system with UR 10 robotic arms, conveyor belts, and AGVs. The system picked parts from a conveyor, disposed faulty items, assembled orders, and delivered them using AGVs.

Webpage

Designed a PID controller for Turtlebot 3

April 2020

Webpage

Modelled a UR 5 arm with Parallel Gripper in Rviz

Sep-Dec 2019

Simulated a 7 DOF UR5 arm using Moveit and Rviz. Calculated DH parameters and computed forward kinematics manually, verified via Peter Corke’s Robotics toolbox. Simulation videos

Webpage

Designed a LQR and LQG controller

Dec 2019

Developed controllers for two inverted pendulums on a moving cart.

Teleoperated gesture controlled robotic arm

Aug 2018-May 2019

Engineered a prototype to transport objects between rooms via web-based remote access with live video and gesture control. The robot featured on-board power, a custom LED light source for low-light navigation, smart device control, and a speaker for prerecorded messages.

Webpage

Pick n place transporter bot

Aug-Dec 2016

Awarded First Prize in IIT Roorkee and placed 6th out of 400 teams at IIT Bombay. Video 1 Video 2

Smart Garden

March 2017

Won First Prize in the Texas Instruments Hardware Hackathon.

Research Affiliations

I am fortunate to be mentored by distinguished faculty at the intersection of robotics, natural language processing, and agricultural technology:

DASLAB - Distributed Autonomous Systems Laboratory

Director: Prof. Girish Chowdhary

Focus: Agricultural robotics, field robots, autonomous systems, and machine learning for agriculture

Lab Website Prof. Chowdhary

HMR Lab - Hockenmaier Research Group

Director: Prof. Julia Hockenmaier

Focus: Natural language processing, computational linguistics, vision and language, semantic parsing

Lab Website Prof. Hockenmaier

Research Focus: Natural Language Grounding for Agricultural Robots - Advancing Physical AI for Humanity

Miscellaneous

First prize in B.Tech. Major Project, 2019

Kulbir Ahluwalia, Akash Sharma, Garvit Periwal

Punjab Engineering College

First prize in final year MAJOR PROJECT in the B.Tech. Examination of Electrical Engineering, 2015-19 titled Teleoperated Gesture controlled Robotic arm.

Video Webpage

Certificate of Appreciation, (2017,2018)

IEEE, Punjab Engineering College

Received certificate of appreciation for contributions to IEEE PEC (2017,2018).

Webpage

Webpage

Talk is cheap. Show me the code. Show me the results.- Linus Torvalds, Shivansh Patel