Shopping Cart Bot

The Shopping Cart Bot is a robotics project designed to automate the shopping experience by utilizing computer vision and autonomous navigation. The bot follows a person, detects and classifies items placed in the cart, and categorizes them based on predefined labels such as food, electronics, and clothing. Additionally, it integrates barcode recognition, label detection, and a payment system to streamline the checkout process.

Problem Statement

The project aims to address the following challenges:

• Object Detection and Classification: Implement computer vision techniques to identify and categorize products.
• Person Following: Develop a system that enables the bot to follow a shopper autonomously.
• Barcode and Label Recognition: Use image processing and OCR to extract information from labels and barcodes.
• Discount Application and Payment System: Calculate the final bill with category-wise discounts and generate a QR code for payment.

Motivation

This project was developed as part of the Round 3 induction project for my college robotics club. The objective was to explore the integration of robotics and computer vision in real-world applications, enhancing the retail shopping experience.

Hardware Design

• Frame: Designed using aluminum extrusion in Fusion 360.
• Wheels: Mecanum wheels were chosen due to the restricted motion in malls (tight spaces, high foot traffic, and the need for omnidirectional movement).

Person Following Implementation

Simulation Environment

• Implemented in Webots simulation.
• The world was divided into separate tracks for the person to walk and collect items.
• The cart was equipped with a camera and GPS, mounted on a mecanum-wheeled base.
• The person was equipped with a GPS sensor embedded in their body slot, transmitting signals via a dedicated channel.
• The bot’s camera continuously monitored the person, following them wherever they moved.
• The GPS signal served as an additional reference for tracking the person’s location.

Unimplemented Features Due to Time Constraints

• Handling the scenario where the person moves out of the camera frame, using GPS as a fallback.
• Detecting when the bot is stuck without any input stimulus.

Object Detection and Classification

• YOLOv5 Model: Trained using a Roboflow dataset to detect and categorize shopping items.

• Label Detection Pipeline:

  • Convert the image from RGB to grayscale.
  • Apply Canny edge detection.
  • Identify the largest contour and create a bounding box around the label.
  • Extract the bounded region from the original image.
  • Perform OCR (Optical Character Recognition) using an API to extract text.
  • Utilize Gemini API for text correction and cost retrieval.

Implemented a ROS2 node to enable communication between mecanum base motors and code via a UDP server.

Payment System

• The total cost is calculated based on item quantity, category-wise discounts, and final pricing.

• A UPI URL is used to generate a QR code.

• Scanning the QR code prompts the user to pay the exact calculated amount.

The Shopping Cart Bot successfully integrates multiple robotics and computer vision techniques to automate the shopping experience. Despite some unimplemented features due to time constraints, the project demonstrates a proof of concept for autonomous retail assistance.

Future Improvements

• Implement fallback mechanisms for lost person tracking.
• Enhance navigation strategies for obstacle avoidance.
• Develop a fully integrated hardware prototype for real-world testing.