Abstract
The rapid advancement of computer vision and augmented reality has enabled the development of intelligent systems that enhance real-time user interaction. This project presents a Python-based real-time virtual try-on system that allows users to dynamically overlay fashion accessories such as glasses, hats, earrings, and eyelashes onto a live camera feed. The system is implemented using OpenCV for image processing, MediaPipe Face Mesh for accurate facial landmark detection, and Tkinter for providing a simple graphical user interface.
The proposed system captures live video from a webcam and detects facial landmarks in real time using a deep learning–based face mesh model. Based on these landmarks, selected accessories are precisely positioned, scaled, and aligned to match the user’s face geometry. Background removal techniques using HSV colour space and alpha channel manipulation are applied to ensure seamless blending of accessories with the live video. The system supports dynamic switching of accessories through a user-friendly interface without interrupting the video stream.
This virtual try-on framework offers an efficient, lightweight, and cost-effective solution for augmented reality applications in fashion and e-commerce. It eliminates the need for physical trials, enhances user experience, and demonstrates the practical use of real-time computer vision techniques. The system can be further extended to support additional accessories, improved realism, and deployment in web or mobile platforms.
Keywords:
Virtual Try-On, Computer Vision, OpenCV, Augmented Reality, Python







Introduction
The rapid growth of computer vision, artificial intelligence, and augmented reality technologies has significantly transformed the way humans interact with digital systems. In recent years, real-time visual processing applications have gained widespread attention across multiple domains such as healthcare, education, entertainment, security, and e-commerce. Among these, virtual try-on systems have emerged as an innovative solution that bridges the gap between physical and digital experiences. Virtual try-on technology enables users to visualize accessories or products on themselves in real time without physically wearing them, thereby enhancing convenience, engagement, and decision-making.
Traditional shopping methods require customers to physically try products such as glasses, hats, or jewelry to evaluate their suitability. This process is time-consuming, inconvenient, and sometimes impractical, especially in online shopping environments. With the growth of e-commerce platforms, customers often rely only on static images, which fail to provide a realistic understanding of how a product will appear when worn. This limitation can lead to dissatisfaction, incorrect purchases, and higher return rates. To address these challenges, virtual try-on systems have been introduced as a digital alternative that allows users to preview products interactively using a camera-enabled device.
A virtual try-on system combines computer vision techniques with real-time image processing to detect facial features and overlay digital objects accurately on a user’s face. The success of such a system depends heavily on precise facial landmark detection, correct alignment of virtual assets, and seamless blending with the live video feed. Advances in deep learning-based facial landmark detection models have made it possible to achieve high accuracy and real-time performance even on consumer-grade hardware. This has opened new opportunities for developing lightweight and efficient augmented reality applications using widely available programming tools.
Python has emerged as one of the most popular programming languages for computer vision and artificial intelligence applications due to its simplicity, extensive library support, and strong community ecosystem. Libraries such as OpenCV provide powerful tools for image processing, video capture, and visualization, while MediaPipe offers robust and optimized machine learning models for face detection and facial landmark tracking. Together, these technologies enable the development of real-time vision-based systems without requiring complex hardware or proprietary software.
This project focuses on the development of a real-time virtual try-on system using Python, OpenCV, and MediaPipe Face Mesh. The system captures live video from a webcam and detects facial landmarks with high precision. Based on these landmarks, virtual accessories such as glasses, hats, earrings, and eyelashes are dynamically overlaid onto the user’s face. The system ensures that the accessories scale and move naturally with the user’s facial movements, providing a realistic augmented reality experience. A graphical user interface is implemented using Tkinter to allow users to easily select and switch between different accessories during runtime.
One of the key challenges in virtual try-on applications is accurate facial landmark detection. Human faces are highly dynamic and vary significantly in shape, size, and orientation. Lighting conditions, camera quality, and head movements further complicate the detection process. MediaPipe Face Mesh addresses these challenges by using a deep learning-based model that detects hundreds of facial landmarks in real time. These landmarks represent critical facial regions such as eyes, nose, mouth, ears, and forehead, enabling precise placement of virtual objects. The use of MediaPipe ensures high accuracy while maintaining real-time performance.
Another important aspect of the system is background removal and blending of virtual assets. Accessories such as glasses or earrings are typically stored as image files with visible backgrounds that may not blend naturally with the live video feed. To overcome this issue, the system applies background removal techniques using color space analysis and alpha channel manipulation. By converting images to appropriate color spaces and generating transparency masks, the background of the accessory images is removed, allowing them to blend seamlessly with the user’s face. This improves visual realism and enhances user experience.
The overlay process involves resizing and positioning virtual assets based on facial measurements derived from detected landmarks. For example, the width of the face is used to scale accessories proportionally, while specific landmark points determine their exact placement. Earrings are aligned with ear landmarks, glasses are positioned between eye landmarks, and hats are placed relative to forehead landmarks. This geometric approach ensures that accessories remain correctly aligned even when the user moves their head or changes facial expressions.
Real-time performance is a crucial requirement for virtual try-on systems. Any noticeable lag or delay can negatively impact user experience and reduce system usability. The proposed system is designed to be computationally efficient and capable of running smoothly on standard desktop hardware. By leveraging optimized libraries and multithreading techniques, the system maintains a continuous video stream while processing facial landmarks and rendering overlays simultaneously. This ensures smooth interaction and responsiveness.
The graphical user interface plays an important role in making the system user-friendly. Instead of relying on command-line inputs, the system provides interactive buttons that allow users to select different accessories with a single click. This design choice makes the system accessible even to users with minimal technical knowledge. The interface operates independently of the video processing loop, allowing users to change accessories without interrupting the live feed.
Virtual try-on technology has significant real-world applications beyond fashion and entertainment. In e-commerce, it can reduce product return rates and increase customer satisfaction by enabling informed purchasing decisions. In marketing, interactive try-on experiences can increase user engagement and brand visibility. In education and research, such systems demonstrate the practical application of computer vision concepts and serve as effective learning tools. The proposed project highlights how theoretical knowledge of image processing and machine learning can be translated into a practical, real-time application.
This project also serves as an example of how augmented reality systems can be developed using open-source tools. Unlike commercial AR platforms that require expensive hardware or proprietary software, the proposed system uses freely available libraries and standard webcams. This makes the solution cost-effective and scalable. The modular structure of the code allows easy extension of the system to support additional accessories, improved rendering techniques, or integration with web and mobile platforms.

Objectives
1. To design and develop a real-time virtual try-on system using Python and computer vision techniques.
2. To implement accurate facial landmark detection using MediaPipe Face Mesh for precise accessory placement.
3. To enable real-time webcam-based video processing for interactive user experience.
4. To overlay virtual accessories such as glasses, hats, earrings, and eyelashes on detected facial regions.
5. To apply background removal and alpha blending techniques for seamless integration of virtual assets.
6. To ensure dynamic scaling and alignment of accessories based on facial geometry and head movement.
7. To develop a user-friendly graphical interface for selecting and switching accessories in real time.
8. To achieve efficient and smooth system performance on standard desktop hardware.
9. To demonstrate the practical application of augmented reality in fashion and e-commerce domains.
10. To create a modular and extensible framework that can be enhanced with additional accessories or features in the future.

Block Diagram