Developing a phone-based imaging tool to inform on fruit volume and potential optimal harvest time

This project arises from two developments: a deepening understanding of the chemistry associated with grape maturation; and rapid development of mobile imaging techniques, especially associated with smartphones.

The mature development of mobile phones with high-quality cameras, well-developed software environments, image processing, and statistical analysis (including machine learning) for data and image classification could offer a less expensive means to measure berry skin colour, fruit volume and distribution, grape berry homogeneity versus heterogeneity. That forms the justification for the present project.

White grape colour transition (from green to yellow, and yellow to orange) during the ripening is linked to chlorophylls and carotenoids degradation, and the formation of C13 norisoprenoids. Deloire 2013, showed that using a Dyostem™ instrument (Vivelys and Montpellier SupAgro, France) to measure berry volume and skin colour in white cultivars could help the harvest decision process to achieve desirable wine styles. However, this technique is expensive, slow, destructive, and cannot be deployed in the vineyard. Other techniques such as optical fibre evanescent field absorption (FEFA) have been used to measure grape colour but they also rely on specialised equipment and the destruction of grapes.

The link between sugar accumulation and flavour development in the berry is often non-existent, compound and environment dependent, and highly influenced by harvest time. Berry size can be manipulated most efficiently with irrigation strategies and is one of the key determinants of berry shrinkage. Heterogeneity in berry volume significantly alters grape composition and can only be rectified by expensive sorting techniques in the winery. Thus, the development of in-field decision support tools will assist vineyard managers to more effectively operate commercial vineyards for irrigation management and harvest decisions for specific wine styles.

This project commenced by determining the feasibility of using desktop computing systems for berry detection and colour measurements of images of bunches of grapes. A colour reference card located within the image allowed for colour correction algorithms to be applied and served as a depth of field calibration for accurate berry size estimation. Initial image analysis showed good correlations between berry measures from the image and laboratory measures for berry size and colour in white varieties, and berry size for red varieties. Several reference items were trialled for depth of field calibrations and for the purposes of this project a probe that could be inserted into the target bunches was chosen. Inserting a probe into the bunch of grapes facilitates image capture as the camera operator is not required to hold or place an object in the image area; the probe is not obscured by fingers or the hands of the operator and the probe can be easily removed and reused. Berry illumination is an important consideration to image capture prior to analysis as colour casting arises from differentially shaded portions of the vine canopy, which in turn is influenced by the degree of cloud cover, light angle and time of day. Several image databases have been created for grape bunches at varying levels of maturity and for several cultivars, along with traditional measures of grape composition. These databases will provide a valuable resource for the development of machine learning algorithms for future projects.
To determine the spatial and temporal variability of fruit composition within a single bunch, an investigation to characterise bunch heterogeneity at two developmental stages (veraison and full maturity immediately prior to harvest) was undertaken. Berry position within bunches and branches did not influence berry composition, however, berry seed number was strongly correlated with sugar concentration fresh and dry mass of the pericarp and skin colour at veraison.

Smartphone app development was undertaken to incorporate image capture, berry detection, colour and size measures within the Android Studio with OpenCV. This App can be operated using Android version 4.0.3 (Ice Cream Sandwich). To assist in the development of the user interface, feedback from industry representatives was sought prior to final GUI development. User data incorporating vineyard, block and grape cultivar can be entered prior to image capture and running berry detection algorithms. A considerable investment of time to overcome the programming challenge of incorporating image detection and machine learning algorithms from a desktop to a smartphone computing environment was required for this project.

The resulting Android platform App (WineOZ SmartGrape) can be downloaded from the app store and interfaced with an Excel™ based dashboard to monitor the temporal and spatial data created with the image analysis features of the software. The app can be downloaded from:

https://play.google.com/store/apps/details?id=com.nwgic.grapeyield

The dashboard, along with instructions and guidelines for imaging, is available from the NWGIC web site:

http://www.csu.edu.au/nwgic/research/wineoz-smartgrape-mobile-application

The berry detection pipeline was developed within this project using a sequential process that makes use of five stages of detection:

1. Colour image capture with reference probe detection.
2. Greyscale conversion.
3. Equalisation of the grayscale image to a standardised distribution of colour.
4. Berry detection using a circle detection algorithm.
5. Berry detection filtering using a support vector machine learning algorithm.

While the image undergoes a greyscale transformation, pixel colour information within each detected berry is retained and this is used to determine the medium berry colour within the HSV colour scale. Berry size estimation is determined using a ratio of pixels across the reference probe and detected circle following image analysis. Once berry diameter has been determined an estimate of berry volume is calculated using a standard sphere formula. Metadata collected during image capture allows spatial and temporal data curation within three separate data files which are easily linked to an Excel™ dashboard to display results within a desktop environment.

Ground truthing of the app results to the industry standard measures for berry size and colour was undertaken in the 2018 growing season for Chardonnay grapes growing in the Riverina. A suite of grape bunch images was collected in full sunlight and with shading to assess the impact of illumination on berry detection and colour measures. Good comparison for berry hue angle obtained from the smartphone app and a Dyostem™ instrument were obtained for bunches in full sunlight, however, such correlations were not evident for partially shaded or fully shaded bunches. These results were obtained without any colour correction applied to the images and demonstrated a requirement for this feature to be incorporated into future versions of the app. Berry volume estimates were poorly correlated with Dyostem™ values and a number of factors are important when comparing these methods. The Dyostem™ instrument is a proprietary system and the methods used for calculation of berry volume could not be determined. Attempts to access the source code for the instrument and requests to the instrument manufacturer to obtain this information were not successful. Some berry water loss and deformation may have arisen from the berries as these were measured up to 24 hours after their collection. The number of pixels that represented the reference probe diameter (0.8 cm) may vary from image to image depending on camera position and proximity to the probe. This variance associated with the pixel diameter of the probe translates to a significant ratio difference used for berry volume calculations. This arises due to the small size of the probe and can be corrected by enlarging the probe diameter.

Small lot winemaking of Chardonnay and Sauvignon Blanc at early and advanced levels of grape maturity were undertaken to explore the link between berry colour assessment and wine style. Clear and demonstrably different sensory profiles of the wines were noted by a trained sensory panel at the early and mature stages of grape development. Importantly, changes in the wine sensory profiles were correlated to a change in berry hue angle, demonstrating the opportunity to use grape berry colour as a predictor of wine style and establishing a sufficient body of evidence to support future research to move the outcomes of this project from a proof of concept to a mature decision support tool for practical vineyard management.

Objectives

The project will develop a portable, phone-based imaging tool that will allow grape quality grading improvement at the bunch and fruit level in the field. It will measure berry skin colour evolution for white cultivars, and berry volume and number for red and white varieties.

Approach

The project will develop an automatic vision-based colour rating system for accurately measuring grape colours. The system will be calibrated using a reference card (such as ColorChecker) and calibration curves generated for the varying light intensities, illuminations, background colours and viewing angles that would be expected in the field. Machine learning analysis of multiple grape images will be used to determine the colour of a grape sample under different measurement conditions.

Having accurately calibrated the measurements to determine colour tint angles and hue, berry counts and estimates of berry volume, size distributions will be determined using digital image processing techniques. Calibrated models that estimate berry volume and mass will be developed using classic regression algorithms. The suite of codes for image, colour and size analysis will be optimised and then cast into a compact form using ‘open source’ coding methods suitable for incorporation firstly into Android and later, iOS-based phones. The final output will be a smartphone App that incorporates features developed for berry assessment.

Sector benefits

The portable, low entry cost phone based application arising from this research will provide growers and winemakers with a tool that will enable harvest decisions targeting preferred wine styles based upon objective measures in the field and provide real time feedback.