Technical vision system for apple defects recognition: justification, development, testing

The most rational method for identifying the quality of fruits is the optical method using PPE, which has the accuracy and stability of measurement, as well as distance and high productivity. The paper presents classification of fruit quality recognition systems and substantiates the design and technological scheme of the vision system for sorting them, consisting of an optical module with installed structural illumination and a video camera, an electronic control unit with an interface and actuators for the sorter and conveyor for fruits. In the course of the study, a single-stream type of fruit flow in PPE with forced rotation was substantiated, a structural and technological scheme of an STZ with a feeding conveyor, an optical module and a control unit, an algorithm for functioning of the STZ software was developed based on algorithm for segmentation of fruit colors, tracking algorithm, etc. deep learning ANN, which provide recognition of the size and color of fruits, as well as damage from mechanical stress, pests and diseases. The developed STZ has been introduced into the processing line for sorting and packing apples, LSP-4 has successfully passed preliminary tests and production tests at OJSC Ostromechevo. In the course of preliminary tests of the LSP-4 line, it was found that it provided fruit recognition with a probability of at least 95%, while the labor productivity made 2.5 t/h.

1. Gur’yanov D. V. Recognition of fruit quality. Robototekhnika v sel’skokhozyaistvennykh tekhnologiyakh: materialy Mezhdunarodnoi nauchno-prakticheskoi konferentsii, 10-12 noyabrya 2014 g. [Robotics in agricultural technologies: proceedings of the International scientific and practical conference, November 10-12, 2014]. Michurinsk, 2014, pp. 177-182 (in Russian).

2. Kirina M. V., Sychev V. A., Chernega A. A., Deryabina M. A., Panoryadov V. M., Divin A. G. Application of methods of luminescence analysis in computer vision systems when sorting fruits. Robototekhnika v sel’skokhozyaistvennykh tekhnologiyakh: materialy Mezhdunarodnoi nauchno-prakticheskoi konferentsii, 10-12 noyabrya 2014 g. [Robotics in agricultural technologies: proceedings of the International scientific and practical conference, November 10-12, 2014]. Michurinsk, 2014, pp. 188-191 (in Russian).

3. Lakomov D. V. Image processing for image recognition of agricultural products. Tsifrovizatsiya agropromyshlennogo kompleksa: sbornik nauchnykh statei I Mezhdunarodnoi nauchno-prakticheskoi konferentsii, Tambov, 10-12 oktyabrya 2018 g. [Digitalization of Agroindustrial Complex: collection of scientific articles I International scientific and practical conference, Tambov, October 10–12, 2018]. Tambov, 2018, vol. 1, pp. 261-263 (in Russian).

4. Bobrov V. P. Application of technical vision systems. Mekhanizatsiya i avtomatizatsiya proizvodstva [Mechanization and Automation of Production], 1989, no. 9, p. 23-25 (in Russian).

5. Busarov N. A. Collection and commodity processing of fruit and berries. Moscow, Kolos Publ., 1970. 247 p. (in Russian).

6. Rudnik Yu. A., Zhuravlev S. V. Robotic system for sorting apples. Robototekhnika v sel’skokhozyaistvennykh tekhnologiyakh: materialy Mezhdunarodnoi nauchno-prakticheskoi konferentsii, 10-12 noyabrya 2014 g. [Robotics in agricultural technologies: proceedings of the International scientific and practical conference, November 10-12, 2014]. Michurinsk, 2014, pp. 32-34 (in Russian).

7. Khort D. O., Kutyrev A. I., Smirnov I. G., Filippov R. A., Vershinin R. V. Developing algorithms for a berry recognition system used in robotized harvesting of garden strawberry. Elektrotekhnologii i elektrooborudovanie v APK = Electrical Engineering and Electrical Equipment in Agriculture, 2020, no. 1 (38), pp. 133-141 (in Russian). https://doi.org/10.22314/2658-4859-2020-67-1-133-141

8. Gur’yanov D. V. Increasing the efficiency of apple sorting on the basis of color television sensors. Ph.D. Thesis. Michurinsk, 2004. 199 p. (in Russian).

9. Ganganagowdar N. V., Gundad A. V. An intelligent computer vision system for vegetables and fruits quality inspection using soft computing techniques. Agricultural Engineering International: CIGR Journal, 2019, vol. 21, no. 3, pp. 171-178.

10. Lu Y., Lu R. Development of a multispectral Structured Illumination Reflectance Imaging (SIRI) system and its application to bruise detection of apples. Transactions of the ASABE, 2017, vol. 60, no. 4, pp. 1379-1389. https://doi.org/10.13031/trans.12158

11. Gordeev A. S. Automated processing of apples. Ph.D. Thesis. Moscow, 1996. 423 p. (in Russian).

12. Kulikov M. I., Vorob’ev V. F., Kosyakin A. S., Bychkov V. V., Kadykalo G. I., Upadysheva G. Yu. (comp.) (et al.). New technologies and technical means for the mechanization of work in horticulture. Moscow, Rosinformagrotekh Publ., 2012. 164 p. (in Russian).

13. Budagovskaya O. N. Universal semi-automatic plant for sorting fruits and vegetables by quality. Perspektivy otechestvennogo sadovodstva: tezisy dokladov II respublikanskoi konferentsii molodykh uchenykh i spetsialistov [Prospects for domestic gardening: abstracts of the II republican conference of young scientists and specialists]. Kiev, 1991, p. 138 (in Russian).

14. Budagovskaya O. N. Optical-electronic quality control of apples. Abstract of Ph.D. diss. Moscow, 1993. 24 p. (in Russian).

15. Kortylewski A., Schneider A., Gerig T., Egger B., Morel-Forster A., Vetter T. Training deep face recognition systems with synthetic data. Available at: https://arxiv.org/pdf/1802.05891.pdf (accessed 16.04.2018).

16. Chigorin A., Moiseev B. Classification of road signs based on a convolutional neural network trained on synthetic data. 22-ya Mezhdunarodnaya konferentsiya po komp’yuternoi grafike i zreniyu, GrafiKon’2012, 1-5 oktyabrya 2012 g., Moskva, Rossiya: trudy konferentsii = The 22nd International conference on computer graphics and vision: GrafiCon’2012, October 1–5, 2012, Moscow, Russia: conference proceedings. Moscow, 2012, pp. 284-287 (in Russian).

17. Zhirkova A. A., Balabanov P. V., Divin A. G. Automated system for hyperspectral control of apple defects. Sovremennaya nauka: teoriya, metodologiya, praktika: materialy III Vserossiiskoi (natsional’noi) nauchno-prakticheskoi konferentsii, Tambov, 13-14 aprelya 2021 g. [Modern science: theory, methodology, practice: proceedings of the 3rd All-Russian (national) scientific-practical conference, Tambov, April 13-14, 2021]. Tambov, 2021, pp. 291-296 (in Russian).

18. Balabanov P. V., Divin A.G., Mishchenko S.V., Makarova V. S., Markov A. V., Sadomov Yа. O. Robotic complex for sorting apples. Tsifrovizatsiya agropromyshlennogo kompleksa: sbornik nauchnykh statei II mezhdunarodnoi nauchnoprakticheskoi konferentsii, Tambov, 21-23 oktyabrya 2020 g. = Digitalization of the agro-industrial complex: рroceedings of the II International Scientific and Practical Conference, Tambov, Оctober 21-23, 2020. Tambov, 2020, vol. 2, pp. 44-48 (in Russian).

19. Mehl P. M., Chao K., Kim M., Chen Y. R. Detection of defects on selected apple cultivars using hyperspectral and multispectral image analysis. Applied Engineering in Agriculture, 2002, vol. 18, no. 2, pp. 219-226. https://doi.org/10.13031/2013.7790

20. Li C., Heinemann P. H., Reed P. M. Genetic algorithms (GAs) and evolutionary strategy to optimize electronic nose sensor selection. Transactions of the ASABE, 2008, vol. 51, no. 1, pp. 321-330. https://doi.org/10.13031/2013.24208

21. Bennedsen B. S., Peterson D. L., Tabb A. Identifying apple surface defects using principal components analysis and artificial neural networks. Transactions of the ASABE, 2007, vol. 50, no. 6, pp. 2257-2265. https://doi.org/10.13031/2013.24078

22. Uijlings J. R. R., Van de Sande K. E. A., Gevers T., Smeulders A. W. M. Selective search for object recognition. Available at: http://www.huppelen.nl/publications/selectiveSearchDraft.pdf (accessed 28.04.2018).

23. Ren S., He K., Girshick R., Sun J. Faster R-CNN: towards real-time object detection with region proposal networks. Available at: https://arxiv.org/pdf/1506.01497.pdf (accessed 15.05.2018).

24. Szegedy C., Vanhoucke V., Ioffe S., Shlens J. Rethinking the inception architecture for computer vision. Available at: https://arxiv.org/pdf/1512.00567.pdf (accessed 15.05.2018).

25. ImageNet. Available at: http://www.image-net.org (accessed 05.04.2018).

26. LeCun Y., Cortes C., Burges C. J. C. The MNIST database of handwritten digits. Available at: http://yann.lecun.com/exdb/mnist/ (accessed 16.03.2018).

27. Lin T.-Y., Belongie S., Maire M., Perona P., Hays J., Ramanan D., Dollár P., Zitnick C. L. Microsoft COCO: Common objects in context. Computer Vision - ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014: proceedings. Cham, 2014, pt. 5, pp. 740-755. https://doi.org/10.1007/978-3-319-10602-1_48

28. Tzutalin/LabelImg. Available at: https://github.com/tzutalin/labelImg (accessed 18.06.2018).