[Global 'Agri-Talk'] Robots That See, Understand and Act: Physical AI and VLA Technology

[Global 'Agri-Talk': Embracing the Future of Agriculture] Professor Da-eun Choi, University of Florida, USA

When teaching farm work, we do not explain every movement in words. After saying, “Pick only the ripe ones,” we demonstrate it by hand a few times. The worker observes the condition of the fruit and learns the direction to insert the hand and the amount of force to use when gripping it. Even to carry out one short instruction, information obtained through the eyes and judgment drawn from experience are used together.

What would it take to teach a robot the same task? Physical AI, which has recently been attracting attention, is artificial intelligence that recognizes physical environments and performs real tasks in this way.
 

A shift in how robots are taught to work

A Vision-Language Model (VLM) is AI that handles images and language together. It is used to describe what is in a photo or to find the necessary object according to instructions. A Vision-Language-Action Model (VLA) extends these abilities to robot action generation. It receives camera images and work instructions and generates commands corresponding to which direction to move an arm and when to open and close a gripper.[1, 2]

Existing robots also use cameras to find crops and move according to their positions. However, in traditional approaches, the function that finds the location of fruit, the function that calculates a path to that location, and the function that operates the harvesting device were often developed separately and connected. VLA enables robots to learn a large part of this connection from data by learning together the scene the robot sees, the work instruction, and the actual movement.[1, 2]

For this, simply showing a video of harvesting is not enough. It is necessary to record together what commands were sent to the robot arm and harvesting device when a particular scene appeared on the camera. Through these paired data, the model learns movements suited to the scene and instruction. Later, when a new video comes in, it generates the next action based on what it has learned.[2, 3]

What is noteworthy here is that already trained models can be used. This is a method of additionally training a new task based on knowledge obtained from images and text and movements learned from the work records of various robots. If the robot arm and harvesting device change, adjustments suited to them are needed, but it is not necessary to learn all abilities from scratch using only the records of the new robot.[1, 2]
 

If robots are shown how people harvest

The HarvestFlex study, released as a preprint in March 2026, shows that process. An operator watched the robot’s camera video through virtual reality (VR) equipment and controlled the arm and harvesting device. The operator approached strawberries, picked the fruit, and placed it in a container. The researchers recorded the video and the robot’s movements together at that time.[3]

The approximately 3.7 hours of data did not contain only harvesting that succeeded on the first try. It also included processes such as changing posture while approaching fruit or trying again after failing to pick it. Even the way the operator corrected movements according to the situation was left as training data.[3]

The researchers additionally trained an existing VLA model with these records. Afterward, even without remote operation by a person, the robot generated its next movements based on camera images, harvesting instructions, and its own current state. In actual greenhouse tests, the best-performing model recorded 37 successes out of 50 attempts, or a 74% harvesting success rate. This was the rate at which the task of picking the fruit and placing it in the container was completed. However, these were results obtained under the relevant test conditions, and failures remained, such as cases in which the harvesting device blocked the fruit and the robot could not continue the motion.[3]

What is noteworthy here is that on-site harvesting records were added to a model that had already learned various tasks. Rather than learning all abilities from scratch with only a few hours of records, the existing model was adjusted for strawberry harvesting based on work experience demonstrated by a person operating the robot.
 

Turning agricultural experience into robot capability

In the future, more cases may emerge in which the processes by which farmers work are recorded as video and movement data and used for robot learning. If such records accumulate across diverse crops and cultivation environments, the materials available for teaching robots new farm tasks will also become richer. There is also the possibility of using records collected on one farm to train robots for the conditions of another farm. We can look forward to a future in which the experience of farm work that has been passed on by showing people directly remains as data, and robots that have learned from it work together in the field.
 

References

 [1] Zitkovich, B., et al. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. Proceedings of The 7th Conference on Robot Learning, Proceedings of Machine Learning Research, 229, 2165–2183. Available from https://proceedings.mlr.press/v229/zitkovich23a.html
[2] Kim, M. J., et al. (2025). OpenVLA: An Open-Source Vision-Language-Action Model. Proceedings of The 8th Conference on Robot Learning, Proceedings of Machine Learning Research, 270, 2679–2713.Available from https://proceedings.mlr.press/v270/kim25c.html
[3] Zhao, Z., Wang, S., Miao, Z., & Xiong, Y. (2026). HarvestFlex: Strawberry Harvesting via Vision-Language-Action Policy Adaptation in the Wild. arXiv:2603.05982. https://doi.org/10.48550/arXiv.2603.05982.