It is very useful for the robot to recognize person's gesture by using visual information when thinking about smooth communications of the person and the robot. In this research the gesture recognition based on the image data processing is taken as an element of the action decision of the robot as for the construction of an on-site programmable system. obtaining an operation object and a noteworthy place, etc. in case-by-case from pointing information enables the robot to decide the action from information which is not prepared beforehand.