December 2022
Volume 22, Issue 14
Open Access
Vision Sciences Society Annual Meeting Abstract  |   December 2022
Predicting Multiple behaviors from the activity of Deep Neural Networks: Is one visual hierarchy enough?
Author Affiliations
  • Aryan Zoroufi
    Department of Electrical and Computer Engineering, K.N.TOOSI university of technology
  • Aida Mirebrahimi
    Department of Computer Science, Western University
  • Leslie Ungerleider
    Laboratory of Brain and Cognition, National Institute of Mental Health
  • Chris Baker
    Laboratory of Brain and Cognition, National Institute of Mental Health
  • Maryam Vaziri-Pashkam
    Laboratory of Brain and Cognition, National Institute of Mental Health
Journal of Vision December 2022, Vol.22, 3530. doi:https://doi.org/10.1167/jov.22.14.3530
  • Views
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Aryan Zoroufi, Aida Mirebrahimi, Leslie Ungerleider, Chris Baker, Maryam Vaziri-Pashkam; Predicting Multiple behaviors from the activity of Deep Neural Networks: Is one visual hierarchy enough?. Journal of Vision 2022;22(14):3530. https://doi.org/10.1167/jov.22.14.3530.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

Recent advances in computer vision have enabled machines to have high performance in labeling objects in natural scenes. However, object labeling constitutes only a small fraction of human daily activities. To move towards building machines that can function in natural environments, the usefulness of these models should be evaluated in more diverse and natural tasks. To achieve this, it is necessary to collect databases of a broader set of human behavior towards natural objects. Here, we collected a database of two different behaviors on a large set of 3D-printed objects: 1) a grasping task and 2) a similarity judgment task. For the grasping task, we recorded participants’ finger positions when they grasped the objects. For the similarity judgment task, we asked participants to perform an odd-one-out task on triplets of objects and obtained a similarity matrix based on these judgments. Comparing these matrices across the two tasks suggests distinct features of objects are used for each. We next explored if the features extracted in different layers of the state-of-the-art deep convolutional neural networks (DNNs) could be useful in deriving both outputs. These networks are pre-trained to perform categorization tasks, yet it has been suggested that they could be adopted for other tasks. For similarity judgments, the accuracy of the predictions for similarity judgments increased from low to higher layers of the networks, while those for grasping behavior increased from low to mid-layers and then dropped dramatically at higher layers. These results suggest that for building a system that could perform these two tasks the hierarchy may need to be split starting at the mid-layers. These results could inform future models that can perform a broader set of tasks on natural images.

×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×