Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
A Modular Neurosymbolic Approach for Visual Graph Question Answering
Vienna University of Technology (TU Wien), Vienna, Austria.
Vienna University of Technology (TU Wien), Vienna, Austria.
Vienna University of Technology (TU Wien), Vienna, Austria.ORCID iD: 0000-0002-9902-7662
2023 (English)In: Proceedings of the 17th International Workshop on Neural-Symbolic Learning and Reasoning La Certosa di Pontignano, Siena, Italy, July 3-5, 2023 / [ed] A. S. d'Avila Garcez, T. R. Besold, M. Gori & E. Jiménez-Ruiz, CEUR-WS , 2023, p. 139-149Conference paper, Published paper (Refereed)
Abstract [en]

Images containing graph-based structures are a ubiquitous and popular form of data representation that, to the best of our knowledge, have not yet been considered in the domain of Visual Question Answering (VQA). We use CLEGR, a graph question answering dataset with a generator that synthetically produces vertex-labelled graphs that are inspired by metro networks. Structured information about stations and lines is provided, and the task is to answer natural language questions concerning such graphs. While symbolic methods suffice to solve this dataset, we consider the more challenging problem of taking images of the graphs instead of their symbolic representations as input. Our solution takes the form of a modular neurosymbolic model that combines the use of optical graph recognition for graph parsing, a pretrained optical character recognition neural network for parsing node labels, and answer-set programming, a popular logic-based approach to declarative problem solving, for reasoning. The implementation of the model achieves an overall average accuracy of 73% on the dataset, providing further evidence of the potential of modular neurosymbolic systems in solving complex VQA tasks, in particular, the use and control of pretrained models in this architecture. 

Place, publisher, year, edition, pages
CEUR-WS , 2023. p. 139-149
Series
CEUR Workshop Proceedings, ISSN 1613-0073 ; 3432
Keywords [en]
answer-set programming, neurosymbolic computation, visual question answering, Computation theory, Graph theory, Graphic methods, Logic programming, Natural language processing systems, Text processing, Answer set programming, Data representations, Graph-based, Metro networks, Modulars, Question Answering, Vertex-labeled graphs, Visual Graph, Optical character recognition
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:hj:diva-63555Scopus ID: 2-s2.0-85167445992OAI: oai:DiVA.org:hj-63555DiVA, id: diva2:1838591
Conference
17th International Workshop on Neural-Symbolic Learning and Reasoning, Siena, Italy, July 3-5, 2023
Available from: 2024-02-16 Created: 2024-02-16 Last updated: 2024-02-16Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

ScopusFulltext

Authority records

Oetsch, Johannes

Search in DiVA

By author/editor
Oetsch, Johannes
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar

urn-nbn

Altmetric score

urn-nbn
Total: 45 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf