Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
A Neuro-Symbolic ASP Pipeline for Visual Question Answering
Institute of Logic and Computation, Vienna University of Technology (TU Wien), Austria.
Institute of Logic and Computation, Vienna University of Technology (TU Wien), Austria.
Institute of Logic and Computation, Vienna University of Technology (TU Wien), Austria.ORCID iD: 0000-0002-9902-7662
Institute of Logic and Computation, Vienna University of Technology (TU Wien), Austria.
2022 (English)In: Theory and Practice of Logic Programming, ISSN 1471-0684, E-ISSN 1475-3081, Vol. 22, no 5, p. 739-754Article in journal (Refereed) Published
Abstract [en]

We present a neuro-symbolic visual question answering (VQA) pipeline for CLEVR, which is a well-known dataset that consists of pictures showing scenes with objects and questions related to them. Our pipeline covers (i) training neural networks for object classification and bounding-box prediction of the CLEVR scenes, (ii) statistical analysis on the distribution of prediction values of the neural networks to determine a threshold for high-confidence predictions, and (iii) a translation of CLEVR questions and network predictions that pass confidence thresholds into logic programmes so that we can compute the answers using an answer-set programming solver. By exploiting choice rules, we consider deterministic and non-deterministic scene encodings. Our experiments show that the non-deterministic scene encoding achieves good results even if the neural networks are trained rather poorly in comparison with the deterministic approach. This is important for building robust VQA systems if network predictions are less-than perfect. Furthermore, we show that restricting non-determinism to reasonable choices allows for more efficient implementations in comparison with related neuro-symbolic approaches without losing much accuracy.

Place, publisher, year, edition, pages
Cambridge University Press, 2022. Vol. 22, no 5, p. 739-754
Keywords [en]
answer-set programming, neuro-symbolic computation, visual question answering, Computation theory, Encoding (symbols), Forecasting, Logic programming, Program translators, Answer set programming, Deterministics, Encodings, Network prediction, Neural-networks, Object classification, Question Answering, Symbolic computation, Pipelines
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:hj:diva-63556DOI: 10.1017/S1471068422000229ISI: 000825043500001Scopus ID: 2-s2.0-85136279413OAI: oai:DiVA.org:hj-63556DiVA, id: diva2:1838576
Available from: 2024-02-16 Created: 2024-02-16 Last updated: 2024-02-16Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Oetsch, Johannes

Search in DiVA

By author/editor
Oetsch, Johannes
In the same journal
Theory and Practice of Logic Programming
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 36 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf