More Details of OQT Analysis Framework Evaluation

My paper entitled “An Analysis Framework for Ontology Querying Tools” is accepted in the upcoming i-Semantic 2013 conference in Graz, 4-6 September this year. This paper introduces an Ontology Querying Tools (OQT) analysis framework to enable classification and comparison regarding OQT capabilities and support for the intended user groups, which are supported by the scoring system and usage steps.

While most of the important results is presented in the paper, here we present some detail information that was left out because of space limitation of the paper.

1. Questionnaire Design

(1) Please provide the information about your knowledge in following field in likert scale within the bracket provided “[ ]”
1: No Knowledge
2: Limited Knowledge
3: Quite Good Knowledgeable
4: Good Knowledge
5: Very Good Knowledge

[ ] Ontology / Semantic Web technologies
[ ] Multidiciplinary Engineering Environments

(2) Please provide your opinion regarding the importance of each ontology querying feature below for your work in CDL using likert scale (1-5) in the bracket provided “[ ]”
1: Not important at all
2: May be important
3: Quite important
4: Important
5: Very important

Ontology querying features
[ ] supports for querying in SPARQL language

Query User Interface ( availability of each querying user interface )
[ ] availability of keyword search / guided natural language search
[ ] availability of graphical query formulation assistance user interface
[ ] availability of the user clarification form / user interface to clarify user query

Query Features
Selection Queries
[ ] extract raw values from the ontology, which resulted in tables
[ ] confirm whether a fact in ontology is true or not. Ask for a boolean query
[ ] provide a description about a specific information within the ontology
[ ] create new information based on the information available in the ontology
Modification Queries
[ ] insert new information to the ontology
[ ] update information within the ontology
[ ] delete information from the ontology
[ ] availability of the features to modify the query according to your need (e.g. ordering, distinct, sort, etc.)
Ontology Source
[ ] Ontology Source Modification Support ( i.e. ability to change the ontology that you are querying)
[ ] Multiple Source Support (i.e. ability to query from two or more ontology sources with one query)
[ ] Query Concatenation (i.e. ability to query from other querying result)
Result Visualization
[ ] result visualisation in triples / other raw format for ontology
[ ] result visualisation in tables / textual
[ ] result visualisation in graphical form
[ ] ability to filter or clarify the result using some of the results attribute, e.g. sort by time, filter out result before year 1970

2. Criteria Score by User Group

The result of the questionnaire could be accessed here.”


Sommerville Notes

This is what I got from reading and trying to summarize first chapter of Ian Sommerville’s book: Software Engineering – 9th Edition. Maybe later I could have more time to wrote the rest, as I never able to do that in my undergraduate years, 🙂
Software engineering: engineering discipline that is concerned with all aspect of software productions, ranging from software requirements to software testing.
Fundamental of software engineering: software specifications, development, validation, and software evolution

fundamental notions of software engineering that is universally applicable
1. Maintainability / easy to evolve
2. Dependability and security / robustness
3. Efficiency / resource-friendly
4. Acceptability / used happily by intended user

– Software engineering is important because in the long run it is usually cheaper than just create a software.
– Elegant theories of computer science cannot always applied to large, complex problems that require a software solution.
– “No Silver Bullet – Essence and Accident in Software Engineering” is really worth the read!

Software engineering fundamentals ideas that apply to all type of software systems:
1. Process / managed using agreed and understandable development process
2. Dependability and security / behave and available as expected, secure
3. Requirements and management / user expectations gathering and managing.
4. Reuse / effective

1. Public / make the public interest as their core
2. Client & employer / act on behalf of interest of client, should be inline with public
3. Product / highest standard possible
4. Judgement / not bias toward anything.
5. Management / leader should promote and lead by example toward this ethics
6. Profession / advance this profession integrity and reputation inline with public interest
7. Colleagues / fair and supportive
8. Self / self improving and promote ethics”

Systematic Literature Review (or Systematic Review) Introduction

This is my summary of chapter 1, 2, and 4 of the Guidelines for performing Systematic Review version 2.3, written in 2007 by Software Engineering Group from Keele University in UK.


Systematic Literature Review or Systematic Review is a method to identify, evaluate and interpret all available research document to a particular research question, topic area, or just a phenomenon of interest. Every single research that is used in a systematic review are called primary studies, while the systematic review itself is a form of secondary study.

Reasons to do this systematic review:
1. Summarise a particular technology
2. Identify gaps in research
3. Provide a framework / background to position new research activities
In addition, this could also examine to which empirical evidence support / contradict theoretical hypotheses, or even to assist to create new hypotheses.

Importance of systematic reviews is to identify all research which support and not supporting his/her own research, in order to get broader view while conducting our research.

Features of systematic review are:
* Starting by defining a review protocol that specifies the research question.
* Based on defined search strategy that aims to detect research as much as possible
* Document their literature search strategy so anyone could try out those
* Require inclusion and exclusion criteria to access potential primary study
* Specify information to be obtained from primary study, including quality criteria

Other than systematic review, they are two more type of review that could be conducted to support. When it is discovered that research in a topic area is unlikely to exist, or there are too broad, a systematic mapping studies could be more appropriate to identify a more suitable topic area. In other situation, when they are already several systematic review available for one topic, we could do a tertiary reviews, a systematic review of systematic reviews.

Systematic review processes are:
1. Planning the review
* Identify the need of review
* Specify the research question
* Define the research protocol

2. Conducting the review
* Identification of research
* Select the primary study
* Quality assessment for selecting study is defined
* Performed data extraction and monitoring
* Synthesised the obtained data

3. Reporting the review
* Specify the dissemination mechanism
* Presenting the review report

Must be noted that many stages involve iteration, e.g: selection of primary studies is governed by inclusion and exclusion criteria, which can be refined in later stage. “

OQT/OQUI Systematic Literature Review Results

We conducted an adapted version of the SLR process as we focused on the content of the key papers (see process steps below).

1. Define goal of the SLR. The goal of the SLR is (a) to collect candidates for relevant OQT classification criteria for an OQT analysis framework and (b) to identify the reports on OQTs, which are candidates for the tool evaluation. This SLR is an initial work, and will be further extended in the future work.

2. Scope of the SLR to identify and select primary studies. The scope of SLR sources for the primary studies should cover the conference proceedings and journals most relevant for the OQT scientific community. Therefore, we included the proceedings of six conferences and four journals that can be considered as most relevant in the semantic web field. The SLR covers 1204 papers from the last 5 years, i.e., papers published in 2008 to 2013 and fitting the following keywords: query OR sparql OR user OR interface OR visual OR interactive OR graphic. We are searching for primary studies in the selected conferences and journals manually from DBLP and the official website of conferences and journals in February 2013, which resulted in 78 papers.

3. Inclusion and exclusion criteria. In order to select the most relevant papers from the results of the previous step, we used the following inclusion and exclusion criteria. Inclusion Criteria: Full papers that provide an approach or survey or classification framework regarding ontology querying tools and/or relevant ontology tools (e.g., ontology visualization tools and ontology query user interface). Exclusion Criteria: Papers that do not introduce tools or interfaces related to ontology querying and/or relevant ontology tools. The process of applying these inclusion and exclusion criteria to the full papers reduced the number of papers finally to 24 papers, which were used for the extraction of candidate for criteria and tools (see Sections 7.1 and 8.1).

Here we provide the complete citation lists of the papers:

[1] P. Smart, A. Russell, and D. Braines, “A visual approach to semantic query design using a web-based graphical query designer,” Knowledge Engineering: Practice and Patterns, pp. 275–291, 2008.
[2] V. Tablan, D. Damljanovic, and K. Bontcheva, “A natural language query interface to structured information,” The Semantic Web: Research and Applications, pp. 361–375, 2008.
[3] H. Wang, K. Zhang, Q. Liu, T. Tran, and Y. Yu, “Q2semantic: A lightweight keyword interface to semantic search,” The Semantic Web: Research and Applications, pp. 584–598, 2008.
[4] D. F. Huynh, R. C. Miller, and D. R. Karger, “Potluck: Data mash-up tool for casual users,” Web Semantics: Science, Services and Agents on the World Wide Web, vol. 6, no. 4, pp. 274–282, 2008.
[5] A. Averbakh, D. Krause, and D. Skoutas, “Exploiting user feedback to improve semantic web service discovery,” The Semantic Web-ISWC 2009, pp. 33–48, 2009.
[6] D. Petrelli, S. Mazumdar, A. S. Dadzie, and F. Ciravegna, “Multi visualization and dynamic query for effective exploration of semantic data,” The Semantic Web-ISWC 2009, pp. 505–520, 2009.
[7] G. Zenz, X. Zhou, E. Minack, W. Siberski, and W. Nejdl, “From keywords to semantic queries—Incremental query construction on the Semantic Web,” Web Semantics: Science, Services and Agents on the World Wide Web, vol. 7, no. 3, pp. 166–176, 2009.
[8] S. Schenk, C. Saathoff, S. Staab, and A. Scherp, “SemaPlorer—interactive semantic exploration of data and media based on a federated cloud infrastructure,” Web Semantics: Science, Services and Agents on the World Wide Web, vol. 7, no. 4, pp. 298–304, 2009.
[9] S. Batsakis and E. Petrakis, “SOWL: spatio-temporal representation, reasoning and querying over the semantic web,” in I-SEMANTICS ’10 Proceedings of the 6th International Conference on Semantic Systems, 2010, p. 15.
[10] H. Paulheim and F. Probst, “Ontology-enhanced user interfaces: A survey,” International Journal on Semantic Web and Information Systems (IJSWIS), vol. 6, no. 2, pp. 36–59, 2010.
[11] G. Ladwig and T. Tran, “Combining query translation with query answering for efficient keyword search,” The Semantic Web: Research and Applications, pp. 288–303, 2010.
[12] S. Araujo, G. J. Houben, D. Schwabe, and J. Hidders, “Fusion–Visually Exploring and Eliciting Relationships in Linked Data,” The Semantic Web: Research and Applications, pp. 1–15, 2010.
[13] P. Heim, S. Lohmann, and T. Stegemann, “Interactive relationship discovery via the semantic web,” The Semantic Web: Research and Applications, pp. 303–317, 2010.
[14] D. Damljanovic, M. Agatonovic, and H. Cunningham, “Natural language interfaces to ontologies: Combining syntactic analysis and ontology-based lookup through the user interaction,” The Semantic Web: Research and Applications, pp. 106–120, 2010.
[15] T. Tran, T. Mathäß, and P. Haase, “Usability of keyword-driven schema-agnostic search,” The Semantic Web: Research and Applications, pp. 349–364, 2010.
[16] A. Harth, “VisiNav: A system for visual search and navigation on web data,” Web Semantics: Science, Services and Agents on the World Wide Web, vol. 8, no. 4, pp. 348–354, 2010.
[17] A. S. Dadzie and M. Rowe, “Approaches to visualising Linked Data: A survey,” Semantic Web, vol. 2, no. 2, pp. 89–124, 2011.
[18] E. Motta, P. Mulholland, S. Peroni, M. d’Aquin, J. Gomez-Perez, V. Mendez, and F. Zablith, “A novel approach to visualizing and navigating ontologies,” The Semantic Web–ISWC 2011, pp. 470–486, 2011.
[19] I. Popov, M. Schraefel, W. Hall, and N. Shadbolt, “Connecting the dots: a multi-pivot approach to data exploration,” The Semantic Web–ISWC 2011, pp. 553–568, 2011.
[20] D. Koutsomitropoulos, R. Borillo Domenech, and G. Solomou, “A structured semantic query interface for reasoning-based search and retrieval,” The Semantic Web: Research and Applications, pp. 17–31, 2011.
[21] J. Lehmann and L. Bühmann, “AutoSPARQL: Let users query your knowledge base,” The Semantic Web: Research and Applications, pp. 63–79, 2011.
[22] E. Kaufmann and A. Bernstein, “Evaluating the Usability of Natural Language Query Languages and Interfaces to Semantic Web Knowledge Bases,” Web Semantics: Science, Services and Agents on the World Wide Web, vol. 8, no. 4, 2011.
[23] C. Pradel, “Allowing End Users to Query Graph-Based Knowledge Bases,” Knowledge Engineering and Knowledge Management, pp. 8–15, 2012.
[24] V. Lopez, M. Fernández, E. Motta, and N. Stieler, “PowerAqua: Supporting users in querying and exploring the Semantic Web,” Semantic Web, vol. 3, no. 3, pp. 249–265, 2012.