Mining and Modelling Unstructured Data

Artifacts containing natural language, like Q&A websites (e.g., Stack Overflow), tutorials, and development emails, are essential to support software development. They have become a popular subject for software engineering research. The analysis of such artifacts is particularly challenging because of their heterogeneity: These resources consist of natural language interleaved with fragments of multiple programming and markup languages. Our tutorial is aimed at overcoming the challenge, by first discussing the state of the art of methodologies to analyze unstructured data, and their current limitations and challenges. Then, it focuses on our efforts towards a systematic approach to model contents of such artifacts. This in turn enables novel holistic analyses that fully exploit their intrinsic heterogeneous nature. We describe the theoretical foundations of our StORMeD framework, how it can be used to extract a full-fledged model of a development artifacts, and how it can be leveraged to construct various types of analyses, such as summarization.

Outline

Part 1: Theoretical Session

1. Context: We discuss the structure of heterogeneous artifacts and their importance for SE research.
2. State of the Art: We illustrate the main approaches in literature highlighting their limitations and challenges.
3. Technical Concepts: We introduce StORMeD and its technical concepts, including the grammar implementation and how it models artifacts through a heterogeneous abstract syntax tree (H-AST).

Part 2: Hands-on Session

1. StORMeD service and API Usage: We introduce the practical usage of the service and its API to parse and model artifacts, including the analysis of Stack Overflow discussions.
2. Application to Summarization: We present the implementation of a practical application to summarize Stack Overflow discussions using a holistic similarity metric.
3. Wrap-up: We conclude the tutorial by discussing other potential applications and ideas where our approach can be beneficial.

Targeted Audience

Our tutorial targets academics that consider heterogeneous development artifacts as their target subject for any kind of study. In particular, any analysis that would benefit from a holistic approach - i.e., an approach that considers the intrinsic heterogeneous nature of the artifacts - can potentially benefit from our approach. Thus, we believe that researchers in the mining software repositories community, the program comprehension community, and in general software maintenance and evolution, constitute the best match as a target audience for this tutorial.

We expect the audience to possess basic knowledge about parsing and grammars. We plan to use the Scala programming language for the tutorial, but without using any advanced feature: We expect the audience to be experienced about object-oriented programming and basic functional programming, like lambda expressions and simple higher-order functions (e.g., map, reduce).