Venue: IPSJ/SIGSE Software Engineering Symposium 2017
Date: 31 August 2017
Parsing and Modeling Software Engineering Artifacts
Presented in the Special International Track on Software Analytics, chaired by Shinpei Hayashi (Tokyo Institute of Technology), at the Waseda University Green Computing Systems Research and Development Center.
Abstract
Artifacts containing natural language, like development emails, tutorials, and Q&A websites (e.g., Stack Overflow), are essential in the practice of software development, and thus they have become a popular subject for software engineering research.
The analysis of such artifacts is particularly challenging because of their heterogeneity: These resources consist of natural language interleaved with fragments of multiple programming and markup languages. Moreover, often these languages are not only interleaved, but included in other ones, like XML fragments in Java strings, or method references in JSON values.
The tutorial is aimed at overcoming this challenge by first discussing the state of the art of methodologies to analyze unstructured data, and their current limitations and shortcomings. Then, it focuses on our efforts towards a systematic approach to model contents of such artifacts. This in turn enables novel holistic analyses that fully exploit their intrinsic heterogeneous nature. We describe the theoretical foundations of our StORMeD framework, how it can be used to extract a full-fledged model of a development artifacts, and how it can be leveraged to construct various types of analyses, such as summarization, the analysis of video tutorial fragments, and the construction of a holistic recommender system.
Related Content
4 September 2016 Tutorial
Mining and Modelling Unstructured Data
31st IEEE/ACM International Conference on Automated Software Engineering (ASE 2016)
16 May 2015 Paper
StORMeD: Stack Overflow Ready Made Data
Stack Overflow is the de facto Question and Answer (Q&A) website for developers, and it has been used in many approaches by software engineering researchers to mine useful data. However, the contents of a Stack Overflow discussion are inherently heterogeneous, mixing natural language, source code, stack traces and configuration files in XML or JSON format. We constructed a full island grammar capable of modeling the set of 700,000 Stack Overflow discussions talking about Java, building a heterogeneous abstract syntax tree (H-AST) of each post (question, answer or comment) in a discussion. The resulting dataset models every Stack Overflow discussion, providing a full H-AST for each type of structured fragment (i.e., JSON, XML, Java, Stack traces), and complementing this information with a set of basic meta-information like term frequency to enable natural language analyses. Our dataset allows the end-user to perform combined analyses of the Stack Overflow by visiting the H-AST of a discussion.