Automatic Classification of Development Artifact Contents

Software developers often make use of development artifacts (e.g., StackOverflow posts, software documentation) to obtain information related to the problem they are trying to solve. The content of these resources is heterogeneous as it usually combines source code and other structured information (e.g., interchange formats, stack traces, log outputs) with natural language. Furthermore, the snippets of code in these resources may pertain to multiple programming languages in a single artifact, and such snippets may even be incomplete or erroneous. Extracting the contents from these resources has become a fundamental task for many applications in software analytics and recommendation systems for software engineering (RSSEs). Such tools can help developers in navigating large numbers of artifacts, thus improving their information-gathering efficiency.

To simplify this process, there exist numerous approaches to extract, model and index this information, based on methods such as island parsing, regular expressions, or machine learning. However, they often are limited to single languages, families of languages, or in some cases they require to manually define complex grammars. In addition, snippets that include partial or syntactically-incorrect code may not be correctly processed.

This thesis aims to devise an approach to automatically extract and classify the different types of contents of development artifacts, by harnessing several methods based on deep learning, commonly used for natural language processing tasks. We explore multiple approaches for this task over two scopes: paragraph/snippet-level classification and token-level classification.

First, we aim to classify the language of entire pieces of text, which can be either snippets of code, natural language paragraphs, or a mix of both. This approach is implemented using a convolutional neural network. The overall results are good, achieving 94% classification accuracy on a curated dataset of snippets from StackOverflow posts. However, the model struggles when the content is a mix of natural language and code.

To address such limitation, we further investigate a different approach based on code element identification. With a method similar to part-of-speech tagging, built on a Bi-LSTM neural network, we classify individual tokens in a given fragment of text (natural language, code or a mix of both) with a label that identifies their type. Thus, we can identify code fragments that are embedded in natural language paragraphs or vice-versa. Starting from an existing work that we found had a few issues in the corpus construction process, we improve on it by generating the labeled ground-truth using an island parser, which processes a curated StackOverflow corpus. With this approach we can perform a fine-grained classification of mixed text with improved results (with respect to the state-of-the-art) according to both a quantitative (with 96.7% tagging accuracy) and qualitative evaluation.

Our work shows that we can obtain accurate results using common deep learning methods, with minimal fine-tuning, to classify the contents of development artifacts. While our implementation has several limitations, namely the limited amount of supported programming languages and interchange formats, we claim that this will serve as a starting point to more generalized classification approaches.