Towards AI-native software engineering (SE 3.0): A vision and a challenge roadmap
JournalACM Transactions on Software Engineering and Methodology 35 (9) · 2026
The rise of AI-assisted software engineering (SE 2.0), powered by Foundation Models (FMs) and FM-powered coding assistants, has shown promise in improving developer productivity. However, it has also exposed inherent limitations, such as cognitive overload on developers and inefficiencies. We propose a shift towards Software Engineering 3.0 (SE 3.0), an AI-native approach characterized by intent-centric, conversation-oriented development between human developers and AI teammates. SE 3.0 envisions AI systems evolving beyond task-driven copilots into intelligent collaborators, capable of deeply understanding and reasoning about software engineering principles and intents. We outline the key components of the SE 3.0 technology stack, which includes Teammate.next for adaptive and personalized AI partnership, IDE.next for intent-centric conversation-oriented development, Compiler.next for multi…
Citations by year 67 total
2 2024 21 2025 44 2026ACM
Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States
PreprintarXiv preprint arXiv:2602.03708 · 2026
Large Language Models (LLMs) achieve strong performance across many tasks but suffer from high inference latency due to autoregressive decoding. The issue is exacerbated in Large Reasoning Models (LRMs), which generate lengthy chains of thought. While speculative decoding accelerates inference by drafting and verifying multiple tokens in parallel, existing methods operate at the token level and ignore semantic equivalence (i.e., different token sequences expressing the same meaning), leading to inefficient rejections. We propose SemanticSpec, a semantic-aware speculative decoding framework that verifies entire semantic sequences instead of tokens. SemanticSpec introduces a semantic probability estimation mechanism that probes the model's internal hidden states to assess the likelihood of generating sequences with specific meanings. Experiments on four benchmarks show that SemanticSpec achieves up to 2.7x speedup on DeepSeekR1-32B and 2.1x on QwQ-32B, consistently outperforming token-level and sequence-level baselines in both efficiency and effectiveness.
Citations by year 4 total
4 2026Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment
PreprintarXiv preprint arXiv:2606.14948 · 2026
LLMs have substantially improved software engineering yet real-world development requires architectural understanding. Such understanding is prohibitively expensive to label manually and impossible to verify through tests alone. We propose an agentic judging pipeline using a strong LLM as a scalable proxy for expert architectural evaluation, comprising two judges: the Architecture Complexity Judge (ACJ), which estimates codebase-specific architectural understanding a task demands, and the Architecture Quality Judge (AQJ), which evaluates patch conformance to repository-specific architectural conventions via source-grounded rubrics. Fine-tuning Qwen3-8B/14B/32B on 3,360 curated instances achieves resolved rates of up to 27.2% on SWE-bench Verified – up to 540% over the base model and 256% over unfiltered fine-tuning. Meanwhile, the trained models achieve strong cross-language generalization and consistent improvements in architectural patch quality.
When Elo Lies: Hidden Biases in Codeforces-Based Evaluation of Large Language Models
PreprintarXiv preprint arXiv:2602.05891 · 2026
As Large Language Models (LLMs) achieve breakthroughs in complex reasoning, Codeforces-based Elo ratings have emerged as a prominent metric for evaluating competitive programming capabilities. However, these ratings are often reported without critical experimental details, leading to significant discrepancies illustrated by recent reports where the score of the same model version fluctuated by nearly 500 points. This paper presents a systematic empirical study on the hidden factors biasing Elo evaluations: (1) the temporal ordering of submissions, (2) contest difficulty selection, and (3) run to run stochastic variability of LLMs. Utilizing a controlled benchmark of 37 recent Codeforces contests and 13,691 generated test cases, we demonstrate that Elo scores are highly sensitive to these parameters. Our findings reveal that varying submission orders can shift scores by 394 points, while contest selection can cause differences of up to 1,122 points for the same model. Run to run performance exhibits substantial instability, with a maximum difference of 349 points in mean scores observed when evaluating identical contests. We conclude that direct Elo comparisons are unreliable and potentially misleading without strict standardization and transparent reporting of experimental settings.
LoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding Agent
PreprintarXiv preprint arXiv:2608.00267 · 2026
Coding agent infrastructure is shifting from harness engineering toward loop engineering as coding agents are deployed for sustained long-horizon software development. Existing benchmarks often center on localized tasks or end-state outcomes, offering limited insight into sustained execution. We introduce LOOPSBENCH, a long-horizon benchmark for loop engineering in coding agent evaluation. Each task is a dependency DAG over separately testable development units with source-evidenced prerequisite edges. LOOPSBENCH comprises 112 tasks from authentic sources spanning 8 programming languages and 9 domains. Its flow-aware runtime releases tests along the ready frontier and retains completed nodes as regression obligations. We evaluate frontier coding agents paired with widely used loop implementations. The strongest configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks. Recorded plans recover only part of the source-recovered prerequisite DAG, and regression events remain visible across the evaluated loop profiles. We open source the benchmark data and code, including all tasks, more than 5,300 development units, and executable tests, at microsoft/Loopsbench.
Citations by year 1 total
1 2026Artificial intelligence (ai) systems using layered foundation models with real-time adapting routing, and apparatuses, methods, and non-transitory computer-readable storage …
PatentUS Patent 19329992 · 2026
A computerized method for obtaining a foundation model (FM) output for a first request, the method includes: searching, in a first storage, for a second request similar to the first request; performing a first action if the second request is found and is suitable for processing by a first FM; and performing a second action if the second request is not found; wherein the first action includes: using the first FM to obtain the FM output; and wherein the second action includes: using the first FM to obtain a first output for the first request, using a second FM to obtain a second output as the FM output, the second FM having a greater model size and/or capability than the first FM, and storing the first request in the first storage as suitable for processing by the first FM if the first output is similar to the second output.
Artificial intelligence (ai) systems with cognitive observability for reasoning of foundation model-powered agents, and apparatuses, methods, and non-transitory computer …
PatentUS Patent 19328336 · 2026
A computerized method for generating reasoning of a first agent of a foundation model (FM). The computerized method has the step of: using a second agent to replicate a first completion generated by the first agent corresponding to a prompt while reasoning a thought process of the first agent in generating the first completion from the prompt, for generating the reasoning of the first agent.
Hierarchical dynamic planning of foundation model agents
PatentUS Patent 18894449 · 2026
Methods and system are disclosed for human-FM collaboration. The method includes acquiring an initial requirement in natural language, generating using a first FM-based agent a plan indicative of tasks and skills for achieving an objective, iteratively generating using the first FM-based agent, adjusted versions of the plan. During a given iteration, the method comprises verifying using a second FM-based agent that a current version of the plan matches the initial requirement. The method comprises compiling a latest version of the plan in an executable graph format, and providing the compiled plan for execution by a third FM-based agent. Methods and systems for plan execution are also disclosed. The method includes selecting using a fourth FM-based agent an existing agent, dynamically and automatically generating a new agent, selecting an architecture for communication between the agents, and executing…
LLMSafeGuard: A Training-Free Framework for Safeguarding LLM Decoding via Context-Wise Similarity Validation
ConferenceProceedings of the 34th ACM International Conference on the Foundations of Software Engineering · 2026
Large Language Models (LLMs) have significantly advanced natural language processing (NLP) tasks, but also pose ethical and societal risks due to their propensity to generate harmful content. Existing methods have limitations, including the need for training specific control models and proactive intervention during text generation, which leads to quality degradation and increased computational overhead. To mitigate those limitations, we propose LLMSafeGuard, a lightweight real-time framework that integrates an external validator into decoding, rejecting unsafe outputs while allowing valid ones. We introduce a similarity-based validation approach, simplifying safety constraint validation and eliminating the need for external control model training. Additionally, LLMSafeGuard employs a context-wise timing selection strategy, which allows LLMSafeGuard to intervene in LLMs only when necessary. We evaluate…
Developing FMware includes managing a variety of assets that extend beyond traditional software development. These assets include not only the foundation models themselves but also resources such as datasets, prompts, and agents. Consequently, specialized management methods for the assets may be provided.
Methods and systems for augmenting synthetic data labels with human-generated data labels
PatentUS Patent 18916095 · 2026
There are provided methods and systems for generating a training dataset for use in supervised learning, by augmenting synthetic data labels with human-generated data labels. A preliminary training dataset corresponding to a first set of instructions is generated by a large language model (LLM). A contribution score for the preliminary training dataset is generated. Responsive to determining that the contribution score is below a threshold, human-generated data is obtained, based on a second set of instructions. An updated contribution score may be generated, based on the preliminary training dataset and the human-generated data. Responsive to determining that the updated contribution score meets or is above the threshold, the training dataset may be generated based on the preliminary training dataset and the human-generated data. The disclosed methods and systems may enable robust and efficient…
Agentic Software Engineering: Foundational Pillars and a Research Roadmap
PreprintarXiv preprint arXiv:2509.06216 · 2025
Agentic Software Engineering (SE 3.0) represents a new era where intelligent agents are tasked not with simple code generation, but with achieving complex, goal-oriented SE objectives. To harness these new capabilities while ensuring trustworthiness, we must recognize a fundamental duality within the SE field in the Agentic SE era, comprising two symbiotic modalities: SE for Humans and SE for Agents. This duality demands a radical reimagining of the foundational pillars of SE (actors, processes, tools, and artifacts) which manifest differently across each modality. We propose two purpose-built workbenches to support this vision. The Agent Command Environment (ACE) serves as a command center where humans orchestrate and mentor agent teams, handling outputs such as Merge-Readiness Packs (MRPs) and Consultation Request Packs (CRPs). The Agent Execution Environment (AEE) is a digital workspace where agents perform tasks while invoking human expertise when facing ambiguity or complex trade-offs. This bi-directional partnership, which supports agent-initiated human callbacks and handovers, gives rise to new, structured engineering activities (i.e., processes) that redefine human-AI collaboration, elevating the practice from agentic coding to true agentic software engineering. This paper presents the Structured Agentic Software Engineering (SASE) vision, outlining several of the foundational pillars for the future of SE. The paper culminates in a research roadmap that identifies a few key challenges and opportunities while briefly discussing the resulting impact of this future on SE education. Our goal is not to offer a…
Citations by year 66 total
1 2024 3 2025 62 2026Swe-effi: Re-evaluating software ai agent system effectiveness under resource constraints
PreprintarXiv preprint arXiv:2509.09853 · 2025
The advancement of large language models (LLMs) and code agents has demonstrated significant potential to assist software engineering (SWE) tasks, such as autonomous issue resolution and feature addition. Existing AI for software engineering leaderboards (e.g., SWE-bench) focus solely on solution accuracy, ignoring the crucial factor of effectiveness in a resource-constrained world. This is a universal problem that also exists beyond software engineering tasks: any AI system should be more than correct – it must also be cost-effective. To address this gap, we introduce SWE-Effi, a set of new metrics to re-evaluate AI systems in terms of holistic effectiveness scores. We define effectiveness as the balance between the accuracy of outcome (e.g., issue resolve rate) and the resources consumed (e.g., token and time). In this paper, we specifically focus on the software engineering scenario by re-ranking popular AI systems for issue resolution on a subset of the SWE-bench benchmark using our new multi-dimensional metrics. We found that AI system's effectiveness depends not just on the scaffold itself, but on how well it integrates with the base model, which is key to achieving strong performance in a resource-efficient manner. We also identified systematic challenges such as the "token snowball" effect and, more significantly, a pattern of "expensive failures". In these cases, agents consume excessive resources while stuck on unsolvable tasks – an issue that not only limits practical deployment but also drives up the cost of failed rollouts during RL training. Lastly, we observed a clear trade-off between effectiveness under the token budget and…
Citations by year 35 total
5 2025 30 2026Watson: A cognitive observability framework for the reasoning of LLM-powered agents
Conference2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE)
Large language models (LLMs) are increasingly integrated into autonomous systems, giving rise to a new class of software known as Agentware, where LLM-powered agents perform complex, open-ended tasks in domains such as software engineering, customer service, and data analysis. However, their high autonomy and opaque reasoning processes pose significant challenges for traditional software observability methods. To address this, we introduce the concept of cognitive observability—the ability to recover and inspect the implicit reasoning behind agent decisions. We present Watson, a general-purpose framework for observing the reasoning processes of fast-thinking LLM agents without altering their behavior. Watson retroactively infers reasoning traces using prompt attribution techniques. We evaluate Watson in both manual debugging and automated correction scenarios across the MMLU benchmark…
Citations by year 22 total
1 2024 6 2025 15 2026IEEE
Spice: An automated swe-bench labeling pipeline for issue clarity, test coverage, and effort estimation
Conference2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE)
High-quality labeled datasets are crucial for training and evaluating foundation models in software engineering, but creating them is often prohibitively expensive and labor-intensive. We introduce SPICE, a scalable, automated pipeline for labeling SWE-bench-style datasets with annotations for issue clarity, test coverage, and effort estimation. SPICE combines context-aware code navigation, rationale-driven prompting, and multi-pass consensus to produce labels that closely approximate expert annotations. SPICE’s design was informed by our own experience and frustration in labeling more than 800 instances from SWE-Gym. SPICE achieves strong agreement with human-labeled SWE-bench Verified data while reducing the cost of labeling 1,000 instances from around $100,000 (manual annotation) to only $5.10. These results demonstrate SPICE’s potential to enable cost-effective, large-scale dataset creation…
Citations by year 14 total
3 2025 11 2026IEEE
Promptexp: Multi-granularity prompt explanation of large language models
Conference2025 2nd IEEE/ACM International Conference on AI-powered Software (AIware)
Large Language Models (LLMs) excel in tasks like natural language understanding and text generation. Prompt engineering plays a critical role in leveraging LLM effectively. However, LLM's black-box nature hinders its interpretability and effective prompt engineering. A wide range of model explanation approaches have been developed for deep learning models (e.g., feature attribution-based and attention-based techniques). However, these local explanations are designed for single-output tasks like classification and regression, and cannot be directly applied to LLMs, which generate sequences of tokens. Recent efforts in LLM explanation focus on natural language explanations, but they are prone to hallucinations and inaccuracies. To address this, we introduce PromptExp, a framework for multi-granularity prompt explanations by aggregating tokenlevel insights. PromptExp introduces two token-level…
Citations by year 10 total
6 2025 4 2026IEEE
Metasel: A test selection approach for fine-tuned dnn models
JournalIEEE Transactions on Software Engineering · 2025
Deep Neural Networks (DNNs) face challenges during deployment due to covariate shift, i.e., data distribution shifts between development and deployment contexts. Fine-tuning adapts pre-trained models to new contexts requiring smaller labeled sets. However, testing fine-tuned models under constrained labeling budgets remains a critical challenge. This paper introduces MetaSel, a new approach tailored for DNN models that have been fine-tuned to address covariate shift, to select tests from unlabeled inputs. MetaSel assumes that fine-tuned and pre-trained models share related data distributions and exhibit similar behaviors for many inputs. However, their behaviors diverge within the input subspace where fine-tuning alters decision boundaries, making those inputs more prone to misclassification. Unlike general approaches that rely solely on the DNN model and its input set, MetaSel leverages information…
Citations by year 6 total
6 2026IEEE
Model performance-guided evaluation data selection for effective prompt optimization
ConferenceFindings of the Association for Computational Linguistics: ACL 2025
Optimizing Large Language Model (LLM) performance requires well-crafted prompts, but manual prompt engineering is labor-intensive and often ineffective. Automated prompt optimization techniques address this challenge but the major of them rely on randomly selected evaluation subsets, which fail to represent the full dataset, leading to unreliable evaluations and suboptimal prompts. Existing coreset selection methods, designed for LLM benchmarking, are unsuitable for prompt optimization due to challenges in clustering similar samples, high data collection costs, and the unavailability of performance data for new or private datasets. To overcome these issues, we propose IPOMP, an Iterative evaluation data selection approach for effective Prompt Optimization using real time Model Performance. IPOMP is a two-stage approach that selects representative and diverse samples using semantic clustering and boundary analysis, followed by iterative refinement with real-time model performance data to replace redundant samples. Evaluations on two datasets BIG-bench and LIAR, and two models GPT-3.5 and GPT-4o-mini, show that IPOMP improves effectiveness by at least 1.6% to 3.1%, and stability by at least 50% to 55.5% compared with the best baseline across the studied datasets and models, with minimal computational overhead below 1%. Furthermore, the results demonstrate that our real-time performance-guided refinement approach can be universally applied to enhance existing coreset selection methods.
Citations by year 7 total
2 2025 5 2026Repoforge: Training a sota fast-thinking swe agent with an end-to-end data curation pipeline synergizing sft and rl at scale
PreprintarXiv preprint arXiv:2508.01550 · 2025
Training software engineering (SWE) LLMs is bottlenecked by expensive infrastructure, inefficient evaluation pipelines, scarce training data, and costly quality control. We present RepoForge, an autonomous, end-to-end pipeline that generates, evaluates, and trains SWE agents at scale. Our key contributions include: (1) RepoForge-8B-Agent, achieving 17.4\% on SWE-Bench-Verified~\citep{swebench_verified2024}, establishing new state-of-the-art for 8B non-thinking LLMs; (2) 7,304 executable environments auto-generated from real GitHub commits with zero manual intervention; (3) 14 storage reduction (1.4GB 102MB per instance) via intelligent dependency management and image pruning; (4) 70\% faster evaluation using a Ray-powered~\citep{ray2018} distributed RepoForge harness; (5) 19,000 cheaper labeling through our automated SPICE~\citep{spice2024} difficulty assessment technique. By unifying storage-efficient sandboxing, Ray-powered evaluation harness, automated data generation, SPICE-based labeling, and bubble-free RL scaffold, we demonstrate that even 8B models can reach new state-of-the-art performance on demanding benchmarks like SWE-Bench-Verified. Our approach addresses critical bottlenecks in SWE agent training: high storage costs of container-based evaluation, inefficient sequential reward pipelines, limited availability of high-quality training data, expensive manual labeling, and multi-turn RL pipeline bottlenecks.
Citations by year 7 total
3 2025 4 2026Simclone: detecting tabular data clones using value similarity
JournalACM Transactions on Software Engineering and Methodology 34 (1) · 2025
Data clones are defined as multiple copies of the same data among datasets. The presence of data clones between datasets can cause issues such as difficulties in managing data assets and data license violations when using datasets with clones to build AI software. However, detecting data clones is not trivial. The majority of the prior studies in this area rely on structural information to detect data clones (e.g., font size, column header). However, tabular datasets used to build AI software are typically stored without any structural information. In this article, we propose a novel method called SimClone for data clone detection in tabular datasets without relying on structural information. SimClone method utilizes value similarities for data clone detection. We also propose a visualization approach as a part of our SimClone method to help locate the exact position of the cloned data between a dataset pair. Our results…
Citations by year 6 total
1 2024 2 2025 3 2026ACM
A Framework and Taxonomy for Characterizing the Applicability of Software Architecture Recovery Approaches: A Tertiary‐Mapping Study
JournalSoftware: Practice and Experience 55 (1) · 2025
Software architecture assists developers in addressing non‐functional requirements and in maintaining, debugging, and upgrading their software systems. Consequently, consistency between the designed architecture and the implemented software system itself is important; without this consistency the non‐functional requirements targeted may not be addressed and architectural documentation may mis‐direct maintenance efforts that target the associated code‐base. But often, when software is initially implemented or subsequently evolved, the designed architecture and software architecture become inconsistent, with the implemented structure degraded due to issues like developer time‐pressures, or ambiguous communication of the designed architecture. In such cases, Software Architecture Recovery (SAR) or consistency approaches can be applied to reconstruct the architecture of the software system and…
Citations by year 5 total
4 2025 1 2026Towards Conversational Development Environments: Using Theory-of-Mind and Multi-Agent Architectures for Requirements Refinement
PreprintarXiv preprint arXiv:2505.20973 · 2025
Foundation Models (FMs) have shown remarkable capabilities in various natural language tasks. However, their ability to accurately capture stakeholder requirements remains a significant challenge for using FMs for software development. This paper introduces a novel approach that leverages an FM-powered multi-agent system called AlignMind to address this issue. By having a cognitive architecture that enhances FMs with Theory-of-Mind capabilities, our approach considers the mental states and perspectives of software makers. This allows our solution to iteratively clarify the beliefs, desires, and intentions of stakeholders, translating these into a set of refined requirements and a corresponding actionable natural language workflow in the often-overlooked requirements refinement phase of software engineering, which is crucial after initial elicitation. Through a multifaceted evaluation covering 150 diverse use cases, we demonstrate that our approach can accurately capture the intents and requirements of stakeholders, articulating them as both specifications and a step-by-step plan of action. Our findings suggest that the potential for significant improvements in the software development process justifies these investments. Our work lays the groundwork for future innovation in building intent-first development environments, where software makers can seamlessly collaborate with AIs to create software that truly meets their needs.
Citations by year 5 total
3 2025 2 2026A Tutorial on Software Engineering for FMware
ConferenceProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering · 2025
Foundation Models (FMs) like GPT-4 have given rise to FMware, FM-powered applications representing a new generation of software that is developed under new paradigms. FMware has been widely adopted in both software engineering (SE) research (e.g., test generation) and industrial products (e.g., GitHub copilot), despite the numerous challenges introduced by the stochastic nature of FMs. In our tutorial, we will present the latest research and industry practices in engineering FMware. Our tutorial's perspective is firmly rooted in SE rather than artificial intelligence (AI), ensuring its accessibility to FSE participants.
Citations by year 3 total
2 2025 1 2026Real-time Adapting Routing (RAR): Improving Efficiency Through Continuous Learning in Software Powered by Layered Foundation Models
Conference2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)
To balance the quality and inference cost of a Foundation Model (FM, such as large language models) powered software, people often opt to train a routing model that routes requests to FMs with different sizes and capabilities. Existing routing models rely on learning the optimal routing decision from carefully curated data, require complex computations to be updated, and do not consider the potential evolution of weaker FMs. In this paper, we propose Real-time Adaptive Routing (RAR), an approach to continuously adapt FM routing decisions while using guided in-context learning to enhance the capabilities of weaker FM. The goal is to reduce reliance on stronger, more expensive FMs. We evaluate our approach on different subsets of the popular MMLU benchmark. Over time, our approach routes 50.2 % fewer requests to computationally expensive models while maintaining around 90.5 % of the general…
Citations by year 3 total
1 2025 2 2026IEEE
The hitchhikers guide to production-ready trustworthy foundation model powered software (fmware)
ConferenceProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 · 2025
Foundation Models (FMs) such as Large Language Models (LLMs) are reshaping the software industry by enabling FMware, systems that integrate these FMs as core components. In this KDD 2025 tutorial, we present a comprehensive exploration of FMware that combines a curated catalogue of challenges with real-world production concerns. We first discuss the state of research and practice in building FMware. We further examine the difficulties in selecting suitable models, aligning high-quality domain-specific data, engineering robust prompts, and orchestrating autonomous agents. We then address the complex journey from impressive demos to production-ready systems by outlining issues in system testing, optimization, deployment, and integration with legacy software. Drawing on our industrial experience and recent research in the area, we provide actionable insights and a technology roadmap for…
Citations by year 1 total
1 2025Towards Reliable Generation of Executable Workflows by Foundation Models
JournalACM Transactions on Software Engineering and Methodology · 2025
Recent advancements in Foundation Models (FMs) have demonstrated significant progress in processing complex natural language to perform intricate tasks. Successfully executing these tasks often requires orchestrating calls to FMs alongside other software components. However, manually decomposing a task into a coherent sequence of smaller, logically aggregated steps, commonly referred to as workflows, demands considerable effort and specialized domain knowledge. While FMs can assist in generating such workflows specified in domain-specific languages (DSLs), achieving accuracy and reliability in this process remains a challenge. We introduce a framework that leverages static analysis feedback to enable FMs to detect and repair defects in the DSL-based workflows they generate. We begin by presenting an initial taxonomy of incidences of defects in FM-generated DSL workflows, categorizing them into…
ACM
Keeping deep learning models in check: A history-based approach to mitigate overfitting
JournalIEEE Access 12 · 2024
In software engineering, deep learning models are increasingly deployed for critical tasks such as bug detection and code review. However, overfitting remains a challenge that affects the quality, reliability, and trustworthiness of software systems that utilize deep learning models. Overfitting can be (1) prevented (e.g., using dropout or early stopping) or (2) detected in a trained model (e.g., using correlation-based approaches). Both overfitting detection and prevention approaches that are currently used have constraints (e.g., requiring modification of the model structure, and high computing resources). In this paper, we propose a simple, yet powerful approach that can both detect and prevent overfitting based on the training history (i.e., validation losses). Our approach first trains a time series classifier on training histories of overfit models. This classifier is then used to detect if a trained model is overfit. In addition, our…
Citations by year 199 total
1 2005 3 2006 1 2010 2 2012 2 2013 2 2014 4 2015 3 2016 5 2017 1 2018 2 2019 5 2020 2 2021 4 2022 6 2023 20 2024 87 2025 49 2026IEEE
Rethinking software engineering in the era of foundation models: A curated catalogue of challenges in the development of trustworthy fmware
ConferenceCompanion proceedings of the 32nd ACM international conference on the foundations of software engineering · 2024
Foundation models (FMs), such as Large Language Models (LLMs), have revolutionized software development by enabling new use cases and business models. We refer to software built using FMs as FMware. The unique properties of FMware (e.g., prompts, agents and the need for orchestration), coupled with the intrinsic limitations of FMs (e.g., hallucination) lead to a completely new set of software engineering challenges. Based on our industrial experience, we identified ten key SE4FMware challenges that have caused enterprise FMware development to be unproductive, costly, and risky. For each of those challenges, we state the path for innovation that we envision. We hope that the disclosure of the challenges will not only raise awareness but also promote deeper and further discussions, knowledge sharing, and innovative solutions.
Citations by year 81 total
14 2024 38 2025 29 2026Rethinking software engineering in the foundation model era: From task-driven ai copilots to goal-driven ai pair programmers
PreprintarXiv preprint arXiv:2404.10225 · 2024
The advent of Foundation Models (FMs) and AI-powered copilots has transformed the landscape of software development, offering unprecedented code completion capabilities and enhancing developer productivity. However, the current task-driven nature of these copilots falls short in addressing the broader goals and complexities inherent in software engineering (SE). In this paper, we propose a paradigm shift towards goal-driven AI-powered pair programmers that collaborate with human developers in a more holistic and context-aware manner. We envision AI pair programmers that are goal-driven, human partners, SE-aware, and self-learning. These AI partners engage in iterative, conversation-driven development processes, aligning closely with human goals and facilitating informed decision-making. We discuss the desired attributes of such AI pair programmers and outline key challenges that must be addressed to realize this vision. Ultimately, our work represents a shift from AI-augmented SE to AI-transformed SE by replacing code completion with a collaborative partnership between humans and AI that enhances both productivity and software quality.
Citations by year 35 total
5 2024 21 2025 9 2026A framework for real-time safeguarding the text generation of large language
PreprintarXiv preprint arXiv:2404.19048 · 2024
Citations by year 13 total
2 2024 9 2025 2 2026Teasma: A practical methodology for test adequacy assessment of deep neural networks
JournalIEEE Transactions on Software Engineering 50 (12) · 2024
Successful deployment of Deep Neural Networks (DNNs), particularly in safety-critical systems, requires their validation with an adequate test set to ensure a sufficient degree of confidence in test outcomes. Although well-established test adequacy assessment techniques from traditional software, such as mutation analysis and coverage criteria, have been adapted to DNNs in recent years, we still need to investigate their application within a comprehensive methodology for accurately predicting the fault detection ability of test sets and thus assessing their adequacy. In this paper, we propose and evaluate TEASMA, a comprehensive and practical methodology designed to accurately assess the adequacy of test sets for DNNs. In practice, TEASMA allows engineers to decide whether they can trust high-accuracy test results and thus validate the DNN before its deployment. Based on a DNN model's training set, TEASMA…
Citations by year 10 total
1 2024 5 2025 4 2026IEEE
From cool demos to production-ready fmware: Core challenges and a technology roadmap
JournalACM Transactions on Software Engineering and Methodology · 2024
The rapid expansion of foundation models (FMs), such as large language models (LLMs), has given rise to FMware, software systems that integrate FM(s) as core components. While building demonstration-level FMware is relatively straightforward, transitioning to production-ready systems presents numerous challenges, including reliability, high implementation costs, scalability, and compliance with privacy regulations. Our paper conducts a semi-structured thematic synthesis to identify key challenges in productionizing FMware across diverse data sources, including our industry experience developing FMArts, a FMware lifecycle engineering platform, and its integration into Huawei Cloud; grey literature; academic publications; hands-on involvement in the Open Platform for Enterprise AI (OPEA); organizing the AIware conference and bootcamp; and co-leading the ISO SPDX SBOM working group on AI and…
Citations by year 8 total
1 2024 5 2025 2 2026ACM
Data quality antipatterns for software analytics
PreprintarXiv preprint arXiv:2408.12560 · 2024
Background Objective Methods Results Conclusion
Citations by year 6 total
4 2025 2 2026AI judge systems are designed to automatically evaluate Foundation Model-powered software (i.e., FMware). Due to the intrinsic dynamic and stochastic nature of FMware, the development of AI judge systems requires a unique engineering life cycle and presents new challenges. In this paper, we discuss the challenges based on our industrial experiences in developing AI judge systems for FMware. These challenges lead to substantial time consumption, cost and inaccurate judgments. We propose a framework that tackles the challenges with the goal of improving the productivity of developing high-quality AI judge systems. Finally, we evaluate our framework with a case study on judging a commit message generation FMware. The accuracy of the judgments made by the AI judge system developed with our framework outperforms those made by the AI judge system that is developed without our framework by up to 6.2%, with a significant reduction in development effort.
Citations by year 6 total
2 2025 4 2026Technical Brief on Software Engineering for FMware
ConferenceProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings
Foundation Models (FM) like GPT-4 have given rise to FMware, FM-powered applications, which represent a new generation of software that is developed with new roles, assets, and paradigms. FMware has been widely adopted in both software engineering (SE) research (e.g., test generation) and industrial products (e.g., GitHub copilot), despite the numerous challenges introduced by the stochastic nature of FMs. Such challenges jeopardize the quality and trustworthiness of FMware. In our technical brief, we will present the latest research and industrial practices in engineering FMware, and discuss the SE challenges and opportunities facing both researchers and practitioners in the FMware era. The brief is unique in that it is presented from an SE point of view, not an AI point-of-view ensuring that attendees are not bogged into complex mathematical and AI details unless they are essential for contextualizing…
Citations by year 4 total
3 2025 1 2026Emerging Advanced Technologies for Game Engineering
JournalACM SIGSOFT Software Engineering Notes 49 (3) · 2024
In this paper, the outcomes of the 8th International Workshop on Games and Software Engineering (GAS 2024)1 are reported. The one-day workshop has been held as part of the 46th International Conference on Software Engineering (ICSE 2024) in Lisbon, Portugal on April 14, 2024. The workshop programme includes two exciting keynotes discussing topics related to harnessing video game simulations to generate content and locate bugs, and the experience of maintaining a popular FOSS library, raylib. There are three research paper sessions. The first relates to automation in game engineering; the second explores testing and quality assurance; and the third discusses specification and quality of service. The conclusion of the workshop is anchored by a panel of four researchers, educators, and practitioners discussing the current strengths and limitations of large language models in game engineering.
Citations by year 2 total
2 2024ACM
Foreword to the 8th International Workshop on Games and Software Engineering (GAS 2024)
Conference2024 IEEE/ACM 8th International Workshop on Games and Software Engineering (GAS)
The evolving expectations within gaming communities drive ongoing research and development efforts to devise innovative methods for constructing intricate gameful systems. These methods incorporate cutting-edge technologies like AI, AR/VR, and biometrics, to meet the needs of gaming communities. Gameful systems encompass three main categories: entertainment games, which offers captivating user experiences aimed at immersing and retaining players; serious games, which combines the entertainment aspect of gaming with specific goals in domains such as education, training, and healthcare; and gamified applications, which augments non-entertainment applications, like learning management systems, by integrating game elements.
IEEE
Systems, methods, and non-transitory computer-readable storage devices for detecting and analyzing data clones in tabular datasets
PatentUS Patent 18386023 · 2024
2024-07-25 Assigned to HUAWEI TECHNOLOGIES CO., LTD. reassignment HUAWEI TECHNOLOGIES CO., LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LIN, DAYI, RAJBAHADUR, Gopi Krishnan, YANG, XU
Systems, methods, and non-transitory computer-readable storage devices for training deep learning and neural network models using overfitting detection and prevention
PatentUS Patent 18384634 · 2024
A method for detecting and/or preventing overfitting in training of deep learning and neural network models. The method has a classifier-training method, an overfittingdetection method, and an overfitting-prevention method. The classifier-training method trains one or more classifiers using training histories and labels of one or more trained machine-learning (ML) models. The overfitting-detection method uses the trained classifiers based on the training history such as validation losses of a trained target ML model to identify an overfitting status of the trained target ML model. The overfitting-prevention method is performed during the training of a target ML model and uses the trained classifiers based on the training history of the target ML model to identify and preventing overfitting of the target ML 3, 2022. model.
Games and Software Engineering: Engineering fun, inspiration, and motivation
JournalACM SIGSOFT Software Engineering Notes 48 (1) · 2023
Games are a popular form of entertainment and, due to their nature (i.e., interactive, immersive, etc.), strongly lend themselves for use beyond this original intent. Serious games, or games with a purpose, have been introduced to integrate the entertainment value games with domain specific objectives on important topics within education, health, and the environment to mention a few. In addition, gamification has been used to enhance nonentertainment applications with game elements; it aspires to foster behavioral changes, engagement, motivation, and participation in activities. In this context, the actions performed have meaning/value in the game experience in order to improve workplace performance or learn something in real life. The growing adoption of gameful experiences in all of the previous contexts make their design and development increasingly complex due to, for example, the number and variety of…
Citations by year 18 total
3 2023 8 2024 6 2025 1 2026ACM
Analyzing gamer complaints in reviews of cross-platform video games on steam
Conference2023 IEEE Conference on Games (CoG)
Video gaming now represents the largest category in the entertainment industry in terms of revenue. To expand their market share, game developers are creating more cross-platform games, which are compatible with various platforms, including PCs, consoles, and smartphones. However, creating such games poses challenges as developers encounter platform-specific issues that may only surface on one of the target platforms. Consequently, many ported games fail due to careless adaptation from one exclusive platform to another. This paper presents the first empirical study on cross-platform issues by analyzing game users’ reviews for video games on both PC and game console(s). Our findings reveal that platform-related issues occur more frequently on the PC side, particularly for games that are ported from consoles. To address this challenge, we develop machine learning-based approaches to automatically…
Citations by year 7 total
2 2024 5 2025IEEE
Fostering Collaboration and Advancing Research in Software Engineering and Game Development for Serious Contexts
JournalACM SIGSOFT Software Engineering Notes 48 (4) · 2023
The potential benefits of using the engaging and interactive nature of games to achieve specific objectives have been recognized by researchers and professionals from numerous domains. Serious games have been developed to impart knowledge, skills, and awareness in areas such as education, healthcare and the environment, while gamification has been applied to enhance the engagement, motivation, and participation of users in non-game activities such as sustainability and learning. As a result, the fields of game engineering, software engineering, and user experience are increasingly converging to create innovative solutions that blend the strengths of games with real-world applications.
Citations by year 4 total
1 2024 3 2025ACM
Welcome to the 6th International Workshop on Games and Software Engineering (GAS 2022), held in conjunction with the 44th ACM/IEEE International Conference on Software Engineering (ICSE 2022). GAS is an annual event gathering researchers and practitioners interested in sharing and advancing game engineering and software engineering techniques.
Towards training reproducible deep learning models
ConferenceProceedings of the 44th international conference on software engineering · 2022
Reproducibility is an increasing concern in Artificial Intelligence (AI), particularly in the area of Deep Learning (DL). Being able to reproduce DL models is crucial for AI-based systems, as it is closely tied to various tasks like training, testing, debugging, and auditing. However, DL models are challenging to be reproduced due to issues like randomness in the software (e.g., DL algorithms) and non-determinism in the hardware (e.g., GPU). There are various practices to mitigate some of the aforementioned issues. However, many of them are either too intrusive or can only work for a specific usage context. In this paper, we propose a systematic approach to training reproducible DL models. Our approach includes three main parts: (1) a set of general criteria to thoroughly evaluate the reproducibility of DL models for two different domains, (2) a unified framework which leverages a record-and-replay technique to mitigate…
Citations by year 92 total
2 2022 9 2023 30 2024 25 2025 26 2026An Empirical Study of Delayed Games on Steam
PreprintarXiv preprint arXiv:2204.11191 · 2022
The gaming industry is rapidly expanding. With over 2.7 billion players worldwide, game development has become increasingly challenging. To meet the ever-changing demands and expectations of players and due to unforeseen hindrances in the development process, game developers may require to delay the release of their game. We conducted an empirical study of 23,485 games on the Steam platform to analyze how often, and which games delayed their release date. We find that delaying a release is common: 48% of the studied games had a delayed initial release. Games delayed their release by a median of 14 days. Games for which a release date range (e.g., "Q1 2019") was specified, rather than a specific date were more likely to release within that range. Across different game genres, the percentage of games that delay release is similar (ranging from 48% to 52%). Finally, games with a delayed release are rated lower than games that release on time, but the difference is negligible.
Citations by year 6 total
3 2023 1 2024 2 2026An Empirical Study of Trends of Popular Virtual Reality Games and Their Complaints
JournalIEEE Transactions on Games · 2021
The market for virtual reality (VR) games is growing rapidly and is expected to grow from 3.3 billion in 2018 to 13.7 billion in 2022. Due to the immersive nature of such games and the use of VR headsets, players may have complaints about VR games, which are distinct from those about traditional computer games, and an understanding of those complaints could enable developers to better take advantage of the growing VR market. We conduct an empirical study of 750 popular VR games and 17 635 user reviews on Steam in order to understand trends in VR games and their complaints. We find that the VR games market is maturing. Fewer VR games are released each month, but their quality appears to be improving over time. Most games support multiple headsets and play areas, and support for smaller scale play areas is increasing. Complaints of cybersickness are rare and declining, indicating that players are…
Citations by year 54 total
3 2021 3 2022 6 2023 21 2024 15 2025 6 2026IEEE
What Causes Wrong Sentiment Classifications of Game Reviews?
JournalIEEE Transactions on Games · 2021
Sentiment analysis is a popular technique to identify the sentiment of a piece of text. Several different domains have been targeted by sentiment analysis research, such as Twitter, movie reviews, and mobile app reviews. Although several techniques have been proposed, the performance of current sentiment analysis techniques is still far from acceptable, mainly when applied in domains on which they were not trained. In addition, the causes of wrong classifications are not clear. In this article, we study how sentiment analysis performs on game reviews. We first report the results of a large-scale empirical study on the performance of widely used sentiment classifiers on game reviews. Then, we investigate the root causes for the wrong classifications and quantify the impact of each cause on the overall performance. We study three existing classifiers: Stanford CoreNLP, NLTK, and SentiStrength. Our results show that…
Citations by year 52 total
1 2020 6 2022 13 2023 13 2024 12 2025 7 2026IEEE
Towards a consistent interpretation of AIOps models
JournalACM Transactions on Software Engineering and Methodology · 2021
Artificial Intelligence for IT Operations (AIOps) has been adopted in organizations in various tasks, including interpreting models to identify indicators of service failures. To avoid misleading practitioners, AIOps model interpretations should be consistent (i.e., different AIOps models on the same task agree with one another on feature importance). However, many AIOps studies violate established practices in the machine learning community when deriving interpretations, such as interpreting models with suboptimal performance, though the impact of such violations on the interpretation consistency has not been studied. In this article, we investigate the consistency of AIOps model interpretation along three dimensions: internal consistency, external consistency, and time consistency. We conduct a case study on two AIOps tasks: predicting Google cluster job failures and Backblaze hard drive failures. We find that the…
Citations by year 48 total
2 2022 10 2023 12 2024 15 2025 9 2026The Impact of Data Merging on the Interpretation of Cross-Project Just-In-Time Defect Models
JournalIEEE Transactions on Software Engineering · 2021
Just-In-Time (JIT) defect models are classification models that identify the code commits that are likely to introduce defects. Cross-project JIT models have been introduced to address the suboptimal performance of JIT models when historical data is limited. However, many studies built cross-project JIT models using a pool of mixed data from multiple projects (i.e., data merging)—assuming that the properties of defect-introducing commits of a project are similar to that of the other projects, which is likely not true. In this paper, we set out to investigate the interpretation of JIT defect models that are built from individual project data and a pool of mixed project data with and without consideration of project-level variances. Through a case study of 20 datasets of open source projects, we found that (1) the interpretation of JIT models that are built from individual projects varies among projects; and (2) the project-level…
Citations by year 31 total
1 2021 1 2022 6 2023 9 2024 9 2025 5 2026IEEE
Can I use this publicly available dataset to build commercial AI software?–A Case Study on Publicly Available Image Datasets
PreprintarXiv preprint arXiv:2111.02374 · 2021
Publicly available datasets are one of the key drivers for commercial AI software. The use of publicly available datasets is governed by dataset licenses. These dataset licenses outline the rights one is entitled to on a given dataset and the obligations that one must fulfil to enjoy such rights without any license compliance violations. Unlike standardized Open Source Software (OSS) licenses, existing dataset licenses are defined in an ad-hoc manner and do not clearly outline the rights and obligations associated with their usage. Further, a public dataset may be hosted in multiple locations and created from multiple data sources each of which may have different licenses. Hence, existing approaches on checking OSS license compliance cannot be used. In this paper, we propose a new approach to assessing the potential license compliance violations if a given publicly available dataset were to be used for building commercial AI software. We conduct a case study with our approach on 6 commonly used publicly available image datasets. Our results show that there exists potential risks of license violations associated with all of the studied datasets if they were used for commercial purposes.
Citations by year 23 total
5 2023 9 2024 4 2025 5 2026Building the perfect game–an empirical study of game modifications
JournalEmpirical Software Engineering 25 (4) · 2020
Prior work has shown that gamer loyalty is important for the sales of a developer’s future games. Therefore, it is important for game developers to increase the longevity of their games. However, game developers cannot always meet the growing and changing needs of the gaming community, due to the often already overloaded schedules of developers. So-called modders can potentially assist game developers with addressing gamers’ needs. Modders are enthusiasts who provide modifications or completely new content for a game. By supporting modders, game developers can meet the rapidly growing and varying needs of their gamer base. Modders have the potential to play a role in extending the life expectancy of a game, thereby saving game developers time and money, and leading to a better overall gaming experience for their gamer base. In this paper, we empirically study the metadata of 9,521 mods that…
Citations by year 62 total
2 2019 6 2020 8 2021 7 2022 10 2023 8 2024 14 2025 7 2026Springer US
An empirical study of the characteristics of popular Minecraft mods
JournalEmpirical Software Engineering 25 (5) · 2020
It is becoming increasingly difficult for game developers to manage the cost of developing a game, while meeting the high expectations of gamers. One way to balance the increasing gamer expectation and development stress is to build an active modding community around the game. There exist several examples of games with an extremely active and successful modding community, with the Minecraft game being one of the most notable ones. This paper reports on an empirical study of 1,114 popular and 1,114 unpopular Minecraft mods from the CurseForge mod distribution platform, one of the largest distribution platforms for Minecraft mods. We analyzed the relationship between 33 features across 5 dimensions of mod characteristics and the popularity of mods (i.e., mod category, mod documentation, environmental context of the mod, remuneration for the mod, and community contribution for the mod), to…
Citations by year 59 total
2 2019 1 2020 7 2021 9 2022 7 2023 13 2024 11 2025 9 2026Springer US
Identifying gameplay videos that exhibit bugs in computer games
JournalEmpirical Software Engineering 24 (6) · 2019
With the rapid growing market and competition in the gaming industry, it is challenging to develop a successful game, making the quality of games very important. To improve the quality of games, developers commonly use gamer-submitted bug reports to locate bugs in games. Recently, gameplay videos have become popular in the gaming community. A few of these videos showcase a bug, offering developers a new opportunity to collect context-rich bug information. In this paper, we investigate whether videos that showcase a bug can automatically be identified from the metadata of gameplay videos that are readily available online. Such bug videos could then be used as a supplemental source of bug information for game developers. We studied the number of gameplay videos on the Steam platform, one of the most popular digital game distribution platforms, and the difficulty of identifying bug videos from these…
Citations by year 46 total
2 2019 5 2020 5 2021 6 2022 9 2023 5 2024 10 2025 4 2026Springer US
How Can Game Developers Leverage Data from Online Distribution Platforms? A Case Study of the Steam Platform
ThesisQueen's University (Canada) · 2019
Developing a successful game is challenging. Prior work shows that gamers are extremely difficult to satisfy, making the quality of games an important issue. Prior work has yielded important results from mining data that is available on the online distribution platforms for software applications, helping practitioners save valuable resources, and improving the user-perceived quality of software that is distributed through these platforms. However, much of the work on mining online distribution platforms focuses on mining mobile app stores (eg, Google Play Store, Apple App Store). Video game development differs from the development of other types of software. Hence, knowledge derived from mining mobile app stores may not be directly applicable to game development. In this Ph. D. thesis, we focused on mining online distribution platforms for games. In particular, we mined data from the Steam platform, the largest…
Citations by year 2 total
1 2021 1 2024An empirical study of game reviews on the Steam platform
JournalEmpirical Software Engineering · 2018
The steadily increasing popularity of computer games has led to the rise of a multi-billion dollar industry. Due to the scale of the computer game industry, developing a successful game is challenging. In addition, prior studies show that gamers are extremely hard to please, making the quality of games an important issue. Most online game stores allow users to review a game that they bought. Such reviews can make or break a game, as other potential buyers often base their purchasing decisions on the reviews of a game. Hence, studying game reviews can help game developers better understand user concerns, and further improve the user-perceived quality of games. In this paper, we perform an empirical study of the reviews of 6224 games on the Steam platform, one of the most popular digital game delivery platforms, to better understand if game reviews share similar characteristics with mobile app reviews, and…
Citations by year 226 total
15 2019 13 2020 33 2021 24 2022 33 2023 34 2024 41 2025 33 2026Springer US
An empirical study of early access games on the Steam platform
JournalEmpirical Software Engineering 23 (2) · 2018
“Early access” is a release strategy for software that allows consumers to purchase an unfinished version of the software. In turn, consumers can influence the software development process by giving developers early feedback. This early access model has become increasingly popular through digital distribution platforms, such as Steam which is the most popular distribution platform for games. The plethora of options offered by Steam to communicate between developers and game players contribute to the popularity of the early access model. The model is considered a success by the game development community as several games using this approach have gained a large user base (i.e., owners) and high sales. On the other hand, the benefits of the early access model have been questioned as well. In this paper, we conduct an empirical study on 1,182 Early Access Games (EAGs) on the Steam platform to…
Citations by year 104 total
10 2019 15 2020 17 2021 8 2022 18 2023 14 2024 18 2025 4 2026Springer US
Studying the urgent updates of popular games on the Steam platform
JournalEmpirical Software Engineering 22 (4) · 2017
The steadily increasing popularity of computer games has led to the rise of a multi-billion dollar industry. This increasing popularity is partly enabled by online digital distribution platforms for games, such as Steam. These platforms offer an insight into the development and test processes of game developers. In particular, we can extract the update cycle of a game and study what makes developers deviate from that cycle by releasing so-called urgent updates. An urgent update is a software update that fixes problems that are deemed critical enough to not be left unfixed until a regular-cycle update. Urgent updates are made in a state of emergency and outside the regular development and test timelines which causes unnecessary stress on the development team. Hence, avoiding the need for an urgent update is important for game developers. We define urgent updates as 0-day updates (updates that are released on…
Citations by year 99 total
1 2017 6 2018 14 2019 17 2020 15 2021 16 2022 15 2023 6 2024 5 2025 4 2026Springer US
Chinese word segmentation based on conditional random fields with character clustering
ConferenceAsian Language Processing (IALP), 2016 International Conference on
Chinese word segmentation plays an important role in Chinese text mining. It is the foundation of automatic relation extraction and identification in Chinese information processing. In this paper, we propose a method for Chinese word segmentation based on conditional random fields (CRF) with character clustering. For the character clustering, we firstly use the Skip-Gram model to obtain character embedding from a raw corpus (without word delimiters). We then apply two different clustering algorithms, K-means and Brown clustering algorithm, to get the clusters of character embedding. The effect of different numbers of dimensions of character embedding, the number of clusters, and different clustering algorithms have been studied. We verify our method using the 4th CCF Conference on Natural Language Processing and Chinese Computing (NLPCC2015) Weibo text segmentation task. Our system achieves an F…
Citations by year 17 total
2 2017 5 2018 2 2019 2 2020 3 2021 1 2022 2 2023IEEE
Chinese term extraction from web pages based on expected point-wise mutual information
ConferenceNatural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), 2016 12th International Conference on
Point-wise Mutual Information(PMI) has been widely used in many areas of lexicon construction, term extraction and text mining. However, PMI has a well-known tendency, which is overvaluing the relatedness of word pairs that involve low-frequency words. To overcome this limitation, Expected Point-wise Mutual Information (PMIK) has been proposed empirically. In this paper, we propose an automatic term recognition system for Chinese and theoretically prove that with variant k ≥ 3, PMIK method can overcome the bias of low-frequency words. The experiment results on Chinese SINA blog and Baidu Tieba corpus show that with a proper k value of 5, the system can achieve a precision greater than 81% for top 1000 extracted terms without decreasing the recall.
Citations by year 8 total
1 2020 2 2021 1 2022 1 2023 2 2024 1 2026IEEE
No publications match your filters.