ARCHIVES
Original Article
Three Measurement Hazards in Analyzer-in-the-Loop Repair of Smart Contracts
Ian Staley1
1 Independent Researcher, United States.
Published Online: July-August 2026
Pages: 160-170
Cite this article
↗ https://www.doi.org/10.59256/ijire.20260704018
References
1. C. Wang, J. Zhang, J. Gao, L. Xia, Z. Guan, and Z. Chen, "ContractTinker: LLM-empowered vulnerability repair for real-world smart contracts," in Proc. 39th IEEE/ACM Int. Conf. Automated Software Engineering (ASE), Sacramento, CA, USA, 2024, pp. 2350–2353, doi: 10.1145/3691620.3695349.
2. L. Zhang et al., "ACFix: Guiding LLMs with mined common RBAC practices for context-aware repair of access control vulnerabilities in smart contracts," IEEE Trans. Softw. Eng., vol. 51, no. 9, pp. 2512–2532, Sep. 2025, doi: 10.1109/TSE.2025.3590108.
3. X. Li, S. Ye, W. Li, and Z. Li, "SCPatcher: Automated smart contract code repair via retrieval-augmented generation and knowledge graph," in Proc. 34th ACM Int. Conf. Foundations of Software Engineering (FSE Companion), 2026, pp. 1212–1216, doi: 10.1145/3803437.3805558.
4. J. Huang et al., "Large language models cannot self-correct reasoning yet," in Proc. 12th Int. Conf. Learning Representations (ICLR), Vienna, Austria, 2024. [Online]. Available: https://openreview.net/forum? id=IkmD3fKBPQ (accessed Jul. 25, 2026).
5. T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Solar-Lezama, "Is self-repair a silver bullet for code generation?," in Proc. 12th Int. Conf. Learning Representations (ICLR), Vienna, Austria, 2024. [Online]. Available: https://openreview.net/forum?id=y0GJXRungR (accessed Jul. 25, 2026).
6. A. Madaan et al., "Self-Refine: Iterative refinement with self-feedback," in Advances in Neural Information Processing Systems 36 (NeurIPS), New Orleans, LA, USA, 2023, pp. 46534–46594. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/hash/91edff07 232fb1b55a505a9e9f6c0ff3-Abstract-Conference.html (accessed Jul. 25, 2026).
7. N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao, "Reflexion: Language agents with verbal reinforcement learning," in Advances in Neural Information Processing Systems 36 (NeurIPS), New Orleans, LA, USA, 2023, pp. 8634–8652. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b87 8bb782e6954cd888628510e90-Abstract-Conference.html (accessed Jul. 25, 2026).
8. X. Chen, M. Lin, N. Schärli, and D. Zhou, "Teaching large language models to self-debug," in Proc. 12th Int. Conf. Learning Representations (ICLR), Vienna, Austria, 2024. [Online]. Available: https://openreview.net/forum?id=KuPixIqPiq (accessed Jul. 25, 2026).
9. Z. Gou et al., "CRITIC: Large language models can self-correct with tool-interactive critiquing," in Proc. 12th Int. Conf. Learning Representations (ICLR), Vienna, Austria, 2024. [Online]. Available: https://openreview.net/forum?id=Sx038qxjek (accessed Jul. 25, 2026).
10. C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in Software Engineering. Berlin, Heidelberg, Germany: Springer, 2012, doi: 10.1007/978-3-642-29044-2.
11. P. Ralph et al., "Empirical standards for software engineering research," arXiv:2010.03525, 2020, doi: 10.48550/arXiv.2010.03525.
12. R. Feldt and A. Magazinius, "Validity threats in empirical software engineering research: An initial survey," in Proc. 22nd Int. Conf. Software Engineering and Knowledge Engineering (SEKE), Redwood City, CA, USA, 2010, pp. 374–379.
13. L. Hatton, "The T-experiments: Errors in scientific software," IEEE Computational Science and Engineering, vol. 4, no. 2, pp. 27–38, Apr.–Jun. 1997, doi: 10.1109/99.609829.
14. D. C. Ince, L. Hatton, and J. Graham-Cumming, "The case for open computer programs," Nature, vol. 482, no. 7386, pp. 485–488, Feb. 2012, doi: 10.1038/nature10836.
15. J. Spacco, D. Hovemeyer, and W. Pugh, "Tracking defect warnings across versions," in Proc. Int. Workshop Mining Software Repositories (MSR), Shanghai, China, 2006, pp. 133–136, doi: 10.1145/1137983.1138014.
16. P. Avgustinov et al., "Tracking static analysis violations over time to capture developer characteristics," in Proc. IEEE/ACM 37th Int. Conf. Software Engineering (ICSE), vol. 1, Florence, Italy, 2015, pp. 437–447, doi: 10.1109/ICSE.2015.62.
17. ConsenSys Diligence, "mythril.mythril CRITICAL exception in mythril.py," mythril issue 939, GitHub. [Online]. Available: https://github.com/ConsenSysDiligence/mythril/issues/939 (accessed Jul. 25, 2026).
18. ConsenSys Diligence, "--solv throws various errors when not using the docker image," mythril issue 1424, GitHub. [Online]. Available: https://github.com/ConsenSysDiligence/mythril/issues/1424 (accessed Jul. 25, 2026).
19. T. Muske and P. Bokil, "On implementational variations in static analysis tools," in Proc. IEEE 22nd Int. Conf. Software Analysis, Evolution, and Reengineering (SANER), Montréal, QC, Canada, 2015, pp. 512–515, doi: 10.1109/SANER.2015.7081867.
20. J. Feist, G. Grieco, and A. Groce, "Slither: A static analysis framework for smart contracts," in Proc. IEEE/ACM 2nd Int. Workshop Emerging Trends in Software Engineering for Blockchain (WETSEB), Montreal, QC, Canada, 2019, pp. 8–15, doi: 10.1109/WETSEB.2019.00008.
21. ConsenSys Diligence, "Mythril: Security analysis tool for EVM bytecode," version 0.24.8, 2026. [Online]. Available: https://github.com/ConsenSysDiligence/mythril (accessed Jul. 25, 2026).
22. T. Durieux, J. F. Ferreira, R. Abreu, and P. Cruz, "Empirical review of automated analysis tools on 47,587 Ethereum smart contracts," in Proc. ACM/IEEE 42nd Int. Conf. Software Engineering (ICSE), Seoul, South Korea, 2020, pp. 530–541, doi: 10.1145/3377811.3380364.
23. F. Salzano, C. K. Antenucci, S. Scalabrino, G. Rosa, R. Oliveto, and R. Pareschi, "An empirical analysis of vulnerability detection tools for Solidity smart contracts using line level manually annotated vulnerabilities," Empirical Softw. Eng., vol. 31, Art. no. 143, 2026, doi: 10.1007/s10664-026-10867-7.
24. J. F. Ferreira, P. Cruz, T. Durieux, and R. Abreu, "SmartBugs: A framework to analyze Solidity smart contracts," in Proc. 35th IEEE/ACM Int. Conf. Automated Software Engineering (ASE), 2020, pp. 1349–1352, doi: 10.1145/3324884.3415298.
25. M. di Angelo, T. Durieux, J. F. Ferreira, and G. Salzer, "SmartBugs 2.0: An execution framework for weakness detection in Ethereum smart contracts," in Proc. 38th IEEE/ACM Int. Conf. Automated Software Engineering (ASE), 2023, pp. 2102–2105, doi: 10.1109/ASE56229.2023.00060.
26. K. Li et al., "Static application security testing (SAST) tools for smart contracts: How far are we?," Proc. ACM Softw. Eng., vol. 1, no. FSE, Art. no. 65, pp. 1447–1470, Jul. 2024, doi: 10.1145/3660772.
27. S. Bobadilla, M. Jin, and M. Monperrus, “Do automated fixes truly mitigate smart contract exploits?,” IEEE Trans. Softw. Eng., vol. 52, no. 1, pp. 100–115, 2026, doi: 10.1109/TSE.2025.3618123.
28. C. Sadowski, J. van Gogh, C. Jaspan, E. Söderberg, and C. Winter, "Tricorder: Building a program analysis ecosystem," in Proc. IEEE/ACM 37th Int. Conf. Software Engineering (ICSE), vol. 1, Florence, Italy, 2015, pp. 598–608, doi: 10.1109/ICSE.2015.76.
29. C. Sadowski, E. Aftandilian, A. Eagle, L. Miller-Cushon, and C. Jaspan, "Lessons from building static analysis tools at Google," Commun. ACM, vol. 61, no. 4, pp. 58–66, Apr. 2018, doi: 10.1145/3188720.
30. B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge, "Why don't software developers use static analysis tools to find bugs?," in Proc. 35th Int. Conf. Software Engineering (ICSE), San Francisco, CA, USA, 2013, pp. 672–681, doi: 10.1109/ICSE.2013.6606613.
31. E. K. Smith, E. T. Barr, C. Le Goues, and Y. Brun, "Is the cure worse than the disease? Overfitting in automated program repair," in Proc. 10th Joint Meeting Foundations of Software Engineering (ESEC/FSE), Bergamo, Italy, 2015, pp. 532–543, doi: 10.1145/2786805.2786825.
32. Z. Qi, F. Long, S. Achour, and M. Rinard, "An analysis of patch plausibility and correctness for generate-and-validate patch generation systems," in Proc. Int. Symp. Software Testing and Analysis (ISSTA), Baltimore, MD, USA, 2015, pp. 24–36, doi: 10.1145/2771783.2771791.
33. A. Ghaleb, J. Rubin, and K. Pattabiraman, "AChecker: Statically detecting smart contract access control vulnerabilities," in Proc. IEEE/ACM 45th Int. Conf. Software Engineering (ICSE), Melbourne, Australia, 2023, pp. 945–956, doi: 10.1109/ICSE48619.2023.00087.
34. Association for Computing Machinery, "Artifact review and badging, version 1.1," Aug. 2020. [Online]. Available: https://www.acm.org/publications/policies/artifact-review-and-badging-current (accessed Jul. 25, 2026).
35. I. Staley, "Replication package: Three measurement hazards in analyzer-in-the-loop repair of smart contracts," Zenodo, Jul. 2026, doi: 10.5281/zenodo.21586404.
36. S. Baltes et al., "Guidelines for empirical studies in software engineering involving large language models," arXiv:2508.15503, 2025, doi: 10.48550/arXiv.2508.15503.
2. L. Zhang et al., "ACFix: Guiding LLMs with mined common RBAC practices for context-aware repair of access control vulnerabilities in smart contracts," IEEE Trans. Softw. Eng., vol. 51, no. 9, pp. 2512–2532, Sep. 2025, doi: 10.1109/TSE.2025.3590108.
3. X. Li, S. Ye, W. Li, and Z. Li, "SCPatcher: Automated smart contract code repair via retrieval-augmented generation and knowledge graph," in Proc. 34th ACM Int. Conf. Foundations of Software Engineering (FSE Companion), 2026, pp. 1212–1216, doi: 10.1145/3803437.3805558.
4. J. Huang et al., "Large language models cannot self-correct reasoning yet," in Proc. 12th Int. Conf. Learning Representations (ICLR), Vienna, Austria, 2024. [Online]. Available: https://openreview.net/forum? id=IkmD3fKBPQ (accessed Jul. 25, 2026).
5. T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Solar-Lezama, "Is self-repair a silver bullet for code generation?," in Proc. 12th Int. Conf. Learning Representations (ICLR), Vienna, Austria, 2024. [Online]. Available: https://openreview.net/forum?id=y0GJXRungR (accessed Jul. 25, 2026).
6. A. Madaan et al., "Self-Refine: Iterative refinement with self-feedback," in Advances in Neural Information Processing Systems 36 (NeurIPS), New Orleans, LA, USA, 2023, pp. 46534–46594. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/hash/91edff07 232fb1b55a505a9e9f6c0ff3-Abstract-Conference.html (accessed Jul. 25, 2026).
7. N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao, "Reflexion: Language agents with verbal reinforcement learning," in Advances in Neural Information Processing Systems 36 (NeurIPS), New Orleans, LA, USA, 2023, pp. 8634–8652. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b87 8bb782e6954cd888628510e90-Abstract-Conference.html (accessed Jul. 25, 2026).
8. X. Chen, M. Lin, N. Schärli, and D. Zhou, "Teaching large language models to self-debug," in Proc. 12th Int. Conf. Learning Representations (ICLR), Vienna, Austria, 2024. [Online]. Available: https://openreview.net/forum?id=KuPixIqPiq (accessed Jul. 25, 2026).
9. Z. Gou et al., "CRITIC: Large language models can self-correct with tool-interactive critiquing," in Proc. 12th Int. Conf. Learning Representations (ICLR), Vienna, Austria, 2024. [Online]. Available: https://openreview.net/forum?id=Sx038qxjek (accessed Jul. 25, 2026).
10. C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in Software Engineering. Berlin, Heidelberg, Germany: Springer, 2012, doi: 10.1007/978-3-642-29044-2.
11. P. Ralph et al., "Empirical standards for software engineering research," arXiv:2010.03525, 2020, doi: 10.48550/arXiv.2010.03525.
12. R. Feldt and A. Magazinius, "Validity threats in empirical software engineering research: An initial survey," in Proc. 22nd Int. Conf. Software Engineering and Knowledge Engineering (SEKE), Redwood City, CA, USA, 2010, pp. 374–379.
13. L. Hatton, "The T-experiments: Errors in scientific software," IEEE Computational Science and Engineering, vol. 4, no. 2, pp. 27–38, Apr.–Jun. 1997, doi: 10.1109/99.609829.
14. D. C. Ince, L. Hatton, and J. Graham-Cumming, "The case for open computer programs," Nature, vol. 482, no. 7386, pp. 485–488, Feb. 2012, doi: 10.1038/nature10836.
15. J. Spacco, D. Hovemeyer, and W. Pugh, "Tracking defect warnings across versions," in Proc. Int. Workshop Mining Software Repositories (MSR), Shanghai, China, 2006, pp. 133–136, doi: 10.1145/1137983.1138014.
16. P. Avgustinov et al., "Tracking static analysis violations over time to capture developer characteristics," in Proc. IEEE/ACM 37th Int. Conf. Software Engineering (ICSE), vol. 1, Florence, Italy, 2015, pp. 437–447, doi: 10.1109/ICSE.2015.62.
17. ConsenSys Diligence, "mythril.mythril CRITICAL exception in mythril.py," mythril issue 939, GitHub. [Online]. Available: https://github.com/ConsenSysDiligence/mythril/issues/939 (accessed Jul. 25, 2026).
18. ConsenSys Diligence, "--solv throws various errors when not using the docker image," mythril issue 1424, GitHub. [Online]. Available: https://github.com/ConsenSysDiligence/mythril/issues/1424 (accessed Jul. 25, 2026).
19. T. Muske and P. Bokil, "On implementational variations in static analysis tools," in Proc. IEEE 22nd Int. Conf. Software Analysis, Evolution, and Reengineering (SANER), Montréal, QC, Canada, 2015, pp. 512–515, doi: 10.1109/SANER.2015.7081867.
20. J. Feist, G. Grieco, and A. Groce, "Slither: A static analysis framework for smart contracts," in Proc. IEEE/ACM 2nd Int. Workshop Emerging Trends in Software Engineering for Blockchain (WETSEB), Montreal, QC, Canada, 2019, pp. 8–15, doi: 10.1109/WETSEB.2019.00008.
21. ConsenSys Diligence, "Mythril: Security analysis tool for EVM bytecode," version 0.24.8, 2026. [Online]. Available: https://github.com/ConsenSysDiligence/mythril (accessed Jul. 25, 2026).
22. T. Durieux, J. F. Ferreira, R. Abreu, and P. Cruz, "Empirical review of automated analysis tools on 47,587 Ethereum smart contracts," in Proc. ACM/IEEE 42nd Int. Conf. Software Engineering (ICSE), Seoul, South Korea, 2020, pp. 530–541, doi: 10.1145/3377811.3380364.
23. F. Salzano, C. K. Antenucci, S. Scalabrino, G. Rosa, R. Oliveto, and R. Pareschi, "An empirical analysis of vulnerability detection tools for Solidity smart contracts using line level manually annotated vulnerabilities," Empirical Softw. Eng., vol. 31, Art. no. 143, 2026, doi: 10.1007/s10664-026-10867-7.
24. J. F. Ferreira, P. Cruz, T. Durieux, and R. Abreu, "SmartBugs: A framework to analyze Solidity smart contracts," in Proc. 35th IEEE/ACM Int. Conf. Automated Software Engineering (ASE), 2020, pp. 1349–1352, doi: 10.1145/3324884.3415298.
25. M. di Angelo, T. Durieux, J. F. Ferreira, and G. Salzer, "SmartBugs 2.0: An execution framework for weakness detection in Ethereum smart contracts," in Proc. 38th IEEE/ACM Int. Conf. Automated Software Engineering (ASE), 2023, pp. 2102–2105, doi: 10.1109/ASE56229.2023.00060.
26. K. Li et al., "Static application security testing (SAST) tools for smart contracts: How far are we?," Proc. ACM Softw. Eng., vol. 1, no. FSE, Art. no. 65, pp. 1447–1470, Jul. 2024, doi: 10.1145/3660772.
27. S. Bobadilla, M. Jin, and M. Monperrus, “Do automated fixes truly mitigate smart contract exploits?,” IEEE Trans. Softw. Eng., vol. 52, no. 1, pp. 100–115, 2026, doi: 10.1109/TSE.2025.3618123.
28. C. Sadowski, J. van Gogh, C. Jaspan, E. Söderberg, and C. Winter, "Tricorder: Building a program analysis ecosystem," in Proc. IEEE/ACM 37th Int. Conf. Software Engineering (ICSE), vol. 1, Florence, Italy, 2015, pp. 598–608, doi: 10.1109/ICSE.2015.76.
29. C. Sadowski, E. Aftandilian, A. Eagle, L. Miller-Cushon, and C. Jaspan, "Lessons from building static analysis tools at Google," Commun. ACM, vol. 61, no. 4, pp. 58–66, Apr. 2018, doi: 10.1145/3188720.
30. B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge, "Why don't software developers use static analysis tools to find bugs?," in Proc. 35th Int. Conf. Software Engineering (ICSE), San Francisco, CA, USA, 2013, pp. 672–681, doi: 10.1109/ICSE.2013.6606613.
31. E. K. Smith, E. T. Barr, C. Le Goues, and Y. Brun, "Is the cure worse than the disease? Overfitting in automated program repair," in Proc. 10th Joint Meeting Foundations of Software Engineering (ESEC/FSE), Bergamo, Italy, 2015, pp. 532–543, doi: 10.1145/2786805.2786825.
32. Z. Qi, F. Long, S. Achour, and M. Rinard, "An analysis of patch plausibility and correctness for generate-and-validate patch generation systems," in Proc. Int. Symp. Software Testing and Analysis (ISSTA), Baltimore, MD, USA, 2015, pp. 24–36, doi: 10.1145/2771783.2771791.
33. A. Ghaleb, J. Rubin, and K. Pattabiraman, "AChecker: Statically detecting smart contract access control vulnerabilities," in Proc. IEEE/ACM 45th Int. Conf. Software Engineering (ICSE), Melbourne, Australia, 2023, pp. 945–956, doi: 10.1109/ICSE48619.2023.00087.
34. Association for Computing Machinery, "Artifact review and badging, version 1.1," Aug. 2020. [Online]. Available: https://www.acm.org/publications/policies/artifact-review-and-badging-current (accessed Jul. 25, 2026).
35. I. Staley, "Replication package: Three measurement hazards in analyzer-in-the-loop repair of smart contracts," Zenodo, Jul. 2026, doi: 10.5281/zenodo.21586404.
36. S. Baltes et al., "Guidelines for empirical studies in software engineering involving large language models," arXiv:2508.15503, 2025, doi: 10.48550/arXiv.2508.15503.
Related Articles
2026
AI-Based Stomach Cancer Detection Using Biomarkers, Medical Images, and Voice Analysis
2026
Hydrogen-Efficient Eco-Driving and Route Planning for Fuel-Cell Electric Vehicles Using Multi-Objective Optimization Under Traffic and Terrain Uncertainty
2026
A Data-Driven Machine Learning Framework for Assessing Patent Commercial Value and Technological Significance
2026
Evaluating Student Academic Performance Through a Benchmark of Fuzzy Reasoning Models
2026
A Hybrid Soft Computing Approach for Managing Uncertainty in Data Analytics
2026
