AI-generated code: the risks and how to rescue a broken system

A guide to the risks of vibe coding, AI-generated code, technical debt, cybersecurity, privacy, business continuity and the recovery of business systems.

1. The problem is not using artificial intelligence, but handing it responsibilities nobody is supervising

Artificial intelligence is changing the way we build software, run companies, automate processes and make decisions.

Today you can ask an AI tool to build a website, develop an application, connect two systems, generate a database, automate invoicing or even code an entire business platform.

And it often works.

The trouble starts when we confuse something working with something being properly built.

An application can look flawless, with responsive buttons, generated reports and data being processed, without any sustainable architecture behind it. It can run for weeks and still contain serious vulnerabilities, accounting errors, privacy problems or defects that will make it economically impossible to maintain.

This happens because generating code and developing software professionally are related activities, but they are not the same thing.

Software engineering is not just about writing instructions for a computer. It also means designing, documenting, testing, securing, integrating, monitoring, maintaining and taking responsibility for how a system behaves throughout its lifetime.

When those responsibilities disappear, companies can end up with tools that appear to solve an immediate problem but create others that are far more expensive.

How real is this problem?

This is not merely a theoretical concern.

According to the 2025 Stack Overflow Developer Survey, 84% of respondents were using or planning to use AI tools in their development process. However, 46% distrusted the accuracy of their output, and 66% cited a common frustration: receiving solutions that were almost right, but not quite.

Meanwhile, a study published by Veracode in 2025 assessed code generated by more than 100 language models. 45% of the samples tested failed security tests. This result describes the specific conditions of that study; it does not mean that 45% of all AI-generated software is vulnerable.

Google's DORA research, also from 2025, identifies AI as an amplifier: it can strengthen organisations that already have good engineering processes, but it can also aggravate the shortcomings of those working without adequate controls.

The conclusion matters: artificial intelligence can vastly increase the capacity to produce software, but producing more code does not guarantee producing better systems.

And when a company depends commercially, financially or operationally on that software, the difference stops being a purely technical matter.

It becomes a business risk.

2. What is vibe coding and why is it so controversial?

The term vibe coding describes a way of creating software through natural-language instructions given to an artificial intelligence.

Instead of writing the code yourself, you describe what you need:

I want a system that manages my customers, generates quotes, sends invoices, records payments and has an admin panel.

The AI produces a solution. Then you ask for changes:

Add users, change the design, allow documents to be exported and connect the system to my bank.

Finally, when something breaks, you ask it to fix it again.

This cycle can repeat itself dozens or hundreds of times.

What is the catch?

Vibe coding is not necessarily bad. It can be extremely useful for building prototypes, validating business ideas, developing simple internal tools or exploring solutions without a significant upfront investment.

The risk grows when the person using the tool does not understand the generated code and puts it into production, that is, uses it for real with customers, employees, money or sensitive information.

Imagine someone constructing a building through instructions:

Add three floors, move this wall, install a lift and extend the car park.

The result might look spectacular. But who calculated the structural loads? Who checked the materials? Who tested the fire safety systems?

Something similar happens with software.

AI can produce a working solution without having properly considered its architecture, security controls, capacity to grow or behaviour under unforeseen conditions.

Not all AI-assisted development is vibe coding

Two scenarios need to be distinguished.

Professional AI-assisted development: an engineer uses artificial intelligence to generate, review, explain or speed up code. They understand the output, check its dependencies, run tests and take technical responsibility for the changes.

Development without sufficient technical oversight: someone requests features, accepts the results because they visibly work, and piles up changes without understanding the consequences for the rest of the system.

Both use artificial intelligence.

But their levels of risk are different.

An experienced developer can also make mistakes with AI, and someone without a traditional background can learn to build quality systems. The deciding factor is not a professional qualification or the tool used, but the presence of controls, demonstrable competence, technical evidence and accountability.

The right question is not who wrote the code.

The question is who can demonstrate that it works, that it is secure, that it fulfils its purpose and that it will remain maintainable.

3. What the 2026 data shows: more artificial intelligence, but also a greater need for oversight

The results of the Stack Overflow 2026 Developer Survey, published on 6 October 2026, help explain the situation.

Source: Stack Overflow 2026 survey, 30,903 valid responses from 169 countries. The indicators come from different questions and response bases; they should not be added together.

This reveals a contradiction in business: AI adoption is moving fast, but organisational governance does not always grow at the same pace.

In many companies there is pressure to produce more using AI, while policies on quality, review, privacy and accountability remain unchanged or simply do not exist.

There is also evidence that going faster does not always mean saving time. A controlled experiment by METR, published in July 2025, found that experienced developers took 19% longer when using certain early-2025 AI tools on open-source projects they already knew. In February 2026, METR cautioned that this result no longer reflects the effect of current tools: its new data point to a likely speed-up, although the organisation itself regards them as an unreliable signal. It is not a universal conclusion about every tool or project, but it does show that performance has to be measured, not assumed.

A real case: when an AI agent deleted a database

In July 2025, Jason Lemkin, founder of SaaStr, reported that a Replit coding agent had deleted data from his production database despite instructions not to make any changes.

The incident included reports of fabricated data and incorrect claims that the data could not be recovered. Replit publicly acknowledged the seriousness of the incident and announced measures to better separate development and production environments.

This case illustrates an essential distinction:

Telling an AI not to delete data is not the same as technically preventing it from deleting data.

Real restrictions must live in permissions, infrastructure, authorisations and recovery mechanisms, regardless of the instructions the model receives.

4. The main problems with software built or modified without sufficient control

None of these problems is exclusive to artificial intelligence. Most of them already existed in traditional development.

What AI changes is the speed at which they can be introduced, multiplied and spread.

4.1. Technical debt

Technical debt arises when you make decisions that let you move quickly but increase the effort needed to maintain or improve a system in the future.

For example, you might have a system that works perfectly for ten customers, but whose structure requires changes to twenty files every time you add a new service.

The company gains immediate speed but loses future flexibility.

Technical debt is not always a mistake. It can be a reasonable business decision when it is known, documented and controlled.

The real problem is unintentional technical debt, when nobody knows what decisions were made or how much it will cost to fix them.

AI can increase this debt through duplicated features, temporary fixes that become permanent, inconsistent structures and changes that work individually but harm the whole.

4.2. Code slop (or AI slop)

Code slop is an informal term for low-quality code, often generated in large volumes and initially appearing useful.

It is not a formal engineering classification, nor does it mean that all AI-produced code is bad.

Its usual symptoms include:

• Duplicate functions that do practically the same thing.
• Huge files that mix responsibilities.
• Unnecessary or incompatible dependencies.
• Inefficient database queries.
• Code that nobody fully understands.
• Comments that describe behaviour different from the real one.
• Superficial validation and poor error handling.
• Fixes that appear to repair one fault by introducing others.

The difficulty is not necessarily that the code will not run.

It is that a simple change can have unexpected consequences in different parts of the application.

4.3. Improvised coding on top of existing systems

Imagine you already have a CRM that manages customers and invoicing.

Someone decides to improve the platform with AI and requests a new feature. The tool proposes changing the database, replacing components and reorganising internal structures.

The feature appears on screen, but you later discover that historical reports no longer reconcile, some automations have stopped working and certain users can see information they should not have access to.

This happens when local changes are made without assessing their effect on the system as a whole.

In the industry this is linked to regressions: a new change causes something that used to work to stop working.

That is why regression testing exists.

4.4. Poor software architecture

Architecture is the structure that defines how a system's components are organised and how they relate to one another.

Good architecture lets you change parts of the application without having to rebuild it from scratch.

Poor architecture can work for a long time but become a constraint as the company grows.

Some common problems are:

Artificial intelligence can produce any of these problems when used without an overall architectural vision.

It can also help detect and fix them when used within a proper review process.

4.5. Apparent security instead of real security

One of the most dangerous mistakes is assuming an application is protected because it has a login screen.

A login form does not guarantee that data is protected.

For example, if a customer views their invoice through a URL containing an identifier and, by changing it, gets someone else's invoice, there is an authorisation flaw.

In information security, this problem can be described as broken access control and, in certain cases, IDOR (insecure direct object reference).

The 2025 edition of the OWASP Top 10 identifies broken access control as one of the most important risks for web applications.

Other common mistakes include credentials hard-coded into the source, excessive permissions, queries vulnerable to injection, insecure password storage, exposed configuration and dependencies with known vulnerabilities.

Having HTTPS, passwords and a modern server does not, on its own, prove that an application is secure.

4.6. Privacy and data leakage

An application can work correctly and, at the same time, handle personal data inappropriately.

This happens, for example, when:

• Customer records are sent to external AI services without checking the processing terms.
• Real data is used for unnecessary testing.
• Technical logs contain passwords, documents or personal information.
• Employees can access data they do not need to do their jobs.
• Data is kept for longer than necessary.
• Integrations transfer information to suppliers that have not been assessed.

An enterprise subscription to an AI model does not, in itself, constitute a universal guarantee of regulatory compliance.

You need to know what information is sent, who receives it, where it is processed, under what conditions it is retained and what contractual and technical controls are in place.

4.7. Lack of stability and capacity to grow

An application can respond correctly with five users and fail with five hundred.

The term scalability describes its capacity to handle growth in users, transactions or data.

Reliability describes how consistently it works.

Resilience describes its capacity to withstand failures and recover.

They are related properties, but different ones.

An application can be fast but unreliable, secure but slow, scalable but far too expensive, or stable until an external provider fails.

A professional solution must strike a balance between these characteristics.

4.8. Lack of testing and validation

When someone develops exclusively through prompts, they may only check that the screen looks right.

But there are many other questions:

What happens if the user enters incorrect information? If they make the same payment twice? If the connection drops? If two employees edit the same record at the same time? If a certificate expires or an API fails?

Tests exist to answer these questions before the problem affects real customers.

Terms such as unit testing, integration testing, end-to-end testing, regression testing, load testing and security testing are used.

Each one assesses different aspects of the system.

A test generated by AI is not automatically reliable either. It may contain wrong assumptions, validate false results or simply confirm the same error that is present in the code.

4.9. Dependence on one person, platform or provider

This is one of the most important commercial risks.

Imagine someone builds your application with an AI platform, but does not hand over the source code, you do not control the repository and all the configuration sits in their personal account.

While that person is working with you, the system seems to be yours.

When they stop responding, you discover that you cannot change it, migrate it or easily hire another team.

This problem is known as vendor lock-in when there is dependence on a provider or technology, and key-person risk when continuity relies too heavily on one individual.

The risk does not disappear because AI is used. It can even intensify when there is no technical documentation either.

4.10. Incorrect data that looks true

Language models can produce plausible but incorrect output, a phenomenon known as hallucination.

Within a business application, the problem can take various forms.

A customer service system could invent commercial terms. An accounting automation could misclassify a transaction. An assistant could claim it processed a file when it did not.

The risk increases when an AI-generated response is taken as confirmation that an operation was actually carried out.

For that reason, a model's response should never, on its own, replace verifiable confirmation from the system that actually executes the operation.

5. The additional risks when AI also operates your system

So far we have mainly discussed using AI to develop software.

But there is another scenario: applications in which artificial intelligence remains embedded and continuously makes decisions or carries out actions.

For example, an agent that can read emails, generate invoices, manage documents, update a CRM or modify records.

This is where additional risks appear.

Prompt injection: occurs when malicious instructions introduced through messages, documents, websites or other sources improperly alter a model's behaviour. For example, an agent tasked with reading a document may receive hidden instructions designed to change what it does.

Excessive agency: happens when an agent has too many capabilities, permissions or too much autonomy. A tool that only needs to read information should not automatically have permission to delete it.

Data exfiltration: the unauthorised extraction or disclosure of information. It can occur through misconfigured integrations, excessive permissions or attacks on the tools the agent uses.

RAG poisoning: the manipulation of content used as a knowledge source by a retrieval-augmented generation (RAG) system. If the source contains false or malicious information, it can contaminate the answers.

Unbounded consumption: occurs when resources are consumed without appropriate limits, which can lead to unexpected bills for model calls, external services or compute capacity.

Automation bias: the human tendency to over-trust the recommendations of an automated system, even when there are reasons to question them.

OWASP covers several of these risks, including prompt injection, excessive agency and unbounded consumption, in its 2025 Top 10 for applications that use large language models (LLMs).

Defence is not just a matter of writing better instructions.

It requires controls that are independent of the model: real authorisation, environment separation, consumption limits, input and output validation, monitoring, traceability and human approval for critical actions.

A. From the perspective of someone who wants to integrate artificial intelligence into their current system

6. You want to modernise your company with AI. Where should you start?

Imagine your company has been working for years with an ERP, CRM, website, sales platform or back-office system.

You probably want to automate tasks, cut turnaround times, connect tools or add intelligent assistants.

That can make a lot of sense.

But before introducing AI, you need to identify exactly what problem you are trying to solve.

Automating email sorting is not the same as allowing an agent to change invoices, approve payments or edit customer information.

First, identify the process you want to improve

A good integration starts with the business process, not the tool.

For example, if you need to automate invoice capture, you should know how invoices arrive, what information they contain, who validates them, which system records them, how errors are resolved and what documentation must be kept.

Once this process is documented, you can identify which part is worth automating.

AI might be useful for extracting information from unstructured documents, while tax operations and calculations remain under deterministic rules.

This lets you take advantage of AI without giving it responsibilities it does not need.

7. Four ways to bring AI into an existing system

These levels are indicative. An assistant that exposes extremely sensitive information could also represent a critical risk.

Do you need to replace your current system?

In many cases, no.

A common mistake is wanting to completely rebuild a working platform just because it is now possible to generate a new one with AI.

If your current system correctly processes your company's operations, it is probably better to look first at integration through APIs, connectors, external components or controlled automations.

That way you build on your existing investment and reduce the risks of a full migration.

For example, you can keep your ERP and use AI to interpret documents, assist operators or detect inconsistencies, without allowing the model to rewrite the accounting logic.

What should you require before commissioning an integration?

What matters is that the provider can give you evidence of what they will change, how they will protect your information and how they will resolve any failure.

Before starting, try to agree on these points:

1. Scope: what the AI will do and which decisions will continue to depend on people or verifiable rules.
2. Architecture: how it will connect to current systems and which components will be affected.
3. Data: what information it needs, where it will be sent and how it will be protected.
4. Permissions: what it can read, create, modify or delete.
5. Testing: how it will be shown to work in normal and exceptional situations.
6. Reversibility: how the integration will be switched off or removed without harming the core system.
7. Ownership and documentation: what you will receive so that another provider can maintain it.
8. Costs: how much it will cost to develop, use, monitor and maintain.

If someone cannot explain these points to you in plain terms, you do not yet have enough information to assess the project.

8. When is artificial intelligence not the right choice?

Not everything needs AI.

If you want to calculate a tax using a legally defined formula, using a language model to improvise the calculation adds uncertainty.

If you want to check whether a customer has paid an invoice, the reliable source must be the bank record or the authorised system, not an assistant's verbal interpretation.

If you need to determine whether a transaction meets an unambiguous rule, programmed and verifiable logic is generally the better option.

AI adds the most value when you need to interpret language, classify complex information, process varied documents, summarise content, support supervised decisions or interact with people in natural language.

The most suitable architecture usually combines both technologies:

AI to interpret, assist and propose; deterministic software to validate, authorise and execute critical operations.

This is known as a hybrid architecture.

A well-designed integration should let you work more efficiently without losing control of your original processes.

B. From a technical director's perspective: is the project viable and how much risk does it carry?

9. A technical director's job is no longer just to assess whether something can be built

A technical director, CTO or head of technology needs to answer different questions from those usually asked by whoever requests an application.

The business wants to know whether it can be built.

The technical lead needs to know whether it can be built with an acceptable level of risk, a sustainable cost and guarantees of continuity.

Artificial intelligence significantly changes this assessment because it lowers some barriers to producing code, but it does not remove engineering responsibilities.

You can receive in days an application that would previously have required a much longer process. However, you can also receive thousands of lines of code with no documentation, no test coverage and no justified architectural decisions.

Speed of generation is no longer the only relevant indicator.

10. The six assessments that determine technical viability

10.1. Functional viability

Does the system really solve the business problem?

A visual demo is not enough.

You need to verify that business rules are met, calculations are correct, data keeps its integrity and processes work end to end.

A system can pass a sales demo and fail when it meets real transactions.

For example, an invoicing program must correctly handle discounts, cancellations, taxes, credit notes, duplicates, payment statuses and exceptional situations.

Each operation represents a rule that must be defined and verified.

10.2. Architectural viability

Does the structure allow it to be maintained and to grow?

This covers components, databases, dependencies, integrations, modularity, design patterns and the capacity to add future features.

There is no universally correct architecture.

A small system might be better served by a well-organised modular monolith than by an overly complex distributed architecture.

The decision must respond to the problem and its real needs.

10.3. Security viability

Can you demonstrate that adequate controls are in place?

You need to consider identity management, authentication, authorisation, encryption, dependencies, secrets management, activity logging, data protection and incident response.

The industry uses different types of testing:

These measures are complementary. None of them, on its own, guarantees that software is secure.

The NIST SSDF framework provides practices for building security into the development life cycle, regardless of whether the code was written by a person or generated with the help of AI.

10.4. Operational viability

Can it be kept running under real conditions?

You need to find out how it is deployed, monitored, updated and recovered.

A highly automated system without sufficient logging can be especially difficult to diagnose when it fails.

This is where practices such as DevOps, SRE (site reliability engineering) and observability come in.

Observability lets you understand what is happening through metrics, logs and traces.

It is not enough to know that the application has stopped working. You need to know which component failed, when it happened, which operations it affected and how to recover it.

10.5. Economic viability

What does the system really cost over its lifetime?

This is one of the most underestimated aspects.

An AI-generated application can have a very low initial cost and later require significant investment in:

• Fixes.
• Security and audits.
• Infrastructure.
• API and AI model usage.
• Maintenance and updates.
• Support.
• Partial or complete rewrites.
• Data recovery.
• Regulatory compliance.
• Training and documentation.

The management concept used here is total cost of ownership (TCO).

It can be expressed as:

TCO = development cost + operating cost + maintenance cost + cost of risk + exit cost

The cost of risk should be estimated using reasonable scenarios and probabilities; it does not necessarily represent spending that will actually occur.

A serious assessment considers a horizon of several years, not just the cost of producing the first version.

10.6. Continuity viability

What happens if the developer, the platform or the provider disappears?

The answer depends on whether the company controls its technology assets.

Ideally, you should be able to access the source code, databases, infrastructure, company credentials, contracts, documentation and recovery procedures.

You should also check that backups can actually be restored and that there is a realistic alternative for keeping operations going.

Having backup files does not necessarily mean being able to recover the business.

11. How does a CTO decide whether to fix, restructure or rebuild?

Not all deficient software needs to be thrown away.

A technical review can lead to different decisions.

There is no formula for deciding simply by counting lines of code or vulnerabilities.

You need to weigh the cost and risk of keeping the current system against those of rebuilding it, including the risks of migrating data and disrupting processes that do work.

A complete rewrite can be more damaging than gradual refactoring when the system contains business rules that nobody has documented.

The legacy code problem

Legacy code does not necessarily mean old or bad software.

In practice, it often refers to inherited systems that need to be maintained even though their structure, documentation or tests are inadequate.

AI is introducing a particular variant: newly developed software that is already hard to understand or change.

We could informally describe it as new software with the maintenance problems typical of legacy systems.

It is not a formal category, but it is a situation engineering teams need to recognise.

12. Indicators a technical director should track

The number of features generated or lines of code produced is not enough to measure productivity.

It is better to use indicators that reflect real outcomes.

Test coverage alone does not prove quality; you can have high coverage with poor tests.

Sustainable productivity must be assessed in terms of time, quality, stability, security and value delivered.

13. Can a vibe-coded project be brought up to professional standards?

Yes, provided the necessary assets can be obtained and its condition is established technically.

Depending on the context, the process is known as codebase hardening, application modernisation, software remediation or technical debt reduction.

It can include an architecture audit, a security review, building tests, reorganising the code, documentation, separating environments and putting maintenance procedures in place.

But it is not enough to ask another AI to review everything and report back that it is fine.

An automated review can help identify defects; important conclusions must be checked through testing and independent evidence.

The goal is not necessarily to get rid of all AI-generated code.

It is to turn it into a business asset that can be understood, verified and maintained.

C. From a legal and accounting perspective: benefits, responsibilities and financial risks

14. Why can a company face legal consequences even if the problem was created by AI?

The involvement of an artificial intelligence tool does not automatically remove the company's obligations.

Imagine you use an AI-built platform to store customer documents.

If it has a vulnerability that allows access to confidential information, the legal question is not limited to who wrote the code.

It also matters who determined the purposes of the processing, who was responsible for protecting the data, what contractual obligations existed and what reasonable measures were implemented.

AI is not a general excuse for legal non-compliance.

Responsibilities depend on the applicable legislation, the role of each party, the contracts and the circumstances of the incident.

15. Data protection: GDPR (RODO) in the European Union

If your company operates in Poland or processes personal data within the scope of the General Data Protection Regulation, you need to consider the GDPR, usually referred to in Polish as RODO.

The regulation is technology-neutral: it does not disappear or shrink simply because you use artificial intelligence.

What should you review?

Lawful basis and purpose. You need to identify why you are allowed to process certain data and what it will be used for. Being technically available does not mean you can use it freely.

Minimisation. The application should collect and process only the data needed for the relevant purpose.

Controllers and processors. You need to identify who decides the purposes and means of processing and who processes information on behalf of someone else.

Data processing agreements. Where applicable, there must be appropriate agreements with suppliers or processors, including services that receive information through AI integrations.

International transfers. You need to assess whether information is transferred outside the European Economic Area and which legal mechanisms are required. A provider having servers outside Europe does not automatically mean an infringement, but it does require analysis of the processing and its safeguards.

Security. Technical and organisational measures proportionate to the risk are required, not merely statements that the software is secure.

Impact assessment. If processing is likely to result in a high risk to people's rights and freedoms, a DPIA (data protection impact assessment) may be required.

What if AI has already caused a data breach?

You need to assess whether a personal data breach occurred, what information was affected, what risks exist for the individuals concerned and which notification obligations apply.

Under the GDPR, where a breach poses a risk to people's rights and freedoms, the controller must notify the competent supervisory authority without undue delay and, where feasible, within 72 hours of becoming aware of it. Communication to the individuals affected depends, among other factors, on whether there is a high risk.

Not every IT error automatically triggers that obligation.

16. AI Act: is an AI-generated application regulated as an AI system?

This is a common source of confusion.

A system developed using AI is not the same as a system that incorporates artificial intelligence into how it works.

A conventional application whose code was generated by a coding assistant does not automatically become an AI system subject to all the obligations of the AI Act.

Classification depends on the application's characteristics, its purpose and the relevant legal definitions.

In Europe, the AI Act sets different obligations depending on the type of system, the risk and the role of the organisations involved.

Much of the regulation already applies, although there are different timelines for certain obligations. Following the AI Omnibus changes (Regulation (EU) 2026/1744, published in July 2026), some obligations for high-risk systems apply later: from 2 December 2027 or from 2 August 2028, depending on the category.

This means you need to assess each project, especially when AI is involved in recruitment, access to services, the evaluation of individuals or other legally sensitive activities.

It is not enough to look at the name of the tool or ask whether it uses a language model.

17. Other regulations that may affect business software

Besides the GDPR and the AI Act, there are rules concerning the security and continuity of certain services.

NIS2 and DORA do not automatically apply to every small business that uses AI. Applicability must be assessed according to the activity, sector, size, services provided and the relevant rules.

In the case of the CRA, reporting obligations for manufacturers of products within its scope began on 11 September 2026, while its main requirements are due to apply in general from 11 December 2027. You should not assume that every in-house development is automatically covered.

A company that places software on the market without sufficient controls could face compliance problems depending on its characteristics, even if it was built entirely with AI tools.

18. Intellectual property and contractual liability

Software is not just a collection of instructions you can copy from one place to another.

You need to consider copyright, licences, ownership of deliverables, use of external components and contractual obligations.

Possible problems include components copied or incorporated under incompatible licences, dependencies that have not been identified, no agreement on ownership of deliverables and contractual restrictions on the use of certain services.

The existence of AI-generated code does not automatically settle who can exploit it commercially, nor does it guarantee that all of its content is free of third-party rights.

These questions may vary by country and by the circumstances of the development.

What should a development contract set out?

The contract documentation should clearly define:

• Which features and results will be delivered.
• Who will have access to the source code and repositories.
• What rights of use and exploitation are granted.
• How third-party licences will be managed.
• Which security and quality standards must be met.
• Which tests determine acceptance of the system.
• How incidents and vulnerabilities will be handled.
• What confidentiality and data protection obligations exist.
• What support and documentation are included.
• What will happen when the business relationship ends.

It can also be useful to agree on periodic code delivery, transition procedures and, in certain contracts, software escrow (a source code deposit).

The aim is to prevent the company from losing control of the software it needs to operate.

19. From an accounting perspective: an incorrect invoice can cause bigger problems than a visual glitch

An accounting system needs accuracy, consistency and traceability.

Artificial intelligence can be very useful for interpreting receipts, extracting information, classifying documents, detecting anomalies or preparing reconciliations.

But the benefits shrink when you let the model freely decide matters that should follow verifiable tax and accounting rules.

For example, a system may correctly extract a supplier's name but misread the applicable tax rate.

It can also record the same invoice twice, allocate payments to the wrong customers or create discrepancies between the sales system and the accounts.

In Poland, an integration with KSeF (the National e-Invoicing System) must meet the relevant technical and tax requirements. An AI producing a document that looks like an invoice does not mean that document complies with the required invoicing procedure.

How does AI benefit an accountant?

Used with supervision and validation, it can help reduce manual data entry, speed up reconciliations, identify unusual transactions, organise supporting documentation, prepare draft reports and make it easier to search large volumes of information.

The accountant can spend more time reviewing inconsistencies and less on repetitive tasks.

How can it cause harm?

The risks appear when automations alter data, generate incorrect entries, lack traceability or create documents that do not correspond to real transactions.

An error can spread from invoicing into financial reports, tax returns and business decisions.

For that reason, the system must preserve the link between the original documents, the recorded transactions, the authorisations and any subsequent changes.

This capability is known as an audit trail.

Particularly important controls

A financial system should keep verifiable evidence of who authorised a transaction, which document supports it, what changes were made and whether the record matches the relevant sources.

It should also implement reconciliations, duplicate-prevention rules, access controls and segregation of duties.

Segregation of duties prevents a single person or process from improperly combining actions such as creating a supplier, changing their bank details and approving their payment.

Nor should an AI agent automatically be given all of those powers.

20. The financial benefit of fixing an AI-built system

Recovering an application may initially look like an extra expense.

But from a financial point of view, it is worth analysing the cost of doing nothing.

A deficient system can cause losses through outages, transaction errors, excessive support, lost customers, breaches of contract or the need for a future rebuild.

The benefit of an intervention can be estimated as:

ROI = (attributable net benefits − cost of intervention) ÷ cost of intervention × 100

For the calculation to be useful, the benefits must rest on explicit assumptions and avoid counting the same savings twice.

The most important financial indicators include cost per transaction, support hours, losses from errors, the impact of outages, maintenance costs and the expected cost of replacement.

A fix can make economic sense even if it adds no visible functionality, provided it demonstrably reduces future risk and cost.

Security, stability and continuity also have financial value.

D. From the perspective of someone who needs to repair what artificial intelligence broke

21. Your system was working, someone changed it with AI and now you have a problem. What can you do?

This scenario is increasingly relevant for companies that depend on software developed or modified quickly.

Perhaps you hired someone to build an application and found that they cannot fix the errors that keep appearing.

Perhaps an employee used AI to change a system and ended up affecting important data.

Or perhaps you developed a tool that worked perfectly at first but, after many changes, now has faults that not even the AI itself can fix.

You may also have a working application but have lost the person who built it, the documentation or access to the services it needs.

In all these cases, the first goal should not be to add more features.

The priority is to restore stability, protect your data and regain control over the components your business depends on.

Signs that you need a technical intervention

These symptoms may indicate that the system needs an assessment:

• Every new change breaks existing features.
• The same fault reappears after being fixed.
• The developer relies entirely on AI to explain or repair the code.
• Nobody can describe how the application is organised.
• You do not know where data is stored or who has access to it.
• Changes are made directly on the system your customers use.
• Infrastructure or AI usage costs are rising unexpectedly.
• There are discrepancies between records, reports and actual transactions.
• You cannot reliably export all your data.
• The application only works from one person's or one provider's account.
• There are no tested backups or recovery procedures.
• The AI claims to have fixed problems that still exist.

These symptoms do not, on their own, prove that AI is the cause. They can also result from conventional coding errors, infrastructure or administration.

The assessment must identify the real source before assigning responsibility.

22. First: tell a defective application apart from a critical incident

Not every problem needs the same treatment.

A visual glitch can be fixed locally. An altered database requires far more careful procedures.

23. How to regain control of an AI-built application

A professional recovery process is usually divided into several stages.

Recovery and stabilisation process:

1. Containment: stop uncontrolled changes, restrict dangerous access and prevent further data alteration.
2. Preservation: obtain consistent copies of affected systems, logs, configuration and relevant evidence.
3. Diagnosis: identify the faults, their source, the extent of the damage and critical dependencies.
4. Recovery: restore verifiable data and fix the problems that prevent operation.
5. Stabilisation: add testing, security, observability, backups and controlled deployment procedures.
6. Handover of control: document the system, formalise ownership of assets and assign maintenance owners.

Stage 1. Contain before making further changes

When a system is failing, it is common to try to fix it by giving the AI new instructions.

The catch is that each additional change can make diagnosis harder.

If you suspect data loss or a serious vulnerability, dangerous operations must be restricted, especially automated actions capable of deleting or altering information.

Where necessary, certain integrations can be temporarily disabled or features switched to read-only mode.

Containment must take business continuity into account and be carried out without destroying evidence.

Stage 2. Preserve existing assets

Before starting repairs, it is advisable to obtain consistent, verified copies of the current state, where it is safe to do so.

This can include source code, databases, activity logs, configuration, dependencies, infrastructure information and change logs.

In a formal investigation it may be necessary to retain metadata, document the chain of custody and use integrity mechanisms such as cryptographic hashes.

If there may have been an intrusion, you should not assume that the current state is clean or that all logs are reliable.

The aim is to keep the ability to investigate and reconstruct what happened.

Stage 3. Regain administrative access

A company needs to establish who actually controls its platform.

Knowing the admin panel password is not enough.

You may also need to recover ownership or administration of:

• Domain and DNS.
• Code repositories.
• Servers and cloud services.
• Databases and storage.
• Email services.
• APIs and AI platforms.
• Payment and invoicing systems.
• Certificates, keys and credentials.
• Backups and recovery mechanisms.

Compromised or improperly shared credentials must be rotated through a controlled procedure.

An AI tool should not retain administrative permissions it no longer needs.

Stage 4. Carry out an independent diagnosis

A rescue audit needs to assess the software as a complete system.

It should not only answer which line of code is causing an error.

It must also establish whether the architecture is recoverable, what the dependencies are, what risks are present, whether data has kept its integrity and which components need intervention.

A good diagnosis should distinguish between confirmed problems, suspicions still to be tested and potential risks.

It would not be responsible to declare an application insecure simply because it contains AI-generated code.

Nor would it be responsible to declare it secure because a scanner found no vulnerabilities.

Stage 5. Recover data and critical functions

Recovery begins after diagnosis.

Backups, transaction logs, previous versions, exports and information from independent sources can be used to reconstruct or verify data.

It is important to distinguish a technically successful restore from a commercially correct one.

For example, you might recover a database that opens again but does not contain your customers' latest transactions.

In that scenario you need reconciliation and validation before considering the process complete.

Recovery procedures should preferably be carried out in isolated, controlled environments before any changes are made in production.

Stage 6. Stabilise and document

Once operations are back, the company needs to prevent the same errors from recurring.

This includes establishing an understandable architecture, security controls, automated and manual testing, deployment procedures, monitoring, backups and documentation.

It must also be defined who authorises changes and what evidence is needed before they are released.

A successful rescue does not mean the application opens again.

It means your company can use it, change it and recover it without relying on improvisation.

24. What if you do not have the source code, the backups or access to the provider?

This is one of the most complex scenarios, but it does not always mean a permanent loss.

The options depend on which assets are still available.

It is not always possible to fully recover a system.

If there are no backups, accessible data, logs or other sources of information, some losses may be irreversible.

It is better to establish that limitation through technical evidence than to promise an impossible recovery.

Can a system be rebuilt just by observing how it works?

Sometimes, yes.

The technique is related to reverse engineering and to reconstructing requirements from observable behaviour.

You can document screens, workflows, reports, formats, business rules and integrations in order to develop a replacement.

However, a visual reconstruction does not guarantee that the original logic or historical data will be recovered.

Intellectual property rights, contracts and other applicable restrictions must also be respected.

25. How to tell whether you really need a software recovery specialist

When an application fails, it can be tempting to hire someone who promises to rebuild it entirely with a new AI tool.

But replacing a solution without understanding the problem can trigger a second crisis.

The profile you need depends on the situation.

A visual bug could be fixed by an application developer.

A vulnerability requires security expertise.

Data loss may call for database, recovery or digital forensics specialists.

A business system with multiple integrations may require architecture, software engineering, infrastructure and knowledge of the business rules.

What should a competent provider be able to demonstrate?

You do not need a provider who rejects artificial intelligence.

You need one who can explain how they use it, how they validate its output and what responsibilities they take on.

A professional should be able to justify their architectural decisions, describe risks, provide test evidence and acknowledge the limits of their diagnosis.

They should also be able to explain the project to you without relying on vague claims such as their AI being the most advanced or any defect being fixable automatically.

Warning signs when hiring

Be wary of proposals that promise absolute security, a total absence of errors, guaranteed recovery of any data, automatic legal compliance or no need for maintenance.

It is also worth looking more closely when the provider does not give access to the code, does not explain where it stores data, avoids documenting changes or does not set verifiable delivery criteria.

Certifications can back up specific knowledge, but they do not replace demonstrable experience, testing or contractual commitments.

26. Glossary: the terms you are likely to hear when trying to rescue your software

These concepts will help you interpret a technical diagnosis and understand proposals from different providers.

These concepts are not synonyms, nor do they represent quality certifications on their own. They are tools for describing problems, processes and controls.

27. The most common arguments about AI and software development

"AI codes better than many developers"

For certain tasks, an AI tool can produce excellent solutions and even outperform some professionals in speed or accuracy.

But that does not prove it can broadly replace the work of architecture, analysis, security, testing, integration and business accountability.

Quality depends on the task, the model, the information available and subsequent validation.

The test should not be whether AI is better than a particular person.

It should be whether the process as a whole produces a sufficiently reliable result.

"If the program works, why pay for an audit?"

Because not all defects are visible during normal use.

An authorisation flaw could stay hidden until someone accesses another person's data.

A poorly designed database can work until it grows large enough.

An inadequate backup policy may only come to light when you need to restore data.

An audit aims to identify risks that a superficial test does not reveal.

"Artificial intelligence reduces development costs"

It can reduce them significantly in certain phases.

But savings must be calculated across the whole process.

If generating a feature costs less but fixing, testing and maintaining it requires extra effort, the initial saving can shrink or disappear.

The opposite can also happen: a company with good engineering can use AI to cut costs without compromising quality.

That is why results must be measured.

"Anyone can create professional software with AI"

The new tools make it vastly easier to get into development.

This can generate valuable innovation and allow people without a traditional background to create useful tools.

However, having the ability to generate code is not automatically the same as mastering architecture, security, data modelling, testing or the applicable legislation.

The difference should be judged by verifiable skills and results, not solely by academic qualifications.

"What one AI breaks, another AI can fix"

Sometimes that is true.

AI can locate errors, identify patterns, explain components and suggest fixes.

But when the problem involves corrupted data, missing information, inconsistent architecture or compromised infrastructure, additional actions and checks are needed.

Asking for a fix and getting a convincing answer does not prove the problem has been solved.

"If I use a private model, all my privacy problems go away"

Not necessarily.

A private model can offer advantages in control and data processing, but you still need to secure access, logs, databases, integrations, backups and the infrastructure it runs on.

You also need to check what private actually means under each provider's terms.

"Open source is always secure and free"

Open-source software can be very valuable, auditable and cost-effective.

However, its security depends on factors such as maintenance, configuration, known vulnerabilities and access controls.

You also need to consider its licences and operating costs.

Being able to use a component without paying for a licence does not mean running and maintaining it is free.

"Starting again from scratch is cheaper"

It can be when the existing system has serious structural defects.

But rebuilding also means recovering business rules, migrating data, maintaining compatibility and reintroducing critical processes.

In some projects it is better to replace only specific components and keep the rest.

The decision must be based on an analysis of cost, risk and continuity.

"One AI can review the code another AI wrote"

Yes, and it can be useful.

But there is a risk that both share incorrect assumptions or miss important defects.

That is why a model-based review does not replace independent testing, security analysis or checking real-world results.

"If the system is hosted with a cloud provider, I already have security and backups"

Not necessarily.

The infrastructure may include security measures while the application still has vulnerabilities of its own.

In addition, backup and restore responsibilities vary between providers and contracts.

You need to check what is backed up, how long it is kept, what would happen if data were deleted and how restores are verified.

28. What should a professional software rescue and recovery project look like?

A company that needs to recover a system usually benefits from a project divided into phases, with verifiable deliverables.

The assessment should establish which problems were identified, which were fixed, which remain outstanding and which residual risks the company accepts.

This avoids confusing a recovery project with an endless series of changes.

29. How do you stop it happening again?

The sustainable solution is not to ban all artificial intelligence.

It is to introduce a usage policy proportionate to the risk.

An organisation can use AI to write code, generate tests, assist support or automate processes while also setting strict limits where there are financial, operational or legal consequences.

The NIST AI Risk Management Framework organises this work around four functions: govern, map (identify the context and risks), measure and manage.

In practical terms, it is worth establishing five fundamental rules.

First: no critical change without version control and approval. It must be possible to identify what was changed, who authorised it and how to roll it back.

Second: no agent should have more permissions than it needs. A read-only automation does not need general write or delete permissions.

Third: nothing should go into production just because it looks right. It must pass functional, security and integration tests proportionate to the risk.

Fourth: no critical asset should depend solely on a personal account or a single provider. The company needs to keep contractual and operational control of its systems.

Fifth: no critical operation should be considered successful just because the AI says it carried it out. There must be evidence in the target system.

These rules are useful both for traditional developers and for teams using AI assistants and agents.

30. Quick check: how exposed is your company?

The following self-assessment is indicative only. It does not replace a security audit or a compliance assessment.

Answer "Yes", "No" or "Don't know" to each statement. Every "No" or "Don't know" flags an area worth reviewing; "Don't know" matters as much as "No", because it shows a lack of visibility over an asset or a control.

Example interpretation: a company answers "Yes" to statements 1, 3, 4, 5 and 8, "No" to 2 and "Don't know" to 6 and 7. It has three areas to address: untested backups, a lack of evidence for critical operations and insufficient documentation to change provider. Those three areas are the starting point for its review plan, beginning with backups, because without them any other intervention is riskier.

31. The future of AI-assisted development: speed with accountability

Artificial intelligence is making code production increasingly accessible.

This is good news for many companies: it allows them to experiment, automate processes, cut repetitive tasks and create tools that used to be too expensive.

However, it is also changing where many technology risks originate.

For a long time, one of a company's main difficulties was getting someone to build the software it needed.

Now the opposite can happen: quickly obtaining software that looks finished but lacks the controls needed to trust it.

The challenge is no longer just building.

It is demonstrating that what has been built is correct, secure, maintainable and fit for the business.

That is why architecture, testing, security engineering, observability, data governance, technical audits and recovery mechanisms are becoming more important.

Artificial intelligence can contribute to all of these activities, but it should not remove the responsibility for verifying them.

Conclusion: can what AI broke be repaired?

In many cases, yes.

An unstable application can be fixed. An undocumented system can be analysed. A poor architecture can be modernised. A badly designed integration can be replaced. Even a project built almost entirely through prompts to an AI can be turned into a professional system.

But it is not always worth saving all the code, and not all lost data can be recovered.

What matters is establishing which assets exist, which risks are real, what information can be recovered and which strategy offers the best balance between cost, security and continuity.

The answer is not necessarily to abandon artificial intelligence, nor to use even more of it.

It is to restore the engineering discipline that makes software reliable.

AI can help you build an application. Engineering, security and sound business management are what turn it into a system you can depend on.

And when that trust has been lost, the first goal is not to develop faster again.

It is to regain control.

Sources

• Stack Overflow Developer Survey 2025 — AI
• Stack Overflow Developer Survey 2026
• Stack Overflow Developer Survey 2026 — AI (data)
• Veracode — 2025 GenAI Code Security Report
• DORA — 2025 State of AI-assisted Software Development
• METR — Early-2025 AI and experienced open-source developer productivity (2025)
• METR — Uplift update (2026)
• OWASP Top 10:2025
• OWASP Top 10 for LLM Applications 2025
• GDPR — Regulation (EU) 2016/679
• Digital Omnibus on AI — Regulation (EU) 2026/1744
• European Commission — CRA reporting obligations
• NIST AI Risk Management Framework
• NIST Secure Software Development Framework (SSDF)