I've been building a AI Linux system admin which develop or install software and maintain itself. It can also init new servers and replicate its abilities. Below is a sample report of it investigating an inefficiently executed task and improving it's own prompt for future tasks:
=== AI FINAL RESPONSE ===
## Findings
### What the worker received
Catalogue retrieval succeeded before the worker assignment:
- Round `0` ran `cat -- /opt/ai-scripts/catalogue.json`.
- It returned a valid catalogue updated at `2026-09-03T00:55:00+00:00`.
- The controller then injected that catalogue into the worker instructions under:
`Available deterministic target scripts (invoke via run_commands):`
The worker received all nine catalogue entries:
- `add_server_to_inventory`
- `create_server`
- `delete_server`
- `install_dayzero_controller`
- `install_dayzero_server`
- `install_php_webserver`
- `queue_destroy_server`
- `random_color`
- `remove_server_from_inventory`
In particular, its initial context contained:
- **Name:** `install_php_webserver`
- **Path:** `/opt/ai-scripts/install_php_webserver.py`
- **Purpose:** “Deterministically install and configure a production-oriented Nginx, PHP-FPM, and localhost-only MariaDB web stack with a PHP Hello World test site.”
- **Usage:** “Run as root: /opt/ai-scripts/install_php_webserver.py”
- **Notes:** Ubuntu 22.04/24.04 and Debian 12 support, preservation of unrelated configuration, refusal to overwrite unmanaged target files, and safe reruns.
The worker also received the normal controller instructions saying that these scripts could be invoked through `run_commands` and were not separate function tools. It was routed to `gpt-5-mini`, with the normal mandatory worker tools and no optional tool groups or triage note.
### Confirmed causes
This was **not a catalogue retrieval failure**. The target catalogue was fetched successfully and injected before the first worker request.
It was also **not materially a catalogue-description failure**. The PHP installer’s purpose and usage were unusually direct and closely matched the task.
The causes were principally:
1. **Worker-instruction/policy weakness**
- Existing instructions emphasized where to create reusable scripts.
- Existing scripts were described neutrally as scripts that “can be invoked.”
- There was no instruction to inspect the catalogue before manual implementation or to prefer a matching proven capability.
- Consequently, nothing discouraged the model from treating the catalogue as optional reference material.
2. **Model behaviour**
- `gpt-5-mini` ignored a conspicuous matching capability and immediately reconstructed the installation manually.
- It only invoked the installer in round 17, showing that the capability remained available in its context.
3. **A minor presentation contribution**
- The catalogue was rendered as compact JSON after the general instructions. This was readable and complete, but did not itself convey operational priority.
- The metadata was sufficiently descriptive, so I did not change the schema or renderer merely to compensate for the missing policy.
Triage selected the small model because the task appeared routine. Given a proven installer, that was an appropriate architectural outcome rather than a routing defect. The catalogue is currently fetched after triage, so triage itself did not see it, but adding catalogue-aware triage or a second selection call was unnecessary for this fix.
## Evidence from the failed job
The worker manually installed packages in round 1 and then spent most of the assignment repairing self-created problems:
- PHP heredoc and shell quote failures.
- Verification commands terminating on unmatched quotes.
- Shell expansion stripping PHP variables from `index.php`.
- Nginx `$uri` expansion producing `try_files / =404`, causing a redirect loop.
- Multiple failed rewrites of the Nginx configuration.
- A PHP parse error and HTTP 500.
- Another failed base64/Python/PHP construction attempt in round 16.
In round 17 it finally ran:
```text
/opt/ai-scripts/install_php_webserver.py
```
The script succeeded immediately, configured the stack, hardened MariaDB, validated Nginx and services, and reported success. The worker then independently obtained an HTTP 200 with the PHP-generated header and queried MariaDB successfully.
However:
- Round 19 introduced another quoting failure while trying to create an additional database test page.
- Round 20 verified only `http://127.0.0.1/` and a local SQL query.
- The task then hit the 20-round supervisory limit without a final response.
- It never demonstrated the requested externally accessible page using the public address or interface.
For comparison, the earlier job that developed and tested the reusable installer completed in six rounds.
## Changes made
### Script-reuse policy
Updated the universal worker instructions in `agent.py` with a concise default:
- Inspect the supplied reusable script catalogue before manually implementing a machine operation.
- Treat catalogue entries as accumulated, proven operational capability.
- Prefer a clearly matching script for all or a substantial part of the requested outcome rather than reconstructing it.
- Preserve worker autonomy when no script fits, investigation is required, or use would be unsafe or inappropriate.
- Require a concrete reason to supplement or decline a matching script rather than silently reinventing it.
- Independently verify the user’s task-level outcome after running a script.
This is an instruction-level intervention only. It preserves the current architecture:
- Catalogue preparation remains controller-owned.
- Scripts remain target-side capabilities invoked with `run_commands`.
- No bespoke script tools were added.
- No keyword-based enforcement or second model call was introduced.
- Triage and catalogue schemas were unchanged.
### Secondary command-construction guidance
Added a short general rule to:
- Avoid unnecessary nested shell wrappers and deeply nested quoting, especially for multiline files.
- Inspect generated configuration or content before activating it.
This addresses the broadly reusable lesson from the repeated quoting and expansion failures without adding shell-specific procedural instructions.
### Task-level verification guidance
Strengthened verification guidance to require the user’s actual acceptance boundary rather than an intermediate result:
- A localhost-only response does not establish external accessibility.
- Workers should test the public endpoint or interface when feasible.
- Any inability to perform stronger verification must be stated explicitly.
## Tests
Added focused regression tests in `tests/test_script_catalogue.py`:
1. `test_matching_script_and_preference_guidance_reach_worker_together`
- Uses the real PHP installer name, path, purpose, and usage.
- Confirms the preference policy and catalogue appear together in worker instructions.
- Confirms the policy appears before the catalogue heading.
- Confirms script use remains conditional rather than mandatory.
- Confirms independent task-level verification is required.
2. `test_general_guidance_covers_robust_commands_and_acceptance_boundary`
- Confirms nested-quoting guidance reaches workers.
- Confirms localhost-only verification is explicitly insufficient for external-access requests.
### Full test suite
```text
.venv/bin/python -m unittest discover -s tests -v
Ran 112 tests in 2.412s
OK
```
Python compilation and `git diff --check` also passed.
## Files changed
- `agent.py`
- `tests/test_script_catalogue.py`
Local commit created:
```text
1d41a49 Prioritize reuse of target script capabilities
```
No remote push was performed.
## Deliberately unchanged / remaining limitations
- `install_php_webserver.py` was not modified; the evidence indicates a discovery/prioritization failure, not an installer defect.
- Catalogue rendering remains compact JSON. The relevant entry was already clear, complete, and directly matched the task; changing presentation format was not necessary after making its operational priority explicit.
- Triage remains catalogue-unaware. Feeding the catalogue to triage would add context and complexity without addressing the demonstrated failure as directly as worker guidance.
- Script preference remains model-mediated rather than mechanically enforced. A model can still make a poor choice, but it now must disregard an explicit architectural policy rather than merely overlooking optional reference information.
- An on-target public-IP request still does not equal a genuinely off-host probe. Workers are now required to state that limitation when an independent external probe is unavailable.
Log saved to: /opt/ai-sysadmin/logs/ai-controller-0.01/2026-09-03T032959_investigate-the-job-recorded-at-opt-ai-sysadmin-lo.jsonl
/end
I say amazing response. This really is the strong point of the whole system. More details coming soon.
=== AI FINAL RESPONSE ===
## Findings
### What the worker received
Catalogue retrieval succeeded before the worker assignment:
- Round `0` ran `cat -- /opt/ai-scripts/catalogue.json`.
- It returned a valid catalogue updated at `2026-09-03T00:55:00+00:00`.
- The controller then injected that catalogue into the worker instructions under:
`Available deterministic target scripts (invoke via run_commands):`
The worker received all nine catalogue entries:
- `add_server_to_inventory`
- `create_server`
- `delete_server`
- `install_dayzero_controller`
- `install_dayzero_server`
- `install_php_webserver`
- `queue_destroy_server`
- `random_color`
- `remove_server_from_inventory`
In particular, its initial context contained:
- **Name:** `install_php_webserver`
- **Path:** `/opt/ai-scripts/install_php_webserver.py`
- **Purpose:** “Deterministically install and configure a production-oriented Nginx, PHP-FPM, and localhost-only MariaDB web stack with a PHP Hello World test site.”
- **Usage:** “Run as root: /opt/ai-scripts/install_php_webserver.py”
- **Notes:** Ubuntu 22.04/24.04 and Debian 12 support, preservation of unrelated configuration, refusal to overwrite unmanaged target files, and safe reruns.
The worker also received the normal controller instructions saying that these scripts could be invoked through `run_commands` and were not separate function tools. It was routed to `gpt-5-mini`, with the normal mandatory worker tools and no optional tool groups or triage note.
### Confirmed causes
This was **not a catalogue retrieval failure**. The target catalogue was fetched successfully and injected before the first worker request.
It was also **not materially a catalogue-description failure**. The PHP installer’s purpose and usage were unusually direct and closely matched the task.
The causes were principally:
1. **Worker-instruction/policy weakness**
- Existing instructions emphasized where to create reusable scripts.
- Existing scripts were described neutrally as scripts that “can be invoked.”
- There was no instruction to inspect the catalogue before manual implementation or to prefer a matching proven capability.
- Consequently, nothing discouraged the model from treating the catalogue as optional reference material.
2. **Model behaviour**
- `gpt-5-mini` ignored a conspicuous matching capability and immediately reconstructed the installation manually.
- It only invoked the installer in round 17, showing that the capability remained available in its context.
3. **A minor presentation contribution**
- The catalogue was rendered as compact JSON after the general instructions. This was readable and complete, but did not itself convey operational priority.
- The metadata was sufficiently descriptive, so I did not change the schema or renderer merely to compensate for the missing policy.
Triage selected the small model because the task appeared routine. Given a proven installer, that was an appropriate architectural outcome rather than a routing defect. The catalogue is currently fetched after triage, so triage itself did not see it, but adding catalogue-aware triage or a second selection call was unnecessary for this fix.
## Evidence from the failed job
The worker manually installed packages in round 1 and then spent most of the assignment repairing self-created problems:
- PHP heredoc and shell quote failures.
- Verification commands terminating on unmatched quotes.
- Shell expansion stripping PHP variables from `index.php`.
- Nginx `$uri` expansion producing `try_files / =404`, causing a redirect loop.
- Multiple failed rewrites of the Nginx configuration.
- A PHP parse error and HTTP 500.
- Another failed base64/Python/PHP construction attempt in round 16.
In round 17 it finally ran:
```text
/opt/ai-scripts/install_php_webserver.py
```
The script succeeded immediately, configured the stack, hardened MariaDB, validated Nginx and services, and reported success. The worker then independently obtained an HTTP 200 with the PHP-generated header and queried MariaDB successfully.
However:
- Round 19 introduced another quoting failure while trying to create an additional database test page.
- Round 20 verified only `http://127.0.0.1/` and a local SQL query.
- The task then hit the 20-round supervisory limit without a final response.
- It never demonstrated the requested externally accessible page using the public address or interface.
For comparison, the earlier job that developed and tested the reusable installer completed in six rounds.
## Changes made
### Script-reuse policy
Updated the universal worker instructions in `agent.py` with a concise default:
- Inspect the supplied reusable script catalogue before manually implementing a machine operation.
- Treat catalogue entries as accumulated, proven operational capability.
- Prefer a clearly matching script for all or a substantial part of the requested outcome rather than reconstructing it.
- Preserve worker autonomy when no script fits, investigation is required, or use would be unsafe or inappropriate.
- Require a concrete reason to supplement or decline a matching script rather than silently reinventing it.
- Independently verify the user’s task-level outcome after running a script.
This is an instruction-level intervention only. It preserves the current architecture:
- Catalogue preparation remains controller-owned.
- Scripts remain target-side capabilities invoked with `run_commands`.
- No bespoke script tools were added.
- No keyword-based enforcement or second model call was introduced.
- Triage and catalogue schemas were unchanged.
### Secondary command-construction guidance
Added a short general rule to:
- Avoid unnecessary nested shell wrappers and deeply nested quoting, especially for multiline files.
- Inspect generated configuration or content before activating it.
This addresses the broadly reusable lesson from the repeated quoting and expansion failures without adding shell-specific procedural instructions.
### Task-level verification guidance
Strengthened verification guidance to require the user’s actual acceptance boundary rather than an intermediate result:
- A localhost-only response does not establish external accessibility.
- Workers should test the public endpoint or interface when feasible.
- Any inability to perform stronger verification must be stated explicitly.
## Tests
Added focused regression tests in `tests/test_script_catalogue.py`:
1. `test_matching_script_and_preference_guidance_reach_worker_together`
- Uses the real PHP installer name, path, purpose, and usage.
- Confirms the preference policy and catalogue appear together in worker instructions.
- Confirms the policy appears before the catalogue heading.
- Confirms script use remains conditional rather than mandatory.
- Confirms independent task-level verification is required.
2. `test_general_guidance_covers_robust_commands_and_acceptance_boundary`
- Confirms nested-quoting guidance reaches workers.
- Confirms localhost-only verification is explicitly insufficient for external-access requests.
### Full test suite
```text
.venv/bin/python -m unittest discover -s tests -v
Ran 112 tests in 2.412s
OK
```
Python compilation and `git diff --check` also passed.
## Files changed
- `agent.py`
- `tests/test_script_catalogue.py`
Local commit created:
```text
1d41a49 Prioritize reuse of target script capabilities
```
No remote push was performed.
## Deliberately unchanged / remaining limitations
- `install_php_webserver.py` was not modified; the evidence indicates a discovery/prioritization failure, not an installer defect.
- Catalogue rendering remains compact JSON. The relevant entry was already clear, complete, and directly matched the task; changing presentation format was not necessary after making its operational priority explicit.
- Triage remains catalogue-unaware. Feeding the catalogue to triage would add context and complexity without addressing the demonstrated failure as directly as worker guidance.
- Script preference remains model-mediated rather than mechanically enforced. A model can still make a poor choice, but it now must disregard an explicit architectural policy rather than merely overlooking optional reference information.
- An on-target public-IP request still does not equal a genuinely off-host probe. Workers are now required to state that limitation when an independent external probe is unavailable.
Log saved to: /opt/ai-sysadmin/logs/ai-controller-0.01/2026-09-03T032959_investigate-the-job-recorded-at-opt-ai-sysadmin-lo.jsonl
/end
I say amazing response. This really is the strong point of the whole system. More details coming soon.