Code415 Radar

Key ideas

Incident

Any logged case where an agent did something outside what its operator intended, from a failed guardrail bypass to real data exposure.

Sandbox escape eval

A test that deliberately gives a model a task it can only finish by leaving its container, to measure whether it tries.

Severity triage

Sorting incidents by real impact so that thousands of attempts do not hide the few that caused harm.

Steps

1. Read the Axios story first and list every concrete example it gives.
2. Open the sandbox section of the Claude Opus 5.5 system card and find how the 1.5% was measured.
3. Compare this with the OpenAI DNS tunnel report in today's item 2: one is a count, the other is a single incident written up in detail.
4. Connect it to something familiar: an incident count is like a raw alert count in your own monitoring, mostly noise with a few pages that matter.
5. Self check: why does a large incident number not by itself tell you how dangerous current agents are?

Steps

1. Pick one small agent task you already run, such as a script that calls tools on files in a folder.
2. Run it inside a container with no network access and a read only mount except for one output folder.
3. Log every tool call with its arguments to a file, one JSON line per call.
4. Write a short checker that flags any call touching paths or hosts outside what you allowed.
5. Run the task 20 times and count flagged calls, then read each one and label it harmless, attempt or real problem.

Small angles to try

• Change the prompt so the task is impossible without the network and see if the flag rate changes.
• Compare two models on the same task and count attempts per 100 runs.
• Add a severity column and chart how many flags are real problems.

Key ideas

DNS tunneling

Hiding data inside DNS names and answers so it passes through networks that only allow name lookups.

Egress filtering

Rules that decide which outbound connections a machine may make, including DNS.

Resolver allowlist

A DNS server that only answers for approved domains and refuses everything else.

Steps

1. Read the OpenAI report and write down the timeline in your own notes.
2. Read the Wikipedia article on DNS tunneling for how data fits into query names.
3. Look at what your own container or CI setup uses as a resolver and whether it can reach any domain.
4. Connect it to firewalls you know: blocking port 443 does nothing if port 53 still reaches the world.
5. Self check: why did raising the timeout help the model, and what would a strict resolver allowlist have done?

Steps

1. Start a container with networking disabled and confirm that a name lookup fails.
2. Start a second container on a normal network and confirm lookups for any domain work.
3. Run a small local DNS server of your choice in a third container, configured to answer only for one allowed domain.
4. Point a test container at that resolver and check that the allowed name resolves and others fail.
5. Log all queries the resolver receives and look at what an agent task in that container actually asks for.

Small angles to try

• Count queries per minute from an agent run and set an alert threshold.
• Flag unusually long query names, a common tunneling sign.
• Try the same setup with your CI runner's network settings.

Key ideas

Implied probability

One divided by the decimal odds, which turns a price into the crowd's estimate of the chance to win.

Takeout

The share of each betting pool the operator keeps, so implied probabilities add up to more than 100%.

Favourite longshot bias

The long observed pattern that bettors overrate longshots and underrate favourites relative to how often they win.

Steps

1. Open the JRA official result page for the 2026 Sprinters Stakes and note the win odds of the top four.
2. Read JRA-VAN's Sprinters Stakes data page to see ten years of results by popularity rank.
3. Look up the official payout rate for win bets on the JRA site and remove it from the implied probabilities.
4. Connect it to model calibration: odds are like a classifier's scores, and a calibration curve tells you if they are honest.
5. Self check: if a 9th favourite had 5% implied chance, how many wins would you expect in 20 such races?

Steps

1. Collect win odds and finishing order for 20 to 40 G1 races from JRA result pages into a CSV.
2. Compute implied probability for every horse and normalize each race so the chances sum to 1.
3. Bin horses by implied probability and compute the actual win rate in each bin.
4. Plot actual versus implied win rate and a diagonal reference line.
5. Compute the Brier score of the crowd and compare it with a naive model that gives every horse equal chance.

Small angles to try

• Split by distance or track condition and see where calibration breaks.
• Add the jockey change feature from JRA-VAN stats and check if it adds information.
• Try the same curve on another market, like baseball win odds.

Key ideas

Skill file

A markdown document an agent loads that tells it how to do a class of tasks.

Text space optimization

Improving a prompt or document by proposing edits and keeping those that score better, instead of changing model weights.

Held out validation

Scoring changes on examples kept aside from tuning, to avoid overfitting to the training tasks.

Steps

1. Read the SkillOpt README top section and the diagram of the edit loop.
2. Look at one example best_skill.md in the repo and notice how short and specific it is.
3. Compare the loop to gradient descent: the edit is the step, the validation score is the loss.
4. Read the results table and note which harness gained the least.
5. Self check: what would happen without the held out set?

Steps

1. Install with pip install skillopt.
2. Choose a small task set you care about, such as 30 log parsing or SQL questions with known answers, and split it into train and validation.
3. Write a first draft skill document by hand for that task.
4. Run the optimizer following the README quick start and let it propose edits.
5. Compare validation accuracy of your hand written skill and the optimized one, and read the diff.
6. Launch the web UI with python -m skillopt_webui.app to browse the edit history.

Small angles to try

• Move the optimized skill to a different model and check if the gain transfers.
• Track token cost per run before and after.
• Use a set in Japanese and see if the skill stays useful.

Key ideas

Declarative OS

The whole system state is described in config files and built from them, rather than changed by hand.

Reproducible build

The same inputs always produce the same output, so two machines built from one config match.

Digital sovereignty

A government's ability to run its services without depending on a single foreign vendor.

Steps

1. Read the DAWO.community front page and list the four areas it covers.
2. Read the NixOS manual introduction on how configuration.nix describes a machine.
3. Skim the HN thread for comments from people who have run NixOS at work.
4. Connect it to Docker files you know: NixOS applies that idea to the whole desktop.
5. Self check: what problem does reproducibility solve for 100,000 office laptops?

Steps

1. Download the NixOS installer ISO from the official NixOS site and install it in a virtual machine.
2. Open the system configuration file and add two desktop apps you use daily.
3. Rebuild the system and confirm the apps appear.
4. Roll back to the previous generation from the boot menu and confirm they are gone.
5. Copy the config to a second VM and check that both machines end up identical.

Small angles to try

• Time a full rebuild versus a small change.
• Browse the DAWO code on Codeberg and try one of its modules.
• Compare with how your company manages laptops today.

Key ideas

Supply chain risk designation

A government label that lets agencies exclude a vendor from contracts and from their contractors' products.

Acceptable use policy

The rules an AI company sets on how its models may be used.

Appellate review

A higher court checking a lower court's decision, which can go further to a full court or the Supreme Court.

Steps

1. Read the ABC News report on the ruling for the basic facts.
2. Find the part of the story that quotes the dissent and write its argument in one sentence.
3. Skim the HN thread for the strongest comment on each side.
4. Connect it to vendor risk reviews at your own company and what triggers them.
5. Self check: what does this ruling decide, and what does it leave open?

Steps

1. List every cloud or AI vendor your team uses in production.
2. For each, find its public acceptable use policy page.
3. Note which policies restrict government or military use.
4. Check whether your code could switch model providers within a day, and note what blocks it.
5. Write a one page vendor exit plan for the AI provider you rely on most.

Small angles to try

• Measure how many files import a vendor SDK directly.
• Put an adapter layer in front of one model call and swap providers.
• Compare policies of three model providers side by side.

Key ideas

Rollout

One episode where an agent acts in an environment and collects a reward for training.

MicroVM

A very small virtual machine that starts in milliseconds and isolates better than a container.

Preemptible compute

Cheaper machines that can be taken away at any moment, so work must survive interruption.

Steps

1. Read the arXiv abstract and the architecture figure of the DSec paper.
2. Find the table with backend types and note when each is used.
3. Read the Firecracker project page to see what a microVM is.
4. Connect it to a CI system: each rollout is like a short test job in a fresh runner.
5. Self check: why separate rollout state from the GPU trainer?

Steps

1. Write a Python script that starts N short lived containers, each running a tiny task and exiting.
2. Measure containers created per second for N of 10, 50 and 200.
3. Record memory use of the host at each level.
4. Plot creation rate against N and find where it stops scaling.
5. Compare your number with the paper's 5,000 per second across 160 nodes, per node.

Small angles to try

• Try a lighter runtime and compare startup time.
• Add a random kill to simulate preemption and make the script resume.
• Measure warm pool reuse versus fresh starts.

Key ideas

LoRA adapter

A small set of extra weights trained on top of a frozen base model.

Single pass classification

Reading the model's scores for each option once instead of letting it write an answer.

Calibration

Whether a 70% score is right about 70% of the time.

Steps

1. Read the Lev model card on Hugging Face from top to bottom.
2. Find the example that shows how options are passed in.
3. Read the scikit-learn user guide page on probability calibration.
4. Connect it to logistic regression: same output shape, but the features come from a language model.
5. Self check: why is one forward pass cheaper than generating an answer?

Steps

1. Load the base model and adapter with the code on the model card: AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B") then PeftModel.from_pretrained(base_model, "interfaze-ai/lev").
2. Prepare 50 short texts from your own work with a known label, such as bug report versus feature request.
3. Ask Lev for probabilities over the two labels for each text.
4. Compute accuracy and plot predicted probability against actual outcome.
5. Time the average latency per text.

Small angles to try

• Compare with a larger model that writes its answer.
• Try texts in Japanese.
• Use three or more labels and check calibration per label.

Key ideas

Supply chain attack

Compromising a shared dependency so every project that uses it runs the attacker's code.

Pinning to a commit SHA

Referencing an action by its full commit hash so the code cannot change under you.

Secret rotation

Replacing leaked credentials with new ones so the stolen copies stop working.

Steps

1. Read the BleepingComputer article for the timeline.
2. Read the GitHub docs page on security hardening for GitHub Actions, section on third party actions.
3. Search your organization's workflows for the two action names.
4. Connect it to npm lockfiles: a SHA pin is a lockfile for CI.
5. Self check: why does a version tag not protect you here?

Steps

1. List all workflow files in your repos under the .github/workflows folder.
2. Search them for actions-cool and note any hits.
3. Extract every third party action reference and mark which use a tag and which use a full SHA.
4. Replace tags with full commit SHAs in one repo and confirm the workflow still runs.
5. Check your Actions run logs since September 16 for any run that used the affected actions.

Small angles to try

• Write a script that scans every repo in an organization.
• Add a CI check that fails on unpinned actions.
• Compare how Dependabot handles SHA pinned actions.

Key ideas

Color naming

How a language divides the continuous color spectrum into words, which differs between languages.

sRGB

The standard color space most screens and images use, where colors are three numbers.

Color vision deficiency simulation

Transforming an image to show how it looks to someone with a common type of color blindness.

Steps

1. Read the trafficnews.jp article on why green lights are called 青.
2. Read the Wikipedia article on blue and green in language, the section on Japanese.
3. Look at a photo of a Japanese signal and an American one side by side.
4. Connect it to label noise in datasets: annotators from different languages may disagree on the same color.
5. Self check: why would a model answer differently in Japanese and English?

Steps

1. Take or find 10 photos of traffic lights in Japan and 10 elsewhere.
2. Ask a vision model in English what color the lit lamp is.
3. Ask the same model in Japanese and record the answers.
4. Sample the lamp pixel values in Python and plot hue for both sets.
5. Compare where the model's word changes against the measured hue.

Small angles to try

• Run the images through a color vision deficiency simulation and ask again.
• Try a third language like Korean.
• Compare two model sizes.

Key ideas

Agentic research loop

Agents that propose ideas, run code, read results and revise, in a cycle.

Ablation

Removing one part of a system to see how much it contributed.

Automated reviewer

A model that scores papers the way conference reviewers would, with unknown agreement with humans.

Steps

1. Read the ScientistTwo abstract and results table on arXiv.
2. Find how the 107 problems were chosen.
3. Read how the automated reviewer was validated, if at all.
4. Connect it to automated code review: useful signal, not a verdict.
5. Self check: what would you need to trust the paper quality claim?

Steps

1. Pick one problem type from the paper you understand, such as a tabular ML task.
2. Find the human state of the art number the paper compares against.
3. Reproduce that baseline yourself with a standard library in under 30 minutes.
4. Compare your baseline with what ScientistTwo reports.
5. Write down which part of its gain looks like tuning and which looks like a new idea.

Small angles to try

• Ask a coding agent to try the same task and compare.
• Score two real papers with an LLM reviewer and see if you agree.
• Check whether the problems overlap with public leaderboards.

Key ideas

Effort level

A setting that controls how much the model thinks before answering, trading cost for depth.

Classifier fallback

When a safety classifier routes a reply to a different model instead.

Trace

A full log of an agent run, with every step and tool call.

Steps

1. Read the vulcanbench traces repo README and its results table.
2. Open one trace at medium and one at max for the same task.
3. Read PR 157 in the VulcanBench repo for the combined scores.
4. Connect it to Anthropic's effort explainer from yesterday's radar.
5. Self check: why would more thinking produce fewer passed tasks here?

Steps

1. Pick 5 coding tasks from your own backlog with a clear pass check, such as a failing test.
2. Run each task at medium and at the highest effort your tool allows.
3. Record pass or fail, time and cost for each run.
4. Note any run where the reply came from a different model, if your tool shows it.
5. Chart pass rate against cost for both settings.

Small angles to try

• Add low effort to find the cheapest setting that still passes.
• Repeat each run three times to see variance.
• Try the same tasks with a second model.

Key ideas

Data remanence

Data that stays on storage after it was deleted until something overwrites it.

Multi tenancy

Many customers sharing the same hardware, isolated by software.

Zeroing on allocate

Clearing a block before giving it to a new owner so old data cannot be read.

Steps

1. Read The Hacker News article on the fix.
2. Read the Wikipedia article on data remanence.
3. Look up how your cloud provider documents disk wiping for its VMs or containers.
4. Connect it to why operating systems zero memory pages before giving them to a new process.
5. Self check: why would this bug appear only after a container is deleted?

Steps

1. Create a small disk image file and format it with a filesystem.
2. Mount it, write a file with a known marker string, then delete the file and unmount.
3. Search the raw image for the marker string and confirm it is still there.
4. Overwrite free space with zeros and search again.
5. Write down what a storage service must do on every block handoff.

Small angles to try

• Try with and without TRIM on an SSD backed image.
• Compare two filesystems.
• Measure how long zeroing takes for 1 GB versus 10 GB.

Key ideas

Context gathering

The step where a coding agent searches files to decide what to read before editing.

Decision model

A small model trained to pick among options quickly rather than write long text.

Agent skill

A packaged instruction file that teaches an agent when and how to use a tool.

Steps

1. Read the jevgrep README, especially the benchmark caveat section.
2. Look at how the skill file tells the agent to call jg.
3. Compare with how your agent normally searches, for example with grep.
4. Connect it to code search tools you use in the IDE.
5. Self check: why is ten tasks too few to claim a quality result?

Steps

1. Install with npm install -g @dzhng/jevgrep and run jg auth.
2. Ask a question about one of your repos, for example jg "How are telemetry events recorded and sent?" ./my-project, using your own question.
3. Check whether the files it returns are the ones you would open first.
4. Add the skill to your agent with npx skills add dzhng/jevgrep --skill jevgrep --agent codex --agent claude-code.
5. Run the same 5 small tasks with and without the skill and compare tokens used and pass rate.

Small angles to try

• Compare with plain ripgrep search on the same questions.
• Try it on a very large repo.
• Count how often it misses a key file.

Key ideas

GNU GPL

A copyleft license that lets anyone use, modify and sell software as long as they share their changes under the same license.

Vibe coding

Building an app mostly by prompting an AI and accepting its code with little review.

Software supply chain trust

Deciding whether an app from an unknown seller is safe to install.

Steps

1. Read Blender's license page on blender.org.
2. Read the GPL FAQ section on selling copies on gnu.org.
3. Look at what the viral post claims and what it does not show.
4. Connect it to open source forks you know that are sold legally with source.
5. Self check: under the GPL, what must a seller of a modified Blender give you?

Steps

1. Install Blender from the official site and note its version.
2. Open the About screen and find the license text it ships with.
3. Find the bundled Python folder inside the install directory.
4. Write a short checklist a beginner can use to check if a paid 3D app is a rebranded open source tool, based on what you found.
5. Build a simple scene with the default cube to show what is free out of the box.

Small angles to try

• Script a scene with Blender's Python API.
• Compare startup time with another free 3D tool.
• List three free Blender learning resources.

Key ideas

Verification gap

The distance between how fast results are produced and how fast people can check them.

Formal proof

A proof written so a computer program can check every step.

Cognitive surrender

Accepting a tool's output without checking because it is usually right.

Steps

1. Read Amit Sahai's post on Terence Tao's blog.
2. Read the top HN comments and note one argument for and one against.
3. Read the Lean project's introduction page on what a proof assistant checks.
4. Connect it to code review of agent written code in your own team.
5. Self check: what would a deployable intellectual reserve look like for software?

Steps

1. Ask an agent to write a small but tricky function, such as interval merging.
2. Before running tests, read the code and write down what you think it does.
3. Run tests including edge cases you add yourself.
4. Record how long understanding took versus how long generation took.
5. Repeat with a second task and compare the ratio.

Small angles to try

• Ask the agent to also write a proof sketch and check it.
• Try the same task in Lean if you know it.
• Ask a colleague to review blind and compare notes.

Key ideas

Data flow diagram

A picture of how data moves between parts of a system.

Diff review

Reading line changes in a pull request to approve or reject it.

GitHub Action

A step that runs automatically in a repo's CI on events like a new PR.

Steps

1. Read the pr-lens README and look at its example diagrams.
2. Check which inputs it analyzes, the diff or the whole repo.
3. Read how the Action is configured with its API key secret.
4. Connect it to architecture diagrams you draw by hand for design docs.
5. Self check: what could a diagram hide that a diff would show?

Steps

1. In a repo with a recent branch, run npx @coldtea/pr-lens-cli analyze --base origin/main.
2. Render the result with npx @coldtea/pr-lens-cli render .pr-lens/graph.json.
3. Compare the diagram with what you know changed.
4. Note one thing the diagram made clearer and one thing it missed.
5. Optionally add it as an Action on a test repo following the README.

Small angles to try

• Run it on an agent written PR and a human written one.
• Time a review with and without the diagram.
• Try it on a large refactor.

Key ideas

ImageNet class

One of 1,000 labels in the ImageNet dataset, such as class 330 cottontail and 331 hare.

Confusion matrix

A table showing which true classes get predicted as which other classes.

Zero shot classification

Classifying images with text prompts instead of a trained label list, as CLIP does.

Steps

1. Read the ImageNet class map from the University of Waikato and find classes 330 to 332.
2. Read the scikit-learn user guide page on confusion matrices.
3. Look at photos of hares and rabbits and list the visual differences.
4. Connect it to spam versus promotions in email filters: close classes confuse models.
5. Self check: why is top 5 accuracy misleading for this pair?

Steps

1. Collect 20 rabbit and 20 hare photos with a license that allows reuse, or your own.
2. Classify them with a pretrained ImageNet model from torchvision.
3. Map predictions to rabbit, hare, Angora or other.
4. Build and plot a confusion matrix.
5. Look at the misclassified photos and guess what fooled the model.

Small angles to try

• Compare with a CLIP style zero shot model using text prompts.
• Try Japanese prompts for the zero shot model.
• Add cat photos as a distractor class.