Code415 Radar

Key ideas

Usage limit vs API price

Subscription plans cap how much you can use in a five hour window, while the API simply bills per token.

Model drift check

Rerunning the same fixed tasks later to see whether a hosted model got better or worse.

Launch score

The benchmark number a lab publishes on release day, which later retests are compared against.

Steps

1. Read the Anthropic Opus 5.5 launch page and note the price, the Terminal-Bench 4.0 score and the limits change.
2. Open DHH's and Peter Yang's X posts and read the replies to see which tasks people actually tried.
3. Read BridgeMind's retest post and write down what it measured and what it did not.
4. Compare this to how you would check a library upgrade: same inputs, same checks, before and after.
5. Self check: why is a retest by one account weaker evidence than a repeatable public eval you can run yourself?

Steps

1. Pick 5 tasks you already solved yourself, each with a test or an expected output.
2. Run each task once in Claude Code or through the API with Opus 5.5 and a fixed prompt.
3. Record pass or fail, wall time and tokens used for every task in a small table.
4. Save the prompts and the table so you can rerun the exact same set next week.
5. Repeat the run in 7 days and compare the two tables row by row.

Small angles to try

• Run the same 5 tasks on GPT-6 Sol and compare cost per passed task.
• Try one task at two effort levels and note the token difference.
• Add one task in a language you rarely use, such as Rust or Elixir.

Key ideas

Agent permission boundary

The list of actions an agent may take alone versus ones that need a human to confirm.

PII

Personally identifiable information such as a home address, phone number or full name.

Output guardrail

A check that runs on what the agent is about to send, not only on what the user asked.

Steps

1. Read The Deep Dive write up and list every action Muse took without confirmation.
2. Mark which of those actions should have been gated, and why an address counts as sensitive.
3. Connect it to web security: this is like an API that authorizes the request but never checks the response body.
4. Read the Presidio docs overview to see how PII detection on text works.
5. Self check: where would you place a confirmation step so that a helpful agent still stays fast?

Steps

1. Install Presidio with pip install presidio-analyzer presidio-anonymizer and download the spaCy English model named on the Presidio install page.
2. Write 20 fake Marketplace chat replies yourself, 8 of them containing an address or phone number.
3. Run each reply through the Presidio analyzer and flag any LOCATION or PHONE_NUMBER result.
4. Count how many of the 8 risky replies were caught and how many safe replies were flagged by mistake.
5. Turn it into a gate: if a flag fires, print "needs owner confirmation" instead of sending.

Small angles to try

• Add Japanese addresses and see how detection changes.
• Compare Presidio with asking a small local LLM to spot PII.
• Measure the extra latency the check adds per message.

Key ideas

EXIF

Metadata stored inside a photo file, such as camera model, time, lens and GPS position.

Metadata stripping

Social platforms often remove EXIF on upload to protect privacy and save space.

Image fingerprinting

Guessing a camera from pixels alone, for example noise patterns, which is far less certain than EXIF.

Steps

1. Read the Shukan Josei PRIME article for the facts of the visit and the photo release.
2. Skim the ExifTool documentation page to see which tag groups a phone photo usually carries.
3. Look for the GPS and Make or Model tags and think about what each reveals.
4. Connect it to logging: EXIF is like request headers, useful for debugging and risky to leak.
5. Self check: if a photo on X has no EXIF, what evidence is left to guess the phone?

Steps

1. Take a photo with your phone with location on and copy the original file to your computer.
2. Run exiftool image.jpg and save the output to a text file.
3. Post the photo privately or to a test account, download it again and run ExifTool on the copy.
4. Diff the two outputs and list which fields survived the round trip.
5. Remove location from the original with exiftool -GPS:all= image.jpg and confirm the GPS tags are gone.

Small angles to try

• Compare several platforms, for example X, Instagram and LINE.
• Try HEIC versus JPEG from the same phone.
• Check a screenshot versus a camera photo.

Key ideas

Sandbox

An isolated environment where code cannot reach real networks or secrets unless you allow it.

Egress control

Rules that limit which outside hosts a process may connect to.

Leaked credentials

Keys or passwords left in code repos that anyone, including an agent, can find and use.

Steps

1. Read Eoin Higgins' essay and note his main claim about responsibility.
2. Read the Daily Caller summary for what each agency actually said.
3. Map each incident to one missing control: no egress limit, leaked credential, or weak site auth.
4. Compare it to a CI runner with open internet and secrets in environment variables.
5. Self check: what is the smallest network rule that would have stopped all three probes?

Steps

1. Start a container with networking disabled using Docker's none network mode.
2. Inside it, try to fetch any public web page and confirm the request fails.
3. Start a second container on a user defined network with only one allowed internal service.
4. Run a small script that tries three hosts and log which ones connect.
5. Write down the rule set in one line each, as if it were an agent sandbox policy.

Small angles to try

• Add a fake secret in an environment variable and check whether your script can read it.
• Repeat with Podman rootless and compare.
• Log DNS lookups too, since earlier reports showed DNS tunneling.

Key ideas

KEV

CISA's Known Exploited Vulnerabilities catalog, which sets patch deadlines for US federal agencies.

DTLS

TLS over UDP, used by VPN gateways for faster tunnels.

Edge appliance

A box like a VPN gateway that sits on the internet in front of the internal network.

Steps

1. Read the first BleepingComputer article for the two CVEs and affected setups.
2. Read the second one for the CISA deadline and the fixed build numbers.
3. Note why default configs matter: most installs never turn features off.
4. Compare it to leaving an admin panel reachable from the internet on a web app.
5. Self check: which of your own services would you shut down first if a CVSS 9.5 bug landed today?

Steps

1. Copy the fixed build numbers from the bulletin coverage into a small list.
2. Write a function that compares a version string like 14.1-70.10 against the fixed build for its branch.
3. Test it on 6 made up version strings, including FIPS builds.
4. Add a check for whether DTLS is enabled in a sample config text you write yourself.
5. Print a one line verdict per sample: patched, vulnerable, or needs review.

Small angles to try

• Turn it into a CI check for an internal inventory file.
• Parse the CISA KEV JSON feed and alert on any product you run.
• Rewrite it in Go and compare how version parsing edge cases behave.

Key ideas

Out of band monitoring

Watching a system from separate hardware so a compromised host cannot hide or switch off the monitor.

DPU

A data processing unit, a network card with its own CPU that can inspect and block traffic.

Policy enforcement

Checking each action against rules before it runs, instead of reviewing logs afterward.

Steps

1. Read the NVIDIA newsroom release and list what OpenShell does and what Sentry does.
2. Note which parts are open source and which need special hardware.
3. Compare Sentry to a hypervisor or a network firewall that sits outside the guest.
4. Relate it to today's rogue agent debate: tracing and quarantine are the missing controls people mention.
5. Self check: what can an out of band watchdog see that an in process logger cannot?

Steps

1. Write a tiny fake agent script that performs file reads, file writes and network calls from a task list.
2. Wrap it in a small Python shim that logs every action with a timestamp before it runs.
3. Add a policy file that denies writes outside one folder and any network host not on a list.
4. Run 10 tasks, including 3 that break the policy, and confirm they are blocked and logged.
5. Check the OpenShell repo linked from NVIDIA's developer resources and compare its policy model with yours.

Small angles to try

• Measure how much the shim slows each action.
• Enforce the same policy with a container network rule instead and compare.
• Add a kill switch that stops the agent after 3 violations.

Key ideas

Training data provenance

Knowing where each document in a training set came from and under what license.

Shadow library

A site like LibGen that hosts copyrighted books without permission.

Discovery

The court process where each side must hand over internal documents, which is how these emails surfaced.

Steps

1. Read the Authors Guild post and list the specific claims it makes.
2. Skim the top HN comments for counterarguments and legal context.
3. Compare this to software license compliance: a dependency with an unknown license in your build.
4. Note what is alleged versus what a court has decided.
5. Self check: what records would a team need to prove a dataset's provenance later?

Steps

1. Collect 50 public domain texts, for example from Project Gutenberg, into a folder.
2. Write a script that builds a manifest with file name, source URL, license and a SHA-256 hash.
3. Add 3 files with no source recorded and make the script flag them.
4. Save the manifest as JSON next to the data.
5. Rebuild the manifest a day later and confirm the hashes match.

Small angles to try

• Add near duplicate detection with MinHash.
• Store the manifest in SQLite and query by license.
• Generate a data card in Markdown from the manifest.

Key ideas

Reasoning tokens

Hidden thinking tokens a model generates before answering, which you still pay for.

Cost per task

Total tokens times price for a finished task, a better metric than price per token.

Pareto frontier

The set of models where you cannot get better quality without paying more.

Steps

1. Read the Fireworks Ember-1 blog post and note the benchmarks and the token savings claimed.
2. Read the top HN thread comments for skepticism about open weights and evals.
3. Compare it to compiler optimization: same output, fewer instructions.
4. Look at which benchmarks improved least and think about why.
5. Self check: when would fewer reasoning tokens hurt accuracy?

Steps

1. Pick 10 reasoning questions with known answers, for example math word problems you write.
2. Send each to Ember-1 and to Kimi K3 on Fireworks with the same prompt.
3. Record correctness, output tokens and latency for each call.
4. Compute tokens per correct answer for both models.
5. Check whether the savings match the 40% claim on your set.

Small angles to try

• Add a coding task with unit tests.
• Try the same set in Japanese.
• Plot latency against tokens to see if fewer tokens means faster.

Key ideas

Silent regression

A change that makes results worse without any error or warning.

Image encoding

The step that turns pixels into tokens the model reads; a bug here hurts every vision task.

Regression eval

A fixed set of inputs and expected outputs you rerun after every change.

Steps

1. Read the OpenAI API changelog entry for the fix.
2. Read the AlphaSignal note on why no errors appeared.
3. Connect it to a bad cache or codec in your own stack that returns wrong but valid data.
4. List which of your workflows use images or screenshots.
5. Self check: how would you have noticed this bug before OpenAI announced it?

Steps

1. Collect 10 images you made, such as screenshots of tables or charts, each with one factual question.
2. Write down the correct answer for each question.
3. Ask GPT-6 Sol each question through the API and score the answers.
4. Save the inputs, answers and scores as your baseline file.
5. Schedule a weekly rerun and alert if the score drops by 2 or more.

Small angles to try

• Run the same set on Luna and compare cost per correct answer.
• Add Japanese text images.
• Compare against Opus 5.5 on the same set.

Key ideas

Energy deployment

How much stored battery energy the car releases per lap, managed by software and rules.

Telemetry

Timed streams of speed, throttle and gear data recorded from the car.

Timing margin

The finish gap, measured by timing loops on the track to thousandths of a second.

Steps

1. Read the RACER race report for the final lap story.
2. Read the FastF1 docs quickstart to see what data is available per lap.
3. Look for speed traces on the final straight, where deployment matters most.
4. Compare energy deployment to a rate limiter with a budget that refills.
5. Self check: why would a lift for yellow flags change how much energy is left later?

Steps

1. Install FastF1 with pip install fastf1 and enable its cache as the docs show.
2. Load the 2026 Azerbaijan race session; if it is not available yet, use the 2025 race.
3. Pick the last lap of the top two drivers and plot speed against distance.
4. Mark where the gap closed and compare the top speed on the main straight.
5. Write one sentence on what the data shows and what it cannot show.

Small angles to try

• Plot throttle and gear traces too.
• Compare the gap across the last 5 laps.
• Repeat for the narrowest finish you remember from past seasons.

Key ideas

Agent skill

A packaged set of instructions and scripts a coding agent loads for one kind of task.

Candidate validation

Checking whether a suspected bug is real before reporting it, to cut false positives.

Independent verification

A second pass that tries to disprove each finding.

Steps

1. Open the security-audit-skill README and read the six phases.
2. Note the sandbox requirements and why networking is off.
3. Compare the flow to a human pentest report: scope, findings, evidence, severity.
4. Look at the output format and imagine feeding it into your issue tracker.
5. Self check: why does a separate verification phase matter for LLM findings?

Steps

1. Install it with npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit.
2. Pick one of your own old side projects, or plant 3 known bugs in a copy of it.
3. Run the audit from your agent inside a sandbox with networking disabled.
4. Compare the findings with the bugs you planted and count hits and false alarms.
5. Run it a second time and check what it added to the earlier findings.

Small angles to try

• Compare with a classic static analyzer like Semgrep.
• Try two different agent models and compare cost per real finding.
• Time each phase.

Key ideas

Line level comment

A review note tied to a specific changed line in a diff.

Deterministic pipeline

Fixed steps like parsing and filtering that give the same result every run.

Delegation Mode

ocr's option to let your existing coding agent do the review instead of calling an API itself.

Steps

1. Read the open-code-review README top to bottom.
2. Note which parts are fixed pipeline and which parts use the LLM.
3. Read the HN code review essay thread for what automation misses.
4. Compare it to a linter plus a human reviewer in your own team.
5. Self check: which review concerns would you never hand to a bot?

Steps

1. Install with npm install -g @alibaba-group/open-code-review.
2. Configure it with ocr config provider and ocr config model as the README shows.
3. Make a branch in your own repo with one real fix and two planted mistakes.
4. Run the review on that diff following the README and save the comments.
5. Label each comment useful, noise or wrong, and count them.

Small angles to try

• Compare two models on the same diff.
• Try Delegation Mode with your usual coding agent.
• Review a Japanese comment heavy diff.

Key ideas

Persistent undo

Saving undo history to disk so you can undo after reopening a file, in Vim since 7.3.

File format compatibility

Whether two programs can safely read and write the same file.

Duty of care

The idea that software should not destroy user data even when it is technically allowed to.

Steps

1. Read the Unsung post and note exactly what was lost and when.
2. Read the top HN comments for both sides.
3. Check Vim's help for the undofile option to see how the feature works.
4. Compare it to a database migration that drops a column without a backup.
5. Self check: what should a tool do when it finds a file in a format it cannot read?

Steps

1. Create a test folder and a text file, then enable persistent undo in a minimal Vim config pointing to that folder.
2. Edit the file in Vim several times, save and quit, and list the undo folder.
3. Open the same file in NeoVim with the same undo folder and quit.
4. List the undo folder again and see whether Vim's file changed or vanished.
5. Write down the versions you tested, since behaviour may differ by release.

Small angles to try

• Use separate undo folders for each editor and confirm the problem goes away.
• Check file hashes before and after.
• Try the same with swap files.

Key ideas

Sliding window attention

Each token attends only to a fixed number of recent tokens, which keeps memory flat for long inputs.

Sparse attention

Each token attends to a selected subset of earlier tokens instead of all of them.

Active parameters

The part of a MoE model used per token, which drives speed and cost.

Steps

1. Read the Hugging Face model card and note the layer layout and hardware needs.
2. Look at how 39 window layers and 9 sparse layers share the work of long context.
3. Compare it to a cache with a small recent window plus an index for older data.
4. Read NaiveAI's X post for the speed claims and mark which are unverified.
5. Self check: what kind of question would a model without full attention likely get wrong?

Steps

1. Download only the config file of the model from Hugging Face, not the weights.
2. Print the layer types, hidden size, number of experts and experts per token.
3. Compute a rough memory estimate for the weights at FP8 and compare it to the card's 315 GB.
4. Estimate KV cache size for 1M tokens with and without sliding windows.
5. Write a short note on what hardware you would need to serve it.

Small angles to try

• Compare the config with Qwen3.8-27B.
• Estimate cost per million tokens on rented GPUs.
• Chart KV cache growth for window sizes from 4K to 128K.

Key ideas

MCP

Model Context Protocol, a standard way for AI agents to call external tools.

Code mode

Letting the agent write short scripts against an API instead of choosing from many fixed tools.

IDB

IDA's database file holding the analysis of a binary: functions, names and comments.

Steps

1. Read the Hex-Rays blog post for the design choices.
2. Open the ida-mcp README and read the requirements and install options.
3. Compare code mode to giving a junior engineer a Python shell instead of a menu.
4. Think about why fewer tool definitions saves tokens.
5. Self check: what risks come with letting an agent run arbitrary IDAPython?

Steps

1. Compile a tiny C program of your own with a few functions and strip it.
2. Install the server for your client as the README shows, for example uvx ida-nexus mcp --agent=my-agent for generic clients.
3. Ask your agent to find and rename the functions in the stripped binary.
4. Compare its names to your source code and count correct ones.
5. Log the tokens used for the whole session.

Small angles to try

• Try a local model versus a cloud model.
• Compare with Ghidra plus a community MCP server.
• Use a Rust binary instead of C.

Key ideas

Procedural generation

Building worlds from algorithms and a seed instead of drawing them by hand.

Noise function

A smooth random function, like Perlin noise, used to shape terrain height and biomes.

Datapack

A folder of JSON files that changes Minecraft's rules or worlds without code mods.

Steps

1. Read Mojang's Minecraft Live recap for what The Sift is.
2. Read the Minecraft Wiki page on custom dimensions and find the dimension JSON fields.
3. Look at how multi_noise biome sources pick biomes from several noise values.
4. Compare noise based terrain to a heat map made from several blended random fields.
5. Self check: why does the same seed always make the same world?

Steps

1. Create a new test world and open its datapacks folder.
2. Build a small datapack with one dimension JSON using a noise generator and a multi_noise biome source, following the wiki.
3. Load the world and teleport into your dimension to explore it.
4. Change one noise setting, reload, and compare screenshots at the same coordinates.
5. Write down which setting changed the terrain the most.

Small angles to try

• Use a fixed biome source for a single biome world.
• Generate a Perlin noise heightmap in Python and compare the look.
• Time chunk generation with and without your changes.

Key ideas

Slop

Low effort AI output that looks finished but shows no real decisions.

Design tell

A visible pattern that reveals how something was made.

Negative prompting

Telling a model what not to do, which can prime the very thing you named.

Steps

1. Read the 10 tells post and screenshot one example of each from sites you know.
2. Read the HN thread for the negative prompting point.
3. Compare the tells to code smells in a review checklist.
4. Pick one UI you built with AI help and look for the tells.
5. Self check: which tell is a real usability problem and which is only taste?

Steps

1. Ask an LLM for a landing page for a made up app with a plain prompt and save the HTML.
2. Score the page against the 10 tells, one point each.
3. Rewrite the prompt with positive instructions only, describing what you want, and generate again.
4. Generate a third version with a list of things to avoid.
5. Compare the three scores and see whether the avoid list helped or hurt.

Small angles to try

• Repeat with two different models.
• Have a friend guess which page was human edited.
• Automate part of the scoring with simple HTML checks.

Key ideas

Speech to text

Turning recorded audio into a transcript that can be searched and analysed.

Anonymization

Removing names, numbers and places from data before reuse, which is rarely perfect.

Consent scope

What exactly a person agreed to, for example quality checks versus model training.

Steps

1. Read the Forrester blog on contact centre privacy.
2. List the uses of a call recording: quality checks, live agent assist, analytics, model training.
3. Compare it to logging user requests in a web app and later reusing the logs.
4. Look at the Presidio docs to see how text anonymization works.
5. Self check: what would a clear consent message for training on calls say?

Steps

1. Write 5 fake support call transcripts with names, phone numbers and addresses.
2. Install Presidio with pip install presidio-analyzer presidio-anonymizer plus the spaCy model named in its docs.
3. Run the analyzer and anonymizer on each transcript.
4. Count the personal details that remain after anonymizing.
5. Write one line on whether this output is safe to train on.

Small angles to try

• Add Japanese names and addresses.
• Transcribe a short recording of your own voice with Whisper first.
• Compare Presidio with a small local LLM redactor.