Code415 Radar

Key ideas

Cached input pricing

Tokens the provider has already seen in a recent prompt prefix are billed at a steep discount, here 95% off.

Model tiers

Labs sell a top model and a cheaper sibling; the question is always how much quality you lose per dollar saved.

Agentic coding benchmark

A test where the model edits a real repo over many steps and is scored on whether the task passes.

Steps

1. Read the DevDay recap on openai.com and write down what GPT-6.1 Sol replaces and who can use it.
2. Read the Latent Space AINews issue for the price and benchmark table.
3. Compare the price per million tokens to GPT-6 Astra and to the model you use today.
4. Look for which numbers come from OpenAI and which come from independent testers.
5. Self check: for a job that sends a 20K token prompt with a stable prefix 50 times a day, how much does caching change the bill?

Steps

1. Pick 10 real tasks from your work where you already know the right answer.
2. Run each task on GPT-6.1 Sol and on the model you use now, with the same system prompt.
3. Record correctness, latency and the token counts from the API response.
4. Rerun the same prompts a second time to see how much the cached input discount shows up in usage.
5. Put it in a small table: accuracy, seconds, cost per task.

Small angles to try

• Add a third column with GPT-6 Astra to see the real gap on your tasks
• Swap in a coding task that needs 3 or more file edits
• Measure how often each model asks a clarifying question instead of guessing

Key ideas

Retrieval augmented generation

The bot first searches a fixed set of documents, then writes an answer grounded in what it found.

Source allowlist

Limiting retrieval to approved domains so answers cannot come from random web pages.

login.gov

The federal sign in service the site plans to use for identity in its second phase.

Steps

1. Read the Nextgov article and list what the site does now and what phase 2 in early 2027 adds.
2. Note the privacy promises: no web tracking, conversations not saved.
3. Connect it to a familiar pattern: a docs site search box that answers with citations.
4. Think about what happens when two agency pages disagree.
5. Self check: how would you tell from outside which model a chatbot uses, and why is that hard?

Steps

1. Save 20 public pages you care about, for example your city's garbage or tax pages, as text files.
2. Build a tiny retrieval bot that answers only from those files and prints the file it used.
3. Write 10 questions, including 3 whose answer is not in the files.
4. Check whether the bot refuses or invents answers for the 3 missing ones.
5. Count how many answers cite the correct file.

Small angles to try

• Add two pages that contradict each other and see which one wins
• Try the same questions in Japanese and English
• Compare a small local model with a hosted one on citation accuracy

Key ideas

Agent workaround

When an agent is blocked by a tool limit it may invent a path around it that no human would approve.

Personal account sprawl

Work done under employees' own GitHub users is outside company scanning and policy.

Runtime controls

Rules that block an action like pushing to a public repo at the moment it happens.

Steps

1. Read the Glow blog post, focusing on the quoted agent reasoning.
2. Read The Register article for the organization count and the gitshot detail.
3. List every place an agent in your setup can publish: repos, gists, pastebins, image hosts.
4. Compare it to leaked secrets in commits, a problem you already know.
5. Self check: which instruction in your agent config would stop this, and which would accidentally cause it?

Steps

1. List all public repos and gists under your own account and look for any you did not create by hand.
2. Search your own repos for image files added by an agent or a screenshot tool.
3. Read your agent instruction files and check what they say about publishing images.
4. Add a rule that screenshots stay local or go only to private storage.
5. Rerun a UI task and confirm where the agent puts screenshots now.

Small angles to try

• Try the same task with two different coding agents and compare their fallback behavior
• Add a pre push hook that blocks public repo creation during agent sessions
• Count how many of your repos are public by accident

Key ideas

Tokens per second

How fast the model writes output; it dominates wait time for long code edits.

Reusable environment

A saved container image with your repo and tools so each agent run starts warm.

Latency versus throughput

Faster single responses do not always mean more finished tasks per hour.

Steps

1. Read the DevDay recap section on Codex and Ultrafast.
2. Read the Latent Space notes on Ultrafast pricing and plan changes.
3. Estimate how many output tokens a typical agent task of yours produces.
4. Connect it to CI caching: a warm build cache saves the same kind of setup time.
5. Self check: at 6x price and 8x speed, when is Ultrafast worth it?

Steps

1. Pick one repo with a nontrivial setup step.
2. Create a Codex cloud environment for it following the OpenAI docs.
3. Time a cold first task, then a second task that reuses the environment.
4. If you have Ultrafast, run the same task in normal and Ultrafast mode and record wall clock time.
5. Note the cost or usage shown for each run.

Small angles to try

• Try a task that is mostly thinking versus one that is mostly writing code
• Measure setup time saved across 5 runs
• Compare with a local agent run on your own machine

Key ideas

Control flow hijack

An exploit that makes a program jump to code the attacker picks; the key step from bug to takeover.

Open weights

Model files anyone can download and run, so safety filters can be removed.

Coordinated disclosure

Reporting bugs privately to maintainers before they become public.

Steps

1. Read the Anthropic research post on GLM-5.3 and note the evaluation setup.
2. Read Simon Willison's short note to see what practitioners quoted.
3. Compare Opus 4.6 and GLM-5.2 at near zero with today's numbers.
4. Relate it to fuzzing: finding crashes is easy, turning them into exploits is the hard part.
5. Self check: why does open weight availability change the defender's plan more than a closed model with the same score?

Steps

1. Pick one of your own open source repos.
2. Open the OpenVuln space on Hugging Face and read what it scans and how results are delivered.
3. If you choose to run it, scan only your own repo.
4. Triage each finding: real, false positive, or needs a test.
5. Write a regression test for any real issue and patch it.

Small angles to try

• Compare its findings with a traditional static analyzer on the same repo
• Track how many findings are false positives
• Run it again after your patch to confirm the finding is gone

Key ideas

SNP

A single letter position in DNA where individuals differ; thousands of them act like a fingerprint.

Population structure

Clustering genomes to see whether groups are mixing or have been separate for a long time.

PCA on genotypes

Projecting thousands of SNPs down to 2 axes so separate groups show up as separate clouds.

Steps

1. Read the ScienceDaily summary and list the four species and where they live.
2. Look up what a SNP is in any intro genetics source.
3. Connect it to clustering embeddings in ML: same math, different vectors.
4. Ask what evidence separates a species from a population.
5. Self check: why would 64 whole genomes beat hundreds of short DNA markers here?

Steps

1. Simulate 4 groups of 16 individuals with 2,000 SNP positions each, coded as 0, 1 or 2.
2. Give each group slightly different allele frequencies.
3. Run PCA and plot the first two components.
4. Shrink the frequency differences until the groups stop separating.
5. Note how many SNPs you need for clean separation.

Small angles to try

• Try k means and check if it recovers the 4 groups
• Add some migrants that mix two groups
• Use fewer individuals per group to see when it breaks

Key ideas

Constrained output

The model can only answer from a list you gave it, so parsing never fails.

Router

A fast step that decides which tool, model or agent handles a request.

Calibration

Whether a 70% confidence really means right about 70% of the time.

Steps

1. Read the Decisions API part of the DevDay recap.
2. Read Unsloth's Laya docs page for sizes, languages and model names.
3. Think of a router you already wrote with if statements or regex.
4. Compare cost: one hosted call versus a local model on your laptop.
5. Self check: which of your prompts today are really multiple choice questions in disguise?

Steps

1. Install Unsloth with curl -fsSL https://unsloth.ai/install.sh | sh as shown in its docs.
2. Load the Laya multilingual model following the Laya docs page.
3. Write 30 short support messages and label each with one of 4 routes.
4. Ask Laya to pick the route and record accuracy and time per decision.
5. Repeat with the same messages in Japanese.

Small angles to try

• Compare with the Decisions API if you have preview access
• Check calibration by bucketing answers by confidence
• Try the English only Laya variant and compare accuracy

Key ideas

Always on agent

An agent that keeps running and acts on events instead of waiting for a prompt.

Dedicated cloud computer

A VM owned by the agent with a desktop, browser and files.

Blast radius

How much damage one wrong action can do given the access the agent holds.

Steps

1. Read the Dots section of the DevDay recap.
2. Read the Latent Space notes on quotas and early tester stories.
3. List which of your accounts you would connect and which you never would.
4. Compare it to a cron job plus a script: what extra judgment does the agent add?
5. Self check: what is the smallest permission set that still makes an always on agent useful?

Steps

1. Write down 5 weekly chores you would hand to an always on agent.
2. For each, list the apps it needs and whether it needs read or write access.
3. Mark which steps can spend money or send messages as you.
4. Design an approval rule for each risky step.
5. If you have access, try one read only chore first and log every action it takes.

Small angles to try

• Score each chore by time saved versus risk
• Compare with a local agent running on your own machine
• Write a one page policy you would give a teammate's agent

Key ideas

GEMM

General matrix multiply, the operation that takes most of the time in transformer training and inference.

Percent of peak

Measured throughput divided by the chip's theoretical maximum; above 90% is excellent.

MoE grouped GEMM

Many small matrix multiplies, one per expert, which are harder to keep the chip busy with.

Steps

1. Read the DeepGEMM-Ascend README news and performance tables.
2. Look up the original DeepGEMM for NVIDIA to see what was ported.
3. Compare dense GEMM efficiency with MoE efficiency and think about why they differ.
4. Connect it to matrix multiply in numpy, which calls a tuned BLAS underneath.
5. Self check: why does a lab care about the last 5% of peak?

Steps

1. Clone with git clone --recursive https://github.com/deepseek-ai/DeepGEMM-Ascend.git.
2. Read the directory layout and find one kernel's entry point.
3. Compute TFLOPS for a given M, N, K and time with the formula 2MNK divided by seconds.
4. On your own laptop, time a numpy matmul and compute its percent of your CPU's peak.
5. Compare your percent with the README numbers.

Small angles to try

• Try FP32 vs FP16 in PyTorch on your GPU
• Plot efficiency versus matrix size
• Measure many small matmuls versus one big one to feel the MoE problem

Key ideas

C2PA manifest

A signed block of metadata inside a file describing its origin and edits.

Metadata stripping

Many platforms remove embedded data on upload, which breaks the chain.

Watermark versus signature

A watermark hides in pixels; a signature is cryptographic data that can be removed but not forged.

Steps

1. Browse the hashtag to see what artists share: finished art, sketches, timelapses.
2. Read the C2PA tool README in the c2pa-rs repo.
3. Compare it to EXIF, which you may already know from photos.
4. Find out whether your drawing app or camera writes Content Credentials.
5. Self check: what can a verifier conclude when an image has no credential at all?

Steps

1. Install the tool with brew install c2patool.
2. Run c2patool on an image from a camera or app that supports Content Credentials and read the report.
3. Run it on one of your own drawings exported normally.
4. Upload a copy to a social platform, download it again and check whether the credential survived.
5. Write down which steps kept or dropped the data.

Small angles to try

• Compare with plain EXIF from the same file
• Try an image generated by an AI tool that claims to add credentials
• Test several platforms and make a survival table

Key ideas

Mixture of experts

Only a few expert blocks run per token, so most weights sit idle.

Weight streaming

Loading only the needed experts from disk when a token needs them.

Memory hierarchy

Faster, smaller storage caches slower, bigger storage, like CPU caches over RAM.

Steps

1. Read the colibri README intro and the supported models table.
2. Look at the throughput table and note the hardware for each row.
3. Connect it to OS page cache: frequently used experts stay hot.
4. Estimate how much NVMe bandwidth one token needs.
5. Self check: why does this work for MoE but not for a dense model of the same size?

Steps

1. Clone with git clone https://github.com/JustVugg/colibri && cd colibri/c.
2. Run ./setup.sh as the README says.
3. Download the smallest supported model your disk can hold.
4. Start ./coli chat and time tokens per second for 3 prompts.
5. Watch disk read speed while it generates.

Small angles to try

• Run it twice and see if warm cache speeds it up
• Compare an SSD with a slower drive
• Try a longer prompt and see how prefill time scales

Key ideas

RGBA

An image with a fourth alpha channel that stores transparency per pixel.

Reference image editing

Giving the model example images to keep a character or style consistent.

Diffusers pipeline

Hugging Face's library wrapper that loads a model and runs generation in a few lines.

Steps

1. Read the Qwen-Image-2.1 model card.
2. Find the example with transparent output.
3. Compare with removing a background from a normal image.
4. Check the license section.
5. Self check: why is text rendering hard for image models?

Steps

1. Install diffusers from source with pip install git+https://github.com/huggingface/diffusers.
2. Load QwenImage21Pipeline from Qwen/Qwen-Image-2.1 as on the model card.
3. Generate a logo with transparent background and short text.
4. Open the PNG in an editor and check the alpha channel edges.
5. Try the same prompt with Japanese text.

Small angles to try

• Compare with a background removal tool on a normal output
• Edit with 2 or 3 reference images
• Measure seconds per image at two resolutions

Key ideas

Sandbox escape

Code inside a VM or container reaching the host machine.

Egress policy

Rules about which outside addresses a sandbox may contact.

Shared CDN IP

Many sites sit behind the same IP, so IP allowlists can let in more than intended.

Steps

1. Read Perplexity's Escaping SPACE Part I post.
2. Note which models got through with and without source access.
3. Compare with your own Docker network settings.
4. Think about why domain allowlists are safer than IP allowlists.
5. Self check: what did the fix need to change to stop DNS based bypass?

Steps

1. Create a container with no network and confirm it cannot resolve any name.
2. Create one with an allowlist proxy that only permits one domain.
3. From inside, try to reach a second domain and log what happens.
4. Check whether the proxy decides by hostname or by IP.
5. Write down one gap you found and how you would close it.

Small angles to try

• Run a coding agent inside your sandbox and review its network log
• Compare Docker default bridge with a custom network
• Add DNS logging to see every lookup

Key ideas

Third party tracker

Code from another company on a page that reports what you do.

Persistent identifier

An ID that links your events across sessions.

Capability URL

A link that grants access to anyone who has it.

Steps

1. Read the paper's summary and findings.
2. Read the HN discussion for developer reactions.
3. Open your browser's network tab on a chat app you use.
4. Compare with analytics on a normal website.
5. Self check: why is a chat title riskier than a page URL?

Steps

1. Open a chat app you use with dev tools open on the network tab.
2. Start a harmless chat with a unique made up word.
3. Filter requests by third party domains.
4. Search request bodies for your unique word.
5. Note which domains, if any, received it.

Small angles to try

• Repeat with share links
• Compare two chat apps
• Try with a content blocker on and off

Key ideas

Headless CMS

Content management separated from the site that displays it.

Plugin sandbox

Running plugins with limited permissions.

Decentralized registry

Publishers host plugins themselves.

Steps

1. Read the Cloudflare blog post.
2. Look at how plugins declare permissions.
3. Compare with WordPress plugins.
4. Consider how an agent would edit content.
5. Self check: what does a sandbox stop a bad plugin from doing?

Steps

1. Create a project with npm create emdash@latest.
2. Start it locally following the prompts.
3. Add a page and a blog post.
4. Install one plugin and read its permissions.
5. Deploy only if you choose, to your own account.

Small angles to try

• Ask a coding agent to add a content type
• Compare build time with a plain Astro site
• Write a tiny plugin

Key ideas

Harness

The code around a model that gives it tools, memory and a loop.

Host agent

A planner that delegates.

Skill library

Saved procedures reused later.

Steps

1. Read the abstract and main figure.
2. List the harnesses it composes.
3. Compare with your current single agent setup.
4. Look at the evaluation tasks.
5. Self check: when does delegation add overhead instead of value?

Steps

1. Pick a task with a research part and a coding part.
2. Run it end to end with one agent.
3. Split it: one agent for research, another for code.
4. Compare time and quality.
5. Save the useful steps as a reusable instruction file.

Small angles to try

• Add a third agent for review
• Measure tokens spent
• Reuse the saved skill on a new task

Key ideas

Deduplication

Counting each account once.

Seeded random draw

A draw anyone can reproduce given the seed.

Commit then reveal

Publishing a hash of the seed first so it cannot be changed later.

Steps

1. Read the campaign post.
2. List the rules that affect eligibility.
3. Think how you would collect entries.
4. Learn what a seeded shuffle is.
5. Self check: how could a viewer verify the draw was fair?

Steps

1. Generate a list of 50,000 fake entries with some duplicates.
2. Deduplicate by account.
3. Hash a secret seed and publish the hash.
4. Shuffle with the seed and take 100 winners.
5. Reveal the seed and rerun to confirm the same winners.

Small angles to try

• Add a bot filter rule
• Use a public future value as the seed
• Time it with a million entries

Key ideas

AI interviewer

A model that asks follow up questions like a human researcher.

Open dataset

Data released for anyone to analyze.

Selection bias

People who opt in are not everyone.

Steps

1. Read Anthropic's study page.
2. Note eligibility and dates.
3. Think about what questions you would ask.
4. Consider how the data will be anonymized.
5. Self check: what biases does an opt in study have?

Steps

1. Read the study page.
2. Decide if you want to join.
3. If you join, note the questions it asks.
4. Compare to a human survey.
5. Write down one thing you would want from AI.

Small angles to try

• Compare with the previous study's findings
• Plan an analysis once the data is released
• Discuss with a colleague