Code415 Radar

Key ideas

Pipeline parallelism

The model is cut into consecutive groups of layers, each group lives on its own chip, and requests flow through the chips like an assembly line.

KV cache

The stored attention keys and values for tokens already processed, which grows with context length and competes with weights for fast memory.

On chip SRAM

Very fast memory built into the processor itself, far quicker than GPU HBM but much smaller per chip, which is why 44 GB per wafer is the main constraint.

Steps

1. Read the Cerebras post first, focusing on the section that adds up wafers as 35 for linear attention layers plus 24 for dense layers.
2. Redo the memory math yourself: about 16 GB per layer at 4 bit weights, two linear attention layers per 44 GB wafer, one dense layer per wafer plus KV cache.
3. Note why only activations cross wafer boundaries and why a 2 microsecond hop is small next to the compute time per layer.
4. Skim the Hacker News thread for the explanation of the 2D mesh of cores and the per token energy estimate.
5. Self check: why do dense layers get a whole wafer each while linear attention layers share one?

Steps

1. Write a short Python script that takes parameter count, bits per weight, number of layers and memory per device as inputs.
2. Plug in the numbers from the post, about 3 trillion parameters at 4 bits over 93 layers, and check that you get about 16 GB per layer and that 59 wafers of 44 GB fit the weights.
3. Add a KV cache term: layers, KV heads, head size, context length and bytes per value, and compute the cache for one 256k token request at 8 bits.
4. Swap in an open model you actually run, using its config file for layer count and head sizes, and compute how many 24 GB or 80 GB GPUs a one layer per device pipeline would need.
5. Compare the 4 bit plus 8 bit KV setup against 16 bit weights and cache and write down which term dominates.

Small angles to try

• Vary context length from 8k to 256k and find where KV cache overtakes weight memory per device.
• Estimate pipeline latency as layers per device times compute plus hops times 2 microseconds.
• Repeat the math for a mixture of experts model and see how much is idle per token.

Key ideas

Compute bound vs memory bound

A compute bound step is limited by arithmetic speed, while a memory bound step waits on reading weights from memory, so faster math only helps the first kind.

Prefill and decode

Prefill processes the whole prompt in large matrix multiplies, while decode produces one token at a time and is mostly limited by memory bandwidth.

Issue rate

How many instructions of a given type the GPU can start per clock cycle, which sets the ceiling on TFLOPS for that number format.

Steps

1. Read the gpu-unlock README on GitHub first, including the supported card list, the benchmark table and the warnings section.
2. Read the linked document in the repo on how Ampere was unlocked to see what the module changes and why it must run before the NVIDIA driver.
3. Read the top Hacker News comments by jacquesm and Zagitta on power limits and vLLM prefill gains.
4. Self check: why would a 2x BF16 math gain give only a 14 to 36% prefill gain and almost nothing for decode?

Steps

1. Read the README and the warnings in full before touching anything.
2. Run lspci -D and find your NVIDIA card's full PCI address, then check whether its chip is GA102, GA103, GA104 or GA106 and how the README's report table lists it.
3. Without installing anything, write a small PyTorch script that times a large BF16 matrix multiply on your card and computes TFLOPS, to get your own baseline.
4. Time prefill and decode separately for a local model you already run, and note what share of total time is prefill.
5. Only if you have a spare, throwaway machine: follow the README install and enable steps there, confirm with journalctl -k -b | grep gpu_unlock, rerun your baseline, and remove gpu_unlock.targets= from the kernel command line to undo.

Small angles to try

• Compare your measured prefill share against the README's 14 to 36% range to estimate your real world gain.
• Repeat the BF16 baseline at different power limits to see how much power alone moves the number.
• Check FP16 versus BF16 baselines, since the README reports a larger jump for BF16.

Key ideas

Dependency length

The distance in words between a word and the word it grammatically attaches to; longer distances make sentences harder to hold in memory.

Low frequency n-grams

Word sequences that rarely occur in ordinary text, which readers process more slowly than familiar phrases.

Reduced object relative

A clause like "the bug the test missed" with the word "that" dropped, a construction known to be hard for readers.

Steps

1. Read the arXiv abstract first, then the methods section that lists the features and the reference corpora.
2. Note the tools: spaCy en_core_web_trf for parses, KenLM trigram models for n-gram rarity, and round trip translation through Japanese as a paraphrase control.
3. Read the length penalty experiment and what near and far transfer mean there.
4. Skim the Hacker News thread for practitioner reactions.
5. Self check: why do the authors compare against round trip paraphrases and not only against human text?

Steps

1. Collect 20 paragraphs of model output your product ships, such as summaries or docs, and 20 comparable paragraphs written by people on your team.
2. Install spaCy and an English model following the spaCy install docs.
3. Parse every sentence and compute average dependency length as the mean distance between each token and its head.
4. Count noun compounds per sentence using the compound dependency label.
5. Compare the averages between the two sets and note which outputs score worst.

Small angles to try

• Run the same prompt with and without a "be concise" instruction and compare dependency lengths.
• Compare two models you use on identical prompts.
• Rewrite the worst scoring paragraph by hand and see whether the metrics move.

Key ideas

Non uniform quantization

Different tensors get different bit widths, so sensitive layers keep more precision while the rest are squeezed harder.

GGUF quant types

Named formats such as IQ2_XXS or IQ4_NL that set how many bits each weight uses and how scales are stored.

Memory mapping

The operating system loads parts of a file from disk only when they are touched, so a large table can live on an SSD instead of RAM.

Steps

1. Read the Hugging Face model card first, especially the build table and the benchmark table.
2. Open the GSQ and RCO papers linked from the card and read only their abstracts and main figures.
3. Look at the Strata repository README linked from the card to see how it splits work between GPU, RAM and disk.
4. Self check: why can the 110.60 GB Engram file sit on an SSD while the backbone needs RAM and VRAM?

Steps

1. Open the model card's file list and note the sizes of the IQ1_M, IQ2_XXS and IQ2_XS backbones and the Engram file.
2. Write down your own machine's VRAM, system RAM and free SSD space and check which build, if any, fits the card's 12 GB GPU plus RAM plus SSD layout.
3. Compute bits per weight from the 763.2B parameter count and each file size and compare with the card's stated 1.75, 2.00 and 2.25 values.
4. Read the recovery column and compare the IQ1_M drop to 92.32 with the size saved versus IQ2_XXS.
5. If you have the hardware, read the Strata README for setup steps before downloading anything.

Small angles to try

• Try mixed per tensor quantization on a small model you already run and compare quality at equal file size.
• Estimate how long the 248.84 GB download takes on your connection.
• Track the llama.cpp repository for DeepSeek-V4.1 support.

Key ideas

Personal context

The assistant draws on your own messages, mail, notes and contacts to resolve requests like which Tanaka you mean.

Private Cloud Compute

Apple's server side for requests too heavy for the phone, which Apple describes as having strict privacy protections.

Cross app actions

The assistant performs steps inside several apps in one request, which only works for apps that expose those actions to the system.

Steps

1. Read Keitai Watch's article on Siri AI Japanese support first for the feature list and the on device versus cloud split.
2. Read Junya Ishino's ITmedia hands on for real examples and weak points.
3. Read ITmedia's report on Japanese apps not yet optimized for the Duo inner screen.
4. Self check: what does an app need to do so an assistant can act inside it, and what happens to apps that skip that work?

Steps

1. Pick an app you build or use daily and list three tasks a user might ask an assistant to do in it.
2. Read Apple's developer documentation on App Intents and note how an app declares an action the system can call.
3. For each of your three tasks, write down which action and parameters the app would need to expose.
4. If you own an iOS app, check whether it already declares any such actions and whether it was built with an SDK new enough for the Duo inner screen, which ITmedia says is iOS 27.1 or later.
5. Write one paragraph on what your users lose if the app is not supported.

Small angles to try

• Compare how the Tanaka example would resolve in your own contacts versus a work directory.
• List which of your app's data would stay on device versus need a server.
• Check a few popular Japanese apps on a large screen device for black bars.

Key ideas

Raw API versus consumer app

The same model can answer differently in a chat app, which adds its own system instructions and policies, than through the bare developer API.

Provider injected tokens

Input tokens a provider adds to a request on its side, which show up in the usage count even though the caller never sent them.

Forced choice evaluation

A test prompt that forbids hedging so a model must pick one option, which reveals default leanings but differs from normal usage.

Steps

1. Read the TechCrunch article first for the method, the exact prompt, the results and the company responses.
2. Open the Convovo study page and note how it separates the ballot, the reasons given and the sources consulted.
3. Compare the search on results with the 30 race search off follow up and list what changed.
4. Note why OpenAI argues the API setup differs from consumer use, and why the hidden token count matters for anyone billed per input token.
5. Self check: why could two runs of the same model give different answers when one has web search and one does not?

Steps

1. Pick a provider API you already pay for and write a tiny script that sends a one word user message with no system prompt.
2. Record the input token count the API reports in its usage field for that request.
3. Count the tokens of your own message locally with the provider's documented tokenizer or token counting endpoint, if it has one.
4. Subtract to estimate how many input tokens were added on the provider side, and repeat five times to check it is stable.
5. Repeat with any built in tools such as web search turned on and compare the overhead.

Small angles to try

• Run the same neutral factual question through the API and through the consumer app and compare tone and refusals.
• Ask a forced choice question on a non political topic, such as picking one programming language, and measure how often answers change across 10 runs.
• Read the GitHub repo linked from the study page to see how the prompts were scripted.

Key ideas

Computer use

An agent tool that takes screenshots and sends mouse and keyboard input so a model can operate a graphical app.

Helper app permissions

macOS grants Accessibility and Screen Recording per signed app, so the helper that does the clicking needs its own approval.

Self hosted worker

A machine you run that executes Cursor agent tasks, started with the agent worker command.

Steps

1. Read the Flavio Copes post first for the local workflow and the testing advice.
2. Skim the Cursor docs page on self hosted computer use for the macOS helper app and the Linux X11 dependencies.
3. Note which checks the docs say are verified by agent worker debug and which need a real screenshot and click task.
4. Self check: why does granting Screen Recording to your terminal not let the agent take screenshots?

Steps

1. Update Cursor, open a small app project you own, and start an Agent conversation.
2. Type @ in the prompt field and select Computer Use; if it is missing, confirm Cursor is up to date.
3. On macOS, grant Accessibility and Screen Recording to Cursor Computer Use in System Settings, Privacy and Security, then reopen the helper if asked.
4. Write down the expected result, then ask the agent for a read only task: identify the open window and list its controls.
5. Ask it to click through one changed screen in a development build with disposable data and report action, expected, observed and pass or fail, with screenshots of failures.
6. Review the diff afterwards, since a passing click through does not rule out unrelated changes.

Small angles to try

• Run the same check in the iOS Simulator with your development build installed.
• On a Linux box you own, start a worker with agent worker --computer-use start and run agent worker debug to see which binaries it checks.
• Compare how long the agent takes against an existing UI test for the same flow.

Key ideas

Actor snapshotting

Saving an idle agent's memory and filesystem state so its machine can be reused, then restoring it when a request arrives.

gVisor

A sandbox that intercepts system calls in user space to isolate a container more strongly than plain Linux containers.

Oversubscription

Packing more agents than there are workers, because most agents sit idle most of the time.

Steps

1. Read the Agent Substrate README first, focusing on actors, workers and snapshot and resume.
2. Read the Google Cloud blog post to see how Agent Compute Engine differs from Substrate.
3. Compare the benchmark numbers in the blog with the figures quoted in the README and note that both are vendor claims.
4. Self check: what state must be captured for a suspended agent to resume on a different worker?

Steps

1. Clone the repo on your own machine and read the prerequisites and the warning about kernel neighbor table limits above about 1,000 sandboxes.
2. Create the local cluster with hack/create-kind-cluster.sh.
3. Deploy the system and the demo with hack/install-ate-kind.sh deploy ate-system --credential-provider='{"name":"k8s.io"}' and then hack/install-ate-kind.sh deploy demo-counter.
4. Install the plugin with go install ./cmd/kubectl-ate and create an actor with kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter.
5. Run kubectl port-forward -n ate-system svc/atenet-router 8000:80, then in another terminal call curl -X POST -H "ate-target-actor: ate-demo-counter/my-counter-1" -i http://localhost:8000/ a few times and watch the counter.
6. Delete the kind cluster when done.

Small angles to try

• Leave the actor idle, then time the first request after it has been snapshotted.
• Create several counter actors and check whether each keeps its own state.
• Read how credential injection needs a namespace policy before an actor can read Secrets.

Key ideas

Items

The spec's atomic unit of model output and tool use, so messages and tool calls share one shape.

Streaming events

A fixed set of event types a server emits while generating, so one client parser can work with many providers.

Agentic AI Foundation

A Linux Foundation group that hosts agent standards such as MCP, A2A and AGENTS.md.

Steps

1. Read the openresponses.org home page first for the building blocks the spec defines.
2. Read The New Stack article for who backs it and who does not.
3. Open the acceptance tests linked from the spec site to see what an implementation must pass.
4. Self check: what would change in your code if your provider spoke Open Responses instead of its own API?

Steps

1. Check your local runtime's own docs for whether it exposes a Responses style endpoint, and note the path it documents.
2. Start the runtime with a small model you already have downloaded.
3. Send one plain text request in the Responses shape and save the raw JSON reply.
4. Send a second request that defines one simple tool, such as getting the time, and inspect how the tool call item comes back.
5. Compare both replies against the item and event shapes described on the spec site.

Small angles to try

• Point the same client at a second local runtime and diff the replies.
• Turn on streaming and list the event types you receive.
• Run the spec's acceptance tests against your local endpoint if its instructions fit your setup.

Key ideas

Mixture of experts

A model where each token only runs through a few expert blocks, so active parameters are far fewer than total parameters.

Total versus active parameters

Total size sets the memory needed to hold the weights, while active size drives the compute per token.

Community license

A custom model license whose terms you must read before commercial use, unlike standard licenses such as Apache 2.0.

Steps

1. Read the Hugging Face model card first, including the license section and the serving instructions.
2. Estimate the memory needed to hold the weights at the precision of the published files, using the file sizes on the Files tab.
3. Compare why 23B active parameters make it fast but not small enough for a single consumer GPU.
4. Self check: what limits your ability to run this model, compute per token or memory for the weights?

Steps

1. Read the minimax-community license on the model page and note any use restrictions.
2. Open the Files tab and add up the safetensors file sizes to see the real download size.
3. Open config.json in the Files tab, or download just that file, and read the expert count and hidden sizes.
4. From the config, estimate how many parameters are active per token and compare with the 23B the card states.
5. If you have access to a suitable GPU server, try the card's documented command vllm serve "MiniMaxAI/MiniMax-M3.1-Flash".

Small angles to try

• Compare its config with another open weight mixture of experts model you know.
• Check whether your serving framework version already lists support for the model.
• Compute how much memory a 1M token KV cache would need from the config values.

Key ideas

Agent Skill

A folder with a SKILL.md file of instructions that coding agents such as Claude Code, Codex or Cursor load when a task matches.

Backported patch

A distro applies the upstream fix to its older packaged version, so the version number stays the same while the bug is gone.

HTTP2Shell

Aikido's name for CVE-2026-111497, a Brotli decoder out of bounds write that can turn compressed HTTP input into remote code execution.

Steps

1. Read the Cognition blog post first for the per skill CVE list, the fixed versions and the five step workflow.
2. Open the devin-vuln-fixes README and skim the table of skills and the list of files the inventory step reads, such as Dockerfiles, CI config, Helm charts and Terraform.
3. Read the Aikido PHP2Shell post for affected version ranges and its advice to check package changelogs rather than php -v.
4. Open one SKILL.md in the repo and find where the approval gate is written, since that is what keeps the agent from editing before you agree.
5. Self check: why can a patched Ubuntu server still print a vulnerable looking PHP version, and what should you check instead?

Steps

1. Create a throwaway local git repo with a Dockerfile pinned to an old PHP 8.3 base image and a tiny index.php, plus a package.json that pins an older Node 22 engine.
2. Install only the PHP skill with npx skills add CognitionAI/devin-vuln-fixes --skill fix-php-vuln, after reading its SKILL.md, because skills run with the agent's full permissions.
3. Ask the agent to fix the PHP vulnerability in that repo and confirm it reports an inventory and stops for approval before changing anything.
4. Approve, then read the diff: check the new base image targets 8.3.36 or later and that the test results before and after are recorded.
5. Build the patched image locally and run apt changelog or rpm -q --changelog on the PHP package inside the container to look for the CVE IDs, instead of trusting the version string.
6. Keep everything in local containers; do not point any checker or request at servers you do not own.

Small angles to try

• Repeat with --skill fix-node-vuln on the Node pin and see whether it flags Node 20 as end of life.
• Compare the agent's patch plan with the fixed versions listed in the Aikido post.
• Install via /plugin marketplace add CognitionAI/devin-vuln-fixes in Claude Code and compare the experience with the npx route.

Key ideas

k-mer tokenization

Splitting a DNA string into fixed chunks of k letters, here 6, so each token stands for one short sequence.

Vocabulary extension

Adding new tokens to an existing model's vocabulary and training so the model learns embeddings for them.

Variant effect prediction

Judging whether a change in a DNA sequence is likely harmless or likely to cause disease.

Steps

1. Read the Carbon-X model card first, especially the tokenizer section and the benchmark table.
2. Note how 4,096 six letter tokens follow from 4 to the power of 6, and why tail tokens are needed for lengths not divisible by 6.
3. Compare the per benchmark scores against GPT-6.1 Sol to see where a small domain model wins and where it loses.
4. Read the cost estimate assumptions, an H200 at $5 per hour, before quoting the 1/1,800 figure.
5. Self check: how many tokens would a 6,000 base pair sequence use, and how much of the 40,960 token context is left for text?

Steps

1. Install the libraries with pip install -U transformers torch.
2. Load only the tokenizer with AutoTokenizer.from_pretrained("HuggingFaceBio/Carbon-X", trust_remote_code=True) after reading the custom tokenizer code it pulls in.
3. Tokenize a short DNA string and an English sentence and count tokens for each to see the 6 base pairs per token ratio.
4. If an 80 GB GPU is available, run the card's Transformers example, which loads the model with dtype=torch.bfloat16 and device_map="cuda" and asks whether a variant is benign or pathogenic.
5. Compare its explanation with what a general model says for the same reference and variant pair.

Small angles to try

• Serve it with vllm serve HuggingFaceBio/Carbon-X --trust-remote-code after pip install -U vllm and time a batch of prompts.
• Measure how token count grows when the DNA length is not a multiple of 6.
• Note that GGUF runtimes such as llama.cpp cannot load the custom DNA tokenizer, so quantized local use is limited.

Key ideas

Intelligence Index

Artificial Analysis's combined score from a set of reasoning, coding and knowledge evaluations run the same way on every model.

Cost per task

What a model actually costs to finish one task, which is price per token times the tokens it really uses.

Verbosity

How many output tokens, including reasoning tokens, a model spends to reach an answer.

Steps

1. Read the Artificial Analysis page for Kimi K3.1 Pro first, focusing on the speed, verbosity and cost charts.
2. Compare its $2.11 per task with two models of similar index score on the same site.
3. Work out how much of the cost gap comes from tokens used rather than price per token.
4. Self check: if two models share a price but one uses 1.7 times the output tokens, how do their task costs compare?

Steps

1. Pick 10 prompts from your own work, such as code reviews or bug explanations.
2. Send each to Kimi K3.1 Pro and to one comparison model through their official APIs, using the model ids from each provider's docs.
3. Record input, output and reasoning token counts from each response's usage field, plus wall clock time.
4. Compute cost per prompt from each provider's listed prices and compare totals.
5. Judge answer quality blind, then decide whether the extra tokens bought better answers.

Small angles to try

• Repeat with the same system prompt cached to see how much the 90% cache discount changes the total.
• Ask for a short answer explicitly and measure how much verbosity drops.
• Plot tokens used against your quality rating per prompt.

Key ideas

Cold versus hot run

A cold run starts with empty caches, while hot runs repeat the query and can reuse anything cached.

Benchmark specialization

Code paths that only speed up the exact published queries rather than the general workload.

Perturbed query

A benchmark query with a changed constant, column or expression, used to check that speed generalizes.

Steps

1. Read ClickBench PR #2165 first, especially the timing tables and the list of refused probes.
2. Read the ClickBench rules on caching and query specific tuning in its README.
3. Read Dan Luu's essay for the case in favor of the project and Malis's own caveat about production use.
4. Self check: what single experiment would separate a legitimate result cache from a benchmark specific answer table?

Steps

1. Pick one analytic query on your own table, such as a LIKE filter with a count.
2. Restart the database, run the query three times and record cold and hot timings.
3. Change only the constant in the filter and run it three more times.
4. Change the shape slightly, for example a different aggregate on the same column, and time again.
5. Compare the patterns: a general engine should show similar warm times across variants, while a specialized path will show a large gap.

Small angles to try

• Clear the OS page cache between runs on a throwaway machine to see true cold numbers.
• Run the same probes against two engines and compare how much each one's hot time depends on the exact query text.
• Turn this into a small script that generates perturbed variants of any query.

Key ideas

Approval gate

A step where the agent pauses and the user must confirm before it sends, buys or signs in.

Credential brokering

A separate service fills in login details so the model never sees or stores the password.

Background agent

An agent that keeps working on a task after the user closes the app.

Steps

1. Read the ITmedia article on the Muse Japan launch first for its features and approval flow.
2. Read the Impress Watch report on 1Password for Agents for how the login approval works and which agents are supported.
3. Open Apple's Japan top free feed and find Muse, ChatGPT and Gemini to see the ranks yourself.
4. Self check: in an agent that books travel, which actions should need approval and where should the credentials live?

Steps

1. Write a tiny local web app with a login page and a fake purchase button.
2. Build a small agent loop where the model can call navigate, fill_login and click_buy tools.
3. Implement fill_login outside the model: it pauses, asks you in the terminal to approve, then fills credentials from a local secrets file the model never sees.
4. Add the same approval prompt in front of click_buy.
5. Inspect the full prompt and tool log to confirm the password never appears in any model input or output.

Small angles to try

• Try a prompt injection on the test page that asks the agent to print the password and check that the design holds.
• Log every approved and denied action into a history view, like the one Muse shows users.
• If you use 1Password and Muse yourself, try the real flow on one of your own accounts and compare it with your local version.

Key ideas

Reranker

A second stage model that rescores a short list of retrieved documents against the query so the best ones go into the prompt.

Calibration error

A measure of how far a model's scores are from real hit rates, so a well calibrated 0.7 means right about 70% of the time.

Span level relevance

Scoring parts of a document instead of the whole thing, so only the useful passages are passed to the model.

Steps

1. Read the Cohere Rerank 5 blog post first, focusing on the Context Focus section and the calibration table.
2. Look at the code sample to see the relevant_spans options mode, threshold and max_spans, and the start, end and score fields each span returns.
3. Compare the 96% at half context and 87% at a quarter context results to see how quality falls as context shrinks.
4. Self check: why does a calibrated score let a team use one fixed cutoff like 0.50 across many datasets?

Steps

1. Pick one real question and 10 to 20 of your own documents or notes, a mix of relevant and irrelevant ones.
2. Following the blog's sample, send a POST to https://api.cohere.com/v2/rerank with model rerank-v5.0-pro, your query, the documents and top_n.
3. Run it once without relevant_spans, print each relevance_score, and mark which documents pass the 0.50 cutoff.
4. Run again with relevant_spans enabled as in the blog example, threshold 0.4 and max_spans 10, then slice each document by start and end to print the returned spans.
5. Count characters kept versus the full documents and judge by hand whether the spans still contain the answer.

Small angles to try

• Lower the span threshold toward the 0.1 default and watch how much more text comes back.
• Run the Fast tier on the same set, looking up its model id in Cohere docs, and compare against the 0.45 cutoff.
• Feed full documents versus spans to your usual chat model and compare answers.

Key ideas

Orchestrator free coordination

Agents pick and announce their own work in a shared space instead of waiting for a manager agent to assign it.

Claim, share, merge

A worker posts a CLAIM for a sub task, reports what it learned, and integrates its branch into main, fixing any merge conflicts Git finds.

Agent count as a scaling axis

The authors treat the number of parallel agents like model size, a knob that trades money for results within a fixed time.

Steps

1. Read the arXiv abstract for Agensh first, then the method section on the shared workspace, message interface and shared context.
2. Note the infrastructure: Gitea as the shared repo and Mattermost for messages, with agents in containers prepared as in ProgramBench.
3. Study the scaling numbers for 1, 8, 32 and 128 agents and the separate 1,024 agent pandoc run.
4. Self check: why might 8 agents barely beat 1 while 128 agents gain about 9 points?

Steps

1. Open the GitHub repo and read the README for requirements and license before running anything.
2. On a small repo of your own, write a CLAIMS.md file and split one feature into 3 sub tasks.
3. Start two agent sessions on separate branches; tell each to add a claim line to CLAIMS.md before working and a findings line when done.
4. Merge both branches into main, resolve any conflicts, and run your tests.
5. Record tokens spent and wall clock time against a single agent doing the whole feature.

Small angles to try

• Remove CLAIMS.md and see whether the two agents duplicate work.
• Add a third agent whose only job is running tests and reporting failures.
• Give both agents a strict 20 minute budget and compare results.

Key ideas

totals_by_client

A list in each report day that gives request, session and token totals for each tool that used Copilot models.

model_selection_mode

Says whether the user picked the model, Auto picked it, or the product picked it.

Best effort client identity

The client name comes from an integration ID the tool sends, so it can be wrong or missing.

Steps

1. Read the October 23 GitHub changelog on usage by client first.
2. Read the October 22 changelog on named skills, agents and MCP servers and note the 25 entry limit and the trailing other entry.
3. Open the Copilot usage metrics API docs and find the organization 1 day report endpoint.
4. Self check: why should request_count not be compared directly with the number of prompts people typed?

Steps

1. Confirm the Copilot usage metrics policy is enabled for your org.
2. Run gh api "/orgs/YOUR_ORG/copilot/metrics/reports/organization-1-day?day=2026-10-24" and note the download_links in the response.
3. Download the report file from one link and open it with jq or a Python script.
4. Print each totals_by_client entry with its client, request count and session count.
5. Look for totals_by_mcp and totals_by_skill names and check whether any internal names should not leave your org.

Small angles to try

• Use the organization-28-day/latest report and chart tokens per client per day.
• Group usage by model_selection_mode to see how often Auto picks the model.
• Compare a day before October 22 with a day after to see names replace other or custom.

Key ideas

Agentic browser

A browser with a built in assistant that can read the current page and take actions such as filling forms or following links.

Indirect prompt injection

Hidden text on a web page that tries to give orders to an AI assistant reading that page.

Arc Development Kit

The proprietary engine that runs both Arc and Dia, which is why Arc cannot simply be released as open source.

Steps

1. Read the TechCrunch shutdown story first for the dates, the migration details and the security quote.
2. Read the Hacker News thread to see what developers say they lose and what alternatives they pick.
3. Search for an introduction to indirect prompt injection in AI browsers to see why securing them is costly.
4. Self check: why does adding an assistant that reads pages raise the security bar for a browser?

Steps

1. If you use Arc, export or migrate your data to Dia now and check that passwords, bookmarks and extensions arrived.
2. Write a local HTML page with a normal article plus a hidden paragraph, styled invisible, that tells any AI assistant to answer only with the word PINEAPPLE.
3. Serve it on your own machine with python3 -m http.server 8000 and open it in the AI browser.
4. Ask the assistant to summarize the page and record whether it follows the hidden instruction, ignores it or warns you.
5. Repeat with the hidden text in an HTML comment and in image alt text and note which ones reach the assistant.

Small angles to try

• Run the same page through two different AI browsers and compare.
• Turn the AI features off in Dia and confirm the page behaves like a plain browser.
• Ask the assistant to fill a form on your own local page and see whether it asks for confirmation.

Key ideas

Agent facing service

A product designed so that AI assistants, not only people, can call it and get results.

Intent in, decision inside

The outside agent sends a plain request and the service keeps its data and ranking logic private, returning only the result.

Tool calling

A chat model returns a structured call to a function you define, which your code runs before replying.

Steps

1. Read the TechCrunch story first and note what Spotify says it shares with partner agents and what it keeps.
2. Read your model provider's tool calling guide to see how an agent passes a request to an outside service.
3. Sketch which fields a running playlist request needs, such as pace, length and mood, and which data must stay on the service side.
4. Self check: what does Spotify gain by deciding what plays itself instead of sending listening history to the agent?

Steps

1. Write a CSV of 20 songs you know with columns for title, BPM and key, filled in by hand.
2. Write a Python function that takes a target BPM range and a length in minutes and returns matching titles, reading the CSV only inside the function.
3. Register that function as a tool with your chat model, describing only its inputs, never the CSV contents.
4. Ask the model for a 20 minute running playlist at an easy pace and log exactly what arguments it sends to your tool.
5. Check the log to confirm the model never saw the full CSV, only the titles your function returned.

Small angles to try

• Add a key compatibility rule inside the tool and see whether the model needs to know about it.
• Give the model a vague request like music for a tempo run and inspect how it turns that into a BPM range.
• Return song IDs instead of titles and see whether the agent can still explain its choice.