Code415 Radar

Key ideas

Cached input pricing

Tokens the provider has already seen in a repeated prompt prefix are billed at a much lower rate, here one tenth of normal input.

Tokens per second and P50

Output speed is how fast text streams back, and P50 is the median across requests, so half of calls are at least that fast.

Agentic benchmark

A test such as DeepSWE or OSWorld that scores a model on multi step tasks with tools, code or a computer screen rather than single answers.

Steps

1. Read the MiniMax M3.1 Flash blog post first, focusing on the pricing table and the benchmark chart.
2. Note which numbers are vendor reported and which details are missing, such as context length and parameter count.
3. Compare the per 1M token prices with the cheap model currently used for sub tasks in a project.
4. Estimate the cost of one typical agent run by multiplying its input, cached input and output tokens by the three prices.
5. Self check: if an agent run uses 2M fresh input tokens, 8M cached input tokens and 0.5M output tokens, what does it cost on M3.1 Flash?

Steps

1. Create an API key on the MiniMax platform and read its quick start to find the base URL and request format.
2. Send one short chat request with the model id MiniMax-M3.1-Flash and confirm a reply comes back.
3. Run 10 of your own coding prompts with streaming on and record time to first token and total output tokens per second.
4. Send the same long system prompt twice and compare the reported cached token counts and billed cost.
5. Score the 10 answers by hand against the current default model on the same prompts.

Small angles to try

• Use it only for sub tasks such as file search or test writing inside an existing agent and compare total run cost.
• Measure how speed changes with output length, short replies versus 2,000 token answers.
• Repeat the test at a busy hour of the day to see whether the 170 tokens per second median holds.

Key ideas

Diffusion language model

Instead of writing one token at a time, the model starts from a rough draft of many tokens and refines them in parallel over several steps.

Autoregressive decoding

The usual LLM method of predicting each next token from all previous ones, which makes speed depend on output length.

OpenAI compatible API

An endpoint that accepts the same request format as OpenAI chat completions, so existing clients work by changing the base URL and key.

Steps

1. Read the Inception blog post on Mercury 3 first, noting the speed, context and pricing claims.
2. Read a short explainer on diffusion versus autoregressive text generation to see why parallel refinement can be faster.
3. Read the Inception get started docs to see the base URL and how a request is shaped.
4. List which claims are measured by Inception itself and which benchmark suites are not named.
5. Self check: why would a diffusion model help a voice agent more than a batch summarization job?

Steps

1. Create a key on the Inception API platform and export it as INCEPTION_API_KEY.
2. Point the OpenAI Python client at https://api.inceptionlabs.ai/v1 and look up the current Mercury 3 model id in the Inception docs model list.
3. Send 10 prompts with streaming on and log time to first token, total time and output tokens per second.
4. Run the same prompts against the fast model already in use and put both sets of timings in one table.
5. Read the answers side by side and mark which ones are wrong or weaker.

Small angles to try

• Try a long context prompt of several hundred thousand tokens to test the 1M window.
• Compare low and high reasoning_effort settings for speed and quality.
• Call the same model through OpenRouter and compare latency with the direct API.

Key ideas

LoRA

A fine tuning method that trains small low rank weight updates next to a frozen model, so the adapter is tiny, here 126 MB.

Distillation from a teacher

A stronger model writes example answers, the best are kept by filters, and the small model is trained to imitate them.

Neuron edit

Directly changing the weights of a few specific neurons that were found to drive a behavior, instead of retraining.

Steps

1. Read the author's write up on 100toolkit first, start to finish.
2. Write down the pipeline in four steps: prompts, teacher answers, filters, LoRA, then the neuron edit.
3. Look at how the slop score and em dash rate are defined and why a factual quiz is included.
4. Skim the Hacker News thread for critiques of the metric.
5. Self check: why does the author test 30 factual questions after removing style tics?

Steps

1. Install Ollama from its official site if it is not already present.
2. Run ollama run hf.co/erikkannike/qwen3-4b-deslop-v4-GGUF:Q4_K_M to download and start the model.
3. Pull the plain Qwen3-4B-Instruct-2507 model in a similar quantization for comparison.
4. Ask both models the same five writing prompts, such as a product announcement and a short essay.
5. Count em dashes and stock phrases per answer and note any factual mistakes.

Small angles to try

• Ask both models ten factual questions from your own field to check knowledge loss.
• Use it to rewrite AI written text from another model and judge the result.
• Compare speed and memory use against the base model on the same machine.

Key ideas

Agent harness

The thin layer of code that connects a model to an environment, here mapping shell commands to robot actions.

Tool evolution

The agent writes small programs, tests them, repairs them after failures, and keeps the ones that work as reusable code.

Triangulation

Finding a 3D point by intersecting camera rays from two or more calibrated views, or one ray with a known plane like a table.

Steps

1. Read the arXiv abstract and the method section of the HexaShell paper first.
2. Open the HexaShell GitHub repo and read the tool contract file interface/EpisodeAPI.md.
3. Open one task folder under tasks/ and read a frozen tool.py to see how a triangulation tool is written.
4. Compare the per category results and note where contact heavy tasks fail.
5. Self check: why were tools built on 10 layouts and then tested on 10 new ones?

Steps

1. Clone the HexaShell repository from GitHub.
2. Read interface/EpisodeAPI.md and list the calls a tool is allowed to make.
3. Pick two tasks, one pushing and one tight insertion, and read their enabled_tools.txt and tool code.
4. Open the results folder and compare success for those two tasks.
5. Write a short note on what extra sensing or control an insertion tool would need.

Small angles to try

• Count how many of the 227 tools are enabled per task and see which tasks needed the most.
• Write a small fake api object from the contract and unit test one triangulation tool on synthetic points.
• Apply the same loop, read the log, patch a tool, freeze it, to a coding agent in a software project.

Key ideas

GEO

Generative engine optimization, making content easy for AI assistants to find, cite and recommend, the AI era version of SEO.

Mention rate

The share of AI answers that name a given brand, measured over many repeated runs of the same prompt.

Citation source

The web page an AI answer links to as evidence, which often decides which brands appear.

Steps

1. Read the ITmedia NEWS article first, with a translator if needed, for the adoption numbers.
2. Read the Branding Works PR TIMES release and write down its exact method: models, prompt, search setting, days and runs.
3. Note why ChatGPT and Gemini behaved so differently on the same question.
4. List what makes the study weak: vendor bias, small sample, one prompt.
5. Self check: if a product never appears in AI answers, which pages would you check first, based on the citation data?

Steps

1. Write one neutral prompt asking for a recommendation in a product category you work in.
2. Send it through the API with web search enabled three times, saving the full answer and cited URLs.
3. Repeat the same runs with a second model.
4. Count which brands are named and which domains are cited, as shares of all answers.
5. Schedule the same script daily for a week and see how stable the mentions are.

Small angles to try

• Ask the same question in Japanese and in English and compare the brands named.
• Turn web search off and see how much the answers change.
• Check whether your own product pages appear among the cited domains.

Key ideas

Strict success

A task counts only if the goal is reached and every step along the way followed the rules, not just the final state.

Execution validity

Checks on the path an agent took, such as honoring limits and earlier commitments, scored separately from the outcome.

Tool discovery

The agent has to find the right tool among hundreds instead of receiving a short curated list.

Steps

1. Read the arXiv abstract for 2610.20424 to get the setup and the headline numbers.
2. Open the BizAgentBench README on GitHub and note what is included: tasks, knowledge documents, a simulated database, tools and scoring rules.
3. Compare goal attainment against strict success for Qwen3.8-27B and write down why the gap matters.
4. Look at the tool call count for DeepSeek-V4.1-Flash against the reference runs and think about cost.
5. Self check: why can an agent reach the right final state and still fail BizBench?

Steps

1. Clone the BizAgentBench repo, then run python3.12 -m venv .venv and . .venv/bin/activate.
2. Install dependencies with python -m pip install -r requirements.txt and copy the config with cp configs/model.example.json configs/model.json.
3. Edit the agent and user sections of configs/model.json, then export AGENT_API_KEY and USER_API_KEY in your shell.
4. Check the install without model calls: python -m unittest discover -s tests -v.
5. Run one task as a smoke test: python tools/evaluate.py --config configs/model.json --output runs/smoke --numbers 1 --workers 1.
6. Read the output in runs/smoke and compare the outcome score, the strict score and the number of tool calls.

Small angles to try

• Run the same task with two models and compare tool call counts, not just pass or fail.
• Run a few tasks with a small local model behind an OpenAI compatible server to see where strict checks fail.
• Add --resume to an interrupted run to continue in the same output directory.

Key ideas

Design first delegation

A human sets architecture, abstractions and interfaces, and the model only fills in the implementation.

Plan review

Reviewing the agent's written plan and the resulting behavior instead of reading every generated line.

Staged rollout

Releasing a change to a small slice of traffic first so problems show up before everyone is affected.

Steps

1. Read the top of the Ask HN thread and the first reply chain from bitwizeshift.
2. Read the Silhouette replies on why generated tests can add churn without covering important behavior.
3. List the three or four practices that repeat most often across replies.
4. Self check: which part of a change should a human always own, and why are AI written tests a weak safety net on their own?

Steps

1. Pick a small feature in your own repo and write a short design note: interfaces, data shapes and edge cases.
2. Ask your coding agent to implement exactly that design and to write a plan before editing files.
3. Review the plan against your note and reject any change to the interfaces you defined.
4. Write two or three behavior tests yourself for the edge cases, then run the suite.
5. Compare the agent's own tests with yours and note which important cases it missed.

Small angles to try

• Repeat without a design note and compare how much of the result you can explain afterward.
• Deliberately break the implementation and see whether the agent written tests catch it.
• Time yourself reviewing the plan versus reviewing the full diff.

Key ideas

Change point monitoring

Watching metrics and logs right after a release or config push, when regressions are most likely.

Monitoring spec

A written description of which queries to run, what to compare and what counts as an alert.

Deterministic harness

Ordinary code that runs the spec the same way every time, so the model is not in the loop for each check.

Steps

1. Read Ben Congdon's Agentic Monitoring post from start to end.
2. Note why he says change points are where regressions concentrate.
3. Write down the difference between a pure agent loop and the spec plus harness design, and where tokens are spent in each.
4. Self check: in the spec plus harness design, which part does the model do and which part runs without it?

Steps

1. Export an hour of a metric such as error rate or latency from your own service around a recent deploy.
2. Ask a chat model to write a small JSON spec for that deploy: which metric, the before and after windows, and an alert threshold.
3. Write a short Python script that reads the spec and the data and prints whether the threshold was crossed.
4. Run it on a deploy that went fine and on one that caused trouble, and compare the results.
5. Note how many model calls the whole check needed compared with having the model read the raw data each time.

Small angles to try

• Have the model write specs for a config push instead of a binary release.
• Let the model see raw data directly and compare the noise against the spec plus script version.
• Add a log pattern count to the spec next to the metric check.

Key ideas

Style matching

Giving ChatGPT an example deck so new slides follow its layout and look.

Comment tagging

Mentioning ChatGPT in a slide comment to have it apply the requested change.

Round trip export

Moving a deck to PowerPoint or Google Slides and continuing to edit it there.

Steps

1. Read OpenAI's ChatGPT Slides feature page.
2. Read the Impress Watch article for plan and price details.
3. List which edits you would do by hand and which you would hand to ChatGPT through comments.
4. Self check: which plans include Slides, and which export formats are available?

Steps

1. Open Slides at chatgpt.com/space?view=slides and pick an outline, a file of your own or a template as the starting point.
2. Upload an existing deck of yours and ask for a new deck in the same style.
3. Edit one chart by hand, then tag ChatGPT in a comment to reword one slide.
4. Share the deck with a teammate with comment only access and check what they can change.
5. Export to PowerPoint and check whether charts and notes remain editable.

Small angles to try

• Build the same deck from a chat versus from a plain outline and compare structure.
• Export to Google Slides instead and compare formatting.
• Ask for speaker notes for every slide and check them against your source document.

Key ideas

Blackboard architecture

Agents write findings to a shared board that other agents and human operators read to plan next steps.

Unsafe deserialization

Turning untrusted data back into program objects in a way that can run attacker controlled code.

Exposed control panel

An admin dashboard reachable from the internet that leaks targets, results and secrets to anyone who finds it.

Steps

1. Read NorForge's Inside Cairn post, focusing on the architecture and the AIA0001 section.
2. Read the Slashdot summary to separate what NorForge found from what Reuters reported about victims.
3. List the entry points NorForge describes and match each to a bug class you know.
4. Self check: why do cheap, patient agents make old web bugs more urgent rather than creating new ones?

Steps

1. List every internet reachable admin panel, dashboard and agent console your team runs, including red team or testing tools.
2. For each one, confirm it sits behind authentication or a private network, and move any that do not.
3. Search your own code for file upload handlers and check type validation, storage location and that uploaded files cannot execute.
4. Search your own code for deserialization of request or session data and note any that accept untrusted input.
5. Check that login queries use parameterized statements, and rotate any credential that was ever stored on an exposed panel.

Small angles to try

• Run the same checklist against a staging copy in a local container.
• Review error handling so stack traces and internal hosts never reach users.
• Estimate how long your patch cycle is compared with the roughly 23 hours a successful agent project took.

Key ideas

Prompt processing, or prefill

The phase where the model reads the whole input before producing anything, and its speed decides how long a long prompt waits before the first token.

Token generation, or decode

The phase where the model produces output one token at a time, usually limited by memory bandwidth rather than raw compute.

Quantization

Storing model weights with fewer bits so a large model fits in limited memory, at some cost in quality.

Steps

1. Read the Gigazine article from Oct 19 first for the plain overview and the developer's reported speeds.
2. Open the PS5LM Plus README and study its table of models, quantizations, context sizes and generation speeds, ignoring the console setup steps.
3. Read the llama-bench README in the llama.cpp repo to learn what the pp512 and tg128 columns measure.
4. Compute by hand how long a 20,000 token prompt takes at 40 tokens per second of prefill, then compare that with generating a 500 token reply at 20 tokens per second.
5. Self check: why does a coding agent feel slow on hardware with decent generation speed but weak prefill speed?

Steps

1. Install llama.cpp on your own computer using one of the install options in the official llama.cpp README.
2. Start a local OpenAI compatible server with llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF and note the address and port it prints at startup.
3. Run llama-bench on the same model file with -p 512 -n 128, then again with -p 4096, and write down the prompt processing and generation speeds in tokens per second.
4. Send the server one short chat request and one request padded with a few thousand tokens of tool descriptions, and time how long each takes until the first token appears.
5. Divide your typical agent prompt size by your measured prefill speed to estimate the wait a coding agent would see before every reply.

Small angles to try

• Repeat the benchmark with a larger model or a lower bit quantization and see whether prefill or generation drops more.
• Use the -d option of llama-bench to prefill the KV cache to a deeper context and watch how speed falls as context grows.
• Point a coding agent you already use at the local server and compare its first token delay with your estimate.

Key ideas

Agent trace

A recorded tree of spans for one agent run, holding each LLM call, tool call, input and output.

Silent failure

A step that technically succeeds, such as a tool call that returns, but gives a wrong or useless result the agent then trusts.

Detection F1

A single score that balances how many real failures a detector finds against how many of its flags are false alarms.

Steps

1. Read the flow-1 blog post on laminar.sh first, focusing on the benchmark table and how Signals reads a trace.
2. Read the Launch HN thread to see what practitioners asked, such as how bad tool results differ from agent misreadings.
3. Skim the lmnr README for how traces are captured with the observe decorator in Python or the observe wrapper in TypeScript.
4. Self check: why can a model that is slightly worse on F1 still be the better choice for scanning every production trace?

Steps

1. Self host Laminar: git clone https://github.com/lmnr-ai/lmnr, then cd lmnr, then docker compose up -d, and open the UI at http://localhost:5667.
2. Create a project in the UI and copy its project API key.
3. Install the Python SDK with pip install --upgrade 'lmnr[all]', call Laminar.initialize with your key, and point the SDK base URL and ports at your self hosted instance as the README describes.
4. Add @observe() to a small agent function that calls a tool, run it a few times, and include one run where the tool returns stale or wrong data on purpose.
5. Open the traces in the UI and check whether you can spot the planted failure by reading spans, and note how long it took.

Small angles to try

• Write a JSON schema for the failure you planted and ask a model you already pay for to scan the exported trace, then compare with your manual reading.
• Plant a missing step instead of a wrong one and see whether a trace based check can notice something that never ran.
• Use the TypeScript SDK with npm add @lmnr-ai/lmnr and the observe wrapper instead of Python.

Key ideas

Model deprecation

The vendor stops serving a model, so anything that names it must move to a supported one.

Model policy

An enterprise or organization setting that decides which Copilot models members are allowed to pick.

Policy delegation

An enterprise can hand model control down to organizations, or to teams and apps, so each level manages its own list.

Steps

1. Read the deprecation changelog entry first and match each retired model to its replacement.
2. Read the model policy API changelog entry and note the four endpoint paths.
3. Open the GitHub REST docs page on AI controls linked from that entry to see the request and response shapes.
4. List every place your team names a Copilot model, such as editor settings, scripts or agent configs.
5. Self check: after a deprecation, what must an enterprise admin do that the vendor will not do automatically?

Steps

1. Authenticate the gh CLI as an admin of your own organization.
2. Call GET /orgs/{org}/copilot/model-policies for your organization with gh api and save the JSON to a file.
3. Check in that output whether GPT-6.1 Sol, GPT-6 Luna, Gemini 3.8 Flash and Grok 4.7 are enabled.
4. Using the request body shape from the REST docs, send a PATCH to the same path that enables one replacement model, then GET again to confirm the change.
5. Open the Copilot Chat model selector in VS Code or on github.com and confirm the model now appears.

Small angles to try

• Commit the saved policy JSON to a repo and diff it after each change to keep model access as code.
• Try the enterprise teams path to see how delegation to a single team works.
• Search your repos and editor configs for the retired model names and list what needs updating.

Key ideas

Style based AI detection

Judging a work as AI made from how it looks, which is unreliable and cannot be proven either way.

Content provenance

A record attached to a file that says how and with which tools it was made, such as C2PA Content Credentials.

Eligibility rules

The written conditions a contest sets in advance, which decide disputes far better than after the fact judgment.

Steps

1. Read the J-CAST News report from Oct 16 first for the timeline and the city's statement.
2. Read the Sirabee article from Oct 20 for the organizers' later comments on future rules.
3. Read an introduction to C2PA Content Credentials to see what a provenance record can and cannot prove.
4. Self check: what could the organizers have written into the rules beforehand that would have made this dispute answerable?

Steps

1. Collect three images you own: one hand drawn and scanned, one edited, and one you generated yourself with an AI tool.
2. Show them to a few colleagues without labels and record how many guess the origin of each correctly.
3. Inspect the metadata of each file with an image viewer or metadata tool and note what, if anything, reveals how it was made.
4. Draft three sentences of contest rules that state what AI use is allowed and what process evidence, such as sketches, entrants must keep.
5. Compare your rules against your guessing results and note which disputes they would settle.

Small angles to try

• Strip metadata from the files and see what evidence is left.
• Rewrite your rules for an internal hackathon or design contest instead of an art prize.

Key ideas

RESP compatibility

Speaking the Redis wire protocol so existing Redis clients and redis-cli work without changes.

HNSW index

A graph based approximate nearest neighbor index that finds similar vectors quickly at some cost in memory.

Pipelining

Sending many commands without waiting for each reply, which raises throughput and can flatter benchmark numbers.

Steps

1. Read the KiviDB README first, especially the quick start and the benchmark tables with their hardware notes.
2. Read COMMANDS.md in the repo for the FT commands, including FT.HYBRID.
3. Compare the pipelined and unpipelined GET results to see why one headline number can mislead.
4. Self check: for a small RAG cache, which matters more, peak pipelined GET throughput or memory after vector ingest, and why?

Steps

1. Start the server with docker run -p 6380:6380 kividb/kividb.
2. Create a vector index with redis-cli -p 6380 FT.CREATE idx ON HASH PREFIX 1 doc: SCHEMA vec VECTOR HNSW 10 TYPE FLOAT32 DIM 128 DISTANCE_METRIC COSINE M 16 EF_CONSTRUCTION 128.
3. With a short script in any Redis client, write a few thousand hashes under the doc: prefix, each with a 128 dimension FLOAT32 little endian vector in the vec field.
4. Query the nearest neighbors with FT.SEARCH idx '*=>[KNN 10 @vec $q]' PARAMS 2 q <blob> DIALECT 2 and check the results against a brute force cosine search in your script.
5. Record the container's memory use before and after loading the vectors.

Small angles to try

• Run the same load against a Redis Stack container you start yourself and compare memory and query time.
• Try a FLAT index instead of HNSW and compare recall with the brute force results.
• Add a text field and try FT.HYBRID following COMMANDS.md.

Key ideas

System One extraction model

A model that reads the input once and returns typed values directly, instead of writing an answer token by token like a chat LLM.

Typed field spec

You declare each field's name, type such as number or date, and an instruction, and the model returns one answer per field.

p_null and confidence

Per answer scores that say how likely the value is missing from the input and how sure the model is, used to route uncertain cases to review.

Steps

1. Read the Cloudflare blog post on Clairic, focusing on the field types table and the claims pipeline benchmark.
2. Open the Clairic model page in the Workers AI docs and look at the input fields state and fields and the shape of the answers output.
3. Compare the split of work in the post: Clairic for typed values, Clef for fixed set decisions, an LLM only for open ended text.
4. Read the limitations section on arithmetic, calibration and forecasting.
5. Self check: for an invoice email, which fields would you give Clairic, which decision would you give Clef, and which step still needs an LLM?

Steps

1. Create a Worker with a Workers AI binding named AI, following the Workers AI getting started docs.
2. Call the model with your own text, e.g. env.AI.run("@cf/cloudflare/clairic", { state: email, fields: { order_id: { type: "text", instructions: "Order number" } } }).
3. Add a number field with decimals and unit and a date field, then run it on 10 of your own sample documents.
4. Alternatively call the REST endpoint POST https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai/run/@cf/cloudflare/clairic with a Bearer token and the same JSON body.
5. Record latency, the usage token counts and how many answers had low confidence or high p_null, and check them against the true values.
6. Run the same extraction with an LLM you already use and compare time, cost and accuracy.

Small angles to try

• Add an estimate field with min, max and step and see whether the 80% interval covers the true value
• Send a scanned receipt as an embedded image instead of text
• Leave a field's value out of the input on purpose and see if p_null rises

Key ideas

MultiLangBench

An eval where a model implements SQLite 3.40.1 from its documentation in a chosen language and is graded with tests derived from SQLite's TCL test suite.

Low resource programming language

A language with little public code for models to learn from, such as COBOL or Fortran, where models have historically done much worse.

Reasoning effort

A setting that lets the model spend more internal thinking per task, which raises scores and cost, so results must be compared at the same effort.

Steps

1. Read the OpenAI post on low resource programming languages, noting which reasoning effort each number uses.
2. Open the openai/MultiLangBench README and read how a test is built from initial_database, user_query and validation_query.
3. Note the default 272 test subset and why the full 24k suite is expensive.
4. Self check: why does a vendor score at maximum effort not tell you what your agent will reach on a legacy COBOL codebase at default settings?

Steps

1. Clone the openai/MultiLangBench repo and read its README and requirements file.
2. Build a container, download the SQLite 3.40.1 documentation into it, and remove every SQLite library and binary, using the repo's remove_sqlite script only as a guide.
3. Have your coding agent of choice write an engine in one language you care about, with internet disabled inside the container.
4. Grade only the 272 tests whose names start with 00, 01 or 02, using the provided expected output files to check your runner first.
5. Record pass rate and API cost together, as the benchmark defines the score.

Small angles to try

• Compare one popular language with COBOL or Fortran on the same model
• Run the same model at two reasoning effort settings and compare cost per passed test
• Swap in an open weight coding model running locally

Key ideas

Mixture of experts routing

Each token is sent to a few expert sub networks, so a batch whose tokens pick the same experts needs to load fewer expert weights.

Decode batch

The set of sequences generating their next token together in one step, where memory reads of weights dominate the cost.

Stable argsort by cluster

Reordering prompts by cluster label while keeping the original order inside each cluster, so results can be mapped back afterwards.

Steps

1. Read the Hugging Face post Sort your prompts, starting with the bytes read explanation.
2. Compare the four throughput tables and note how gains shrink for unlabeled data and for gpt-oss-120b.
3. Read the caveats section on online serving, dense models, very large batches and bf16 output drift.
4. Skim the repo README for the expert_friendly_order function.
5. Self check: why would sorting help Qwen3-30B-A3B more than a coarse MoE such as gpt-oss-20b with 32 experts?

Steps

1. Clone the repo with git clone https://github.com/Abdennacer-Badaoui/sort-your-prompts and install pip install -r requirements.txt.
2. Collect a few hundred of your own offline prompts and shuffle them.
3. Generate once in shuffled order and time it, then reorder with order = expert_friendly_order(prompts, n_clusters=16) and generate [prompts[i] for i in order].
4. Restore the original order with outs = [o for _, o in sorted(zip(order, outs))] and compare tokens per second.
5. Optionally use the repo's bench script, e.g. ./run_vllm.sh 0 qwen_b32 --max-num-seqs 32 --save-outputs.
6. Check how many outputs changed between runs before using it in production.

Small angles to try

• Pass known labels such as task or language to skip embedding
• Sweep max num seqs from 16 to 128 and plot the gain
• Try a dense model as a control and expect no gain

Key ideas

Context budget

The fixed number of tokens of repository code you may put in front of the agent for one issue.

Prompt compression

Shortening code text by dropping or rewriting tokens, which saves space but can break the exact lines a patch must match.

SWE-rebench

A benchmark of recent real GitHub issues where an agent must produce a patch that makes the tests pass.

Steps

1. Read the arXiv abstract of 2610.21073 and its results table.
2. Look at how CQ2-CARE ranks files and how it decides how many whole files fit the budget.
3. Read the error analysis on retriever misses and patches that fail to apply after compression.
4. Self check: in your own agent pipeline, what share of failed patches fail because the target file was never in context?

Steps

1. Pick three closed bugs in your own repo and check out the commit before each fix.
2. Set one token budget, for example the same number of tokens for every run.
3. Build context version A from retrieved code chunks and version B from the few most relevant whole files, chosen by file name and issue text.
4. Ask the same model for a patch with each context and apply it.
5. Run your tests and record which patches apply cleanly and which fix the bug.

Small angles to try

• Add a third version using a prompt compressor at the same budget
• Halve the budget and see which approach degrades first
• Log whether the edited file was in context at all

Key ideas

FASCSA order

An order under the 2018 supply chain law that lets an agency bar a source from federal procurement and require its removal from systems.

Open weight model

A model whose trained weights are published so anyone can run it on their own servers, separate from the vendor's app or API.

Third party products developed using DeepSeek implementations

The order's wording for other vendors' tools built on DeepSeek, whose exact reach the notice does not define.

Steps

1. Read the Federal Register public inspection PDF 2026-21447, focusing on the scope and the third party wording.
2. Read the MadRobot analysis for the order date and the national security systems exclusion.
3. List the ways DeepSeek can enter a product: its app, its API, its weights hosted by you or a provider, or a model distilled from it.
4. Self check: which of those paths does the notice's text clearly cover, and which does it leave open?

Steps

1. Search your own repositories for DeepSeek references, for example with git grep -i deepseek.
2. Check your configuration and environment files for model names and API base URLs that point at DeepSeek models.
3. List the AI providers and SaaS tools your product depends on and note which document their underlying models.
4. Write a short inventory of where DeepSeek appears, if anywhere, and which entries are unclear under the order's third party wording.

Small angles to try

• Add a CI check that flags new DeepSeek model ids in configuration
• Check your model gateway logs for which model names were actually served
• Draft the question you would send a vendor about their underlying models