Code415 Radar

Key ideas

.pth files

Python runs import lines found in .pth files inside site-packages at interpreter startup, so a package can execute code without ever being imported.

sitecustomize.py

A module Python imports automatically on startup if it is on the path, which makes it a quiet place to hide a long running daemon.

Build pipeline compromise

The published artifact differs from the tagged source because something changed during packaging or upload, so reading the GitHub code alone does not prove a release is safe.

Steps

1. Read the SafeDep analysis first, focusing on the affected files and the indicators of compromise list.
2. Compare the sdist change SafeDep shows, an extra data line in the maturin section of pyproject.toml, with how your own projects build wheels.
3. Skim the HN thread for how teams pin LiteLLM and which rotation steps they took.
4. Self check: why does deleting the litellm package alone not guarantee the stealer stops running on next Python start?

Steps

1. In each environment that uses LiteLLM, run pip show litellm and note the installed version.
2. Print the site-packages paths with python -c "import site; print(site.getsitepackages())" and check each one for cook.pth, cook.py and sitecustomize.py.
3. Check for the marker files SafeDep lists under /tmp, and search your DNS or proxy logs for the litellm.dev exfiltration domain.
4. Search requirements files, lockfiles and Docker images for 1.107.0 and pin to a version SafeDep lists as clean, for example pip install litellm==1.106.2.
5. If any host had 1.107.0, list every provider key, cloud credential and CI secret it could read and plan rotation.

Small angles to try

• Add hash pinning with pip --require-hashes so a swapped wheel fails to install.
• Write a small CI check that fails the build if any .pth file in site-packages contains an import line you do not expect.
• Compare the PyPI sdist of a package you depend on against its git tag to see how often they differ.

Key ideas

Use after free

Code keeps using memory after it was released, so an attacker who controls what lands there next can steer the program.

Coordinated disclosure

Finders, maintainers and defenders agree on a release time so patches and protections land together, which broke down here once details leaked early.

Version based exposure check

Reading the PHP version from php -v or response headers tells you if you are below the fixed release without sending any attack traffic.

Steps

1. Read the php.net release notes for Oct 21 to get the fixed versions, both CVEs and the interim mitigations.
2. Read the Wiz PHP2Shell post for how the bugs were found and which signals to watch in logs.
3. Read ITmedia on the Prime Minister's post and the GIGAZINE maintenance list to see how far the impact reached in Japan.
4. Self check: why does the PHP team say the php.ini settings reduce exposure but do not make a server safe?

Steps

1. On your server or in a container with your production PHP image, run php -v and compare it with 8.5.12, 8.4.27, 8.3.36 or 8.2.35 for your branch.
2. Clone github.com/openai/php2shell-checker and run node phpcheck.js --local on the same machine; exit 0 means OK, 1 vulnerable, 3 unknown.
3. Run node phpcheck.js against a URL you own and see whether your headers expose the version; an UNKNOWN result does not mean patched.
4. If you cannot upgrade today, read the php.net advisory and set session.upload_progress.enabled = 0 in php.ini or the FPM pool, then restart every worker.
5. Rebuild container images on the fixed base image and confirm php -v again after deploy.

Small angles to try

• Install the two Codex skills from the repo and compare their output with the plain CLI.
• Add the --json flag and wire the checker into CI so an old PHP base image fails the build.
• Grep your logs for the Wiz signals: repeated HTTP 500s, memory limit errors after multipart requests, odd PHPSESSID cookies.

Key ideas

World model

A model that predicts the next state of an environment from the current state and an action, here the next terminal screen after a keypress.

int4 quantization

Storing weights in 4 bits instead of 16 cuts size several times, which is how the model fits in 208 MB.

Speculative decoding

Draft several tokens cheaply, then verify them in one pass, keeping only the ones the model agrees with, to cut latency.

Steps

1. Read the blog post on simedw.com from the data section through the latency section.
2. Note how the episodes were recorded from real programs in Docker and why the vocabulary was cut to 333 tokens.
3. Compare the ShellBench and memory readback results with the prompted baselines.
4. Self check: why does parallel row decoding help large frames much more than single keypresses?

Steps

1. Open the WorldTUI demo at simedw.com/projects/worldtui in Chrome or Edge.
2. Type ls, cat, and clear and judge how believable the screens are.
3. Try pythoon and then ython, as the author suggests, to see that it is not a real shell.
4. Run echo green >> colors then cat colors and check whether the appended line appears.
5. Open man test and vim test.txt and note how long each frame takes on your hardware.

Small angles to try

• Invent a command that does not exist and see whether the model hallucinates plausible output.
• Compare frame latency on a laptop GPU and a desktop GPU.
• Sketch how you would record your own episodes for a different TUI program.

Key ideas

Mixture of experts

Only a few expert subnetworks run for each token, so a 40B model costs roughly what a 2.5B dense model costs per token.

Active parameters

The parameters actually used per token, which drive speed, while total parameters drive memory needs.

SWE-bench Verified

A benchmark of real GitHub issues where a model must produce a patch that passes the repo tests.

Steps

1. Read the Recursive blog post for the claims and the usage advice.
2. Read the Hugging Face model card for context length, sampling settings and benchmark tables.
3. Read TechCrunch for outside context and the caveat that most numbers are company tests.
4. Self check: why can Flint be fast on a laptop yet still need about 23 GB of memory?

Steps

1. Run ollama run flint, which pulls the 23GB flint:40b build with 256K context.
2. Give it a small task from your own repo, such as writing tests for one function, and time the response.
3. Set temperature 1.0, top_p 0.95 and top_k 20 as the model card recommends and repeat the task.
4. Split a bigger task into explicit substeps, as Recursive advises, and compare the result with a one shot prompt.
5. Optional on a GPU server: pip install vllm, then vllm serve "RecursiveSI/flint" and call the OpenAI compatible API on port 8000.

Small angles to try

• Run the same task through your paid coding assistant and compare quality and time.
• Measure tokens per second at short and long context on your own hardware.
• Point an agent tool that accepts an OpenAI compatible endpoint at the local server.

Key ideas

BaxBench

A benchmark of 392 backend tasks where generated apps must pass functional tests and then face real exploit attempts.

Machine unlearning

Editing a trained model's weights to remove a specific learned behavior without retraining from scratch.

Static analysis versus exploit testing

Pattern based scanners read code for known bad shapes, while exploit tests actually attack the running app, which catches logic flaws scanners miss.

Steps

1. Read the Hirundo write-up, including the full caveats list.
2. Open baxbench.com to see how tasks pair an API spec with a framework and exploits.
3. Look at the persona prompts and the p values for each comparison.
4. Self check: why does a 2 versus 1,144 gap between Semgrep and BaxBench matter even if the persona effect turns out smaller?

Steps

1. Clone github.com/logic-star-ai/baxbench and run pipenv install from the repo root.
2. Set the API key env vars the README lists, leaving unused ones as empty strings.
3. Find the scenario id for secret storage in src/scenarios, then run src/main.py in generate, test and evaluate modes with --scenarios set to it and a small --n_samples.
4. Repeat with --safety_prompt generic and compare the share of apps that were exploited.
5. Run a static scanner of your choice over the generated apps in results/ and compare its findings with the exploit results.

Small angles to try

• Edit the task description to add a customer persona and compare two personas with the same model.
• Compare two different models on the same scenario at the same temperature.
• Try --spec_type text instead of the OpenAPI spec and see if security changes.

Key ideas

Regression

A change that breaks behavior that used to work and is still supposed to be supported.

Three arm proof

Running one reproduction against the original, patched and reverted code to prove the patch itself is what fixed the bug.

Threat model file

A short document in the repo that tells agents what assets, trust boundaries and supported configurations matter, so they judge findings against real scope.

Steps

1. Read the Trail of Bits post from Oct 21 end to end, focusing on the section that explains the three arm proof.
2. Note the regression breakdown, 16 functional, 1 performance and 4 build or test, and think about which kind your own test suite would catch.
3. Open the regression-hunt plugin README and list the inputs it asks for: a change list, a campaign directory outside the plugin, and execution and resource limits.
4. Sketch what a THREAT_MODEL.md and a PATCHING_GUIDELINES.md would say for one repo you maintain.
5. Self check: if the patched code passes but the reverted code also passes, what does that tell you about the patch?

Steps

1. In Claude Code, add the Trail of Bits marketplace with /plugin marketplace add trailofbits/skills, or in Codex use codex plugin marketplace add trailofbits/skills then codex plugin list, and check whether patching-bugs and regression-hunt are listed; if they are not yet, read their skill files from the repo directly.
2. Pick a small bug you already fixed in your own repo and check out the commit before the fix.
3. Write one reproduction test and run it three times by hand: on the original code, on the fixed code, and after reverting only the fix, and record which runs fail.
4. Add a short THREAT_MODEL.md and PATCHING_GUIDELINES.md to the repo describing supported configurations and what a regression test must show.
5. Ask the agent to review a recent merged PR of yours for regressions using those files, and compare its findings with what you know broke or did not break.

Small angles to try

• Run the same PR review with and without the two guideline files and count how many findings you would reject.
• Inside the regression-hunt plugin directory, run uv sync --frozen and uv run --frozen pytest to see how its deterministic helpers are tested.
• Apply the three arm check to a fix an agent wrote last week.

Key ideas

Calibration

A score is calibrated when cases labeled 70% turn out right about 70% of the time.

Base rate

The historical share of correct outputs, used as the starting confidence before you have finer signals.

Self consistency signal

Asking the same question several times and using agreement between answers as evidence, instead of the stated number.

Steps

1. Read Amrut Joshi's post on rumple.github.io, especially the weather forecaster calibration table.
2. List the signals your own LLM pipeline already produces, such as retrieval hits, tool errors or answer agreement.
3. Skim the HN thread for counterpoints about token log probabilities and when they help.
4. Self check: a forecaster who always says 30% can be calibrated, so why is that forecaster still useless?

Steps

1. Export or invent a small log of past LLM outputs from your own system with a few signals per row and whether each output was judged correct.
2. Compute the overall base rate of correct outputs.
3. Group rows by signal combination, for example document found and answers agree, and compute each group's accuracy as its confidence.
4. Bucket the rule's confidence values and compare each bucket's predicted rate with the actual rate to see whether it is calibrated.
5. Hold out the newest rows, score them with the rule, and check whether calibration holds on data the rule never saw.

Small angles to try

• Ask your own LLM the same question ten times for a 0 to 1 confidence and plot how much the number moves.
• Add "are you sure" as a follow up and measure how often the stated score changes with no new information.
• Compare a rule based score against the LLM's stated score on the same logged cases.

Key ideas

Durable agent

An agent whose state is saved after every step so it can resume after a crash or restart.

Append only event log

A record where steps are only added, never edited, so the agent's state can be rebuilt by replaying it.

Worker lease

A time limited claim on a job; if the worker dies the lease expires and another worker picks the job up.

Steps

1. Read the Omnara README on GitHub for the feature list and self host requirements.
2. Read the Show HN post for the event log and lease design.
3. Compare that design with how your current agent setup stores its transcript, for example a local JSONL file.
4. Self check: why can one worker serve thousands of agents if most of them are waiting?

Steps

1. Run git clone https://github.com/omnara-ai/omnara.git and cd omnara.
2. Start the published images with docker compose -f compose.yaml --profile app up -d.
3. Open http://localhost:8000 and find the local sign in email with docker compose logs api.
4. Configure a model endpoint with your own key or point it at your local Ollama, then create a simple agent and give it a short task.
5. Restart the containers during a task and check whether the agent resumes where it left off.
6. Connect to the Postgres container and look at the stored agent history; keep this a local test, since the README warns the local defaults are insecure for deployment.

Small angles to try

• Build from source instead with docker compose --profile app up -d --build.
• Create two roles and check what a view only API key can and cannot do.
• Add an HTTP MCP server you run locally as a tool.

Key ideas

Omni modal GUI agent

An agent that operates an app or device using several inputs at once, such as screenshots, audio and video.

Sequential Distraction

The paper's name for errors that pile up when irrelevant context is carried into each new step.

Frozen backbone with adapter

Keeping the large model's weights fixed and training only a small add on module.

Steps

1. Read the abstract on the Hugging Face paper page.
2. Open the arXiv PDF and look at the method overview figure to see where the latent enters the decoder.
3. Read the FocusGUI README to see the four backbones: Qwen2.5-Omni-7B, MiniCPM-o-4.5, VITA-1.5 and Baichuan-Omni-1.5.
4. Self check: why does training on action labels alone avoid the need for labels saying which modality matters?

Steps

1. Clone the repo with git clone https://github.com/fyzhang1/FocusGUI.git and cd FocusGUI.
2. Create an environment with conda create -n focusgui python=3.10 -y and conda activate focusgui.
3. Install with pip install -e ".[qwen]", then install a CUDA build of PyTorch that matches your machine.
4. Read DATA.md and .env.example to list exactly which weights and data you would need to download.
5. Read the adapter code and trace how per step evidence is scored, compressed and passed to the frozen decoder.

Small angles to try

• If you have a GPU, follow the README to prepare data chunks and train only on the Qwen path.
• Apply the idea to your own text agent: rank past tool outputs by relevance each step and pass only the top few.
• Compare the fusion ablation script's settings to see which part of the adapter matters most.

Key ideas

Hidden instructions

Text in a web page that readers never see but that a model reading the page source or index will ingest.

Indirect prompt injection

Steering a model through content it retrieves rather than through the user's own prompt.

User agent based serving

Returning different page content depending on which crawler or browser asks for it.

Steps

1. Read Dan Saltman's post on the Redact blog, including the table of example notes and the crawler test.
2. Read the GIGAZINE summary for the visible deletion rules the hidden text sat next to.
3. Skim the HN thread for how engineers propose to filter hidden text in retrieval pipelines.
4. Self check: if a page hides text from users but not from search crawlers, which AI answers can still be affected?

Steps

1. Write a local HTML file you own with visible help text plus a hidden block containing a harmless marker sentence.
2. Run the file through your own ingestion step, the same parser or loader your RAG system uses, and check whether the marker survives.
3. Add a filter that drops elements hidden from users and rerun to confirm the marker is gone while visible text stays.
4. If your pipeline calls a model, feed it the unfiltered chunk with a benign instruction like "answer in French" and see whether it obeys, then repeat with the filtered chunk.
5. Add the marker page to your test suite so future parser changes are checked.

Small angles to try

• Try several hiding methods in your own test page and see which ones your parser misses.
• Wrap retrieved text in clear delimiters with a system note that it is untrusted data, and compare behavior.
• Log any retrieved chunk containing imperative phrases aimed at AI for manual review.

Key ideas

Long horizon agent benchmark

A test where an agent must make thousands of linked decisions over a long simulated period, so early mistakes compound.

Asset multiplier

The main score here, year end total assets divided by the starting 100,000 yuan.

Deterministic environment

Customer demand and supplier pricing follow fixed seeded models, so score differences come from the agent rather than random noise.

Steps

1. Read the ITmedia NEWS article first for the setup and the headline results.
2. Open the arXiv abstract for E-Commerce Bench and note the seven evaluation dimensions and why no model won all of them.
3. Skim the QwenLM E-CommerceBench README to see the tools, the cash flow rules and the log format.
4. Compare GPT-5.6 Sol, Fable5 and Claude Opus 4.7 on profit, tool calls and scam avoidance, and write one sentence on the trade off between caution and growth.
5. Self check: why can an agent with profitable stores still go bankrupt in this simulation?

Steps

1. Clone https://github.com/QwenLM/E-CommerceBench and run pip install -r requirements.txt in a fresh virtual environment.
2. Export the key for your provider, for example ANTHROPIC_API_KEY or GEMINI_API_KEY, and OPENAI_API_KEY for the supplier dialogue.
3. Run the README smoke test: python run.py --model gemini-3.5-flash --max-days 10 --max-turns 50.
4. Open the new folder under log/ and read the per day balances and one negotiation transcript.
5. Count how many supplier offers the agent accepted and whether any came from a scam supplier.

Small angles to try

• Swap --model to another key from models_config.json and compare 10 day balances.
• Lower --initial-balance to see how fast cash flow pressure appears.
• Run --runs 3 to see how much results vary across episodes.

Key ideas

World model

A model that predicts what the next frames of a scene will look like given the current view and an action.

Egocentric video

Footage filmed from the point of view of the person or robot doing the task, such as a head mounted camera.

Embodiment

The specific body doing the work, such as a human hand or a particular robot arm, each with different joints.

Steps

1. Read the Hugging Face paper page abstract first for the kinematic interface idea.
2. Look at the project page videos to see camera pose and skeleton conditioning side by side.
3. Read the GitHub README sections on clip contents and released checkpoints.
4. Note the licenses: code Apache-2.0, weights CC BY-NC 4.0, Xp-Teleop data CC BY 4.0.
5. Self check: what two signals let one model handle both walking and hand manipulation across bodies?

Steps

1. Clone the repo with git clone https://github.com/Ropedia/ropedia-w0.git and create the env with conda create -n ropedia-w0 python=3.12 -y.
2. Install PyTorch as the README shows, then pip install -r requirements.txt.
3. Download the base model with hf download Wan-AI/Wan2.2-TI2V-5B --local-dir models/Wan2.2-TI2V-5B and fetch w0-5b-midtrain.safetensors from the ropedia-ai/Ropedia-W0 model page.
4. Run scripts/run_single_case_5b.sh on one of the 18 clips in examples/, using CFG=3.0 FPS=30 for a robot clip.
5. Compare the generated video with the input skeleton.mp4 to see how closely the hand motion is followed.

Small angles to try

• Run the same clip with the pretrain and the midtrain checkpoints and compare robot realism.
• Try a human clip with the default CFG instead of the robot settings.
• Without a large GPU, use the hosted demo Space instead.

Key ideas

Background agent

An agent that keeps working between your messages, checking sources on a schedule and contacting you when needed.

Polling interval

How often the agent checks connected accounts, which trades freshness against usage cost.

Human approval gate

A rule that the agent must ask before actions with side effects, such as sending email or paying.

Steps

1. Read the Claude Duet product page first, especially the usage limits and known limitations sections.
2. Read the Dataconomy article for the polling explanation and the link to Claude Code limits.
3. List which of your daily tasks would need approval gates and which could run unattended.
4. Estimate how many checks per day a few hours interval means and what each might cost in model turns.
5. Self check: why can Duet miss something that arrives between check ins, and what design choice causes it?

Steps

1. Join the waitlist from the product page if you are eligible.
2. Write down three recurring inbox or calendar chores you would hand to a background agent.
3. For each chore, mark which steps read data and which send, pay or change something, and so need approval.
4. Pick a polling interval for each chore and estimate checks per day against your plan's usage.
5. Note which chore would break if a message arrived between checks, and how you would cover that gap.

Small angles to try

• Compare a few hours polling with event driven webhooks for the same chore.
• Sketch what the morning brief should contain so it fits on one phone screen.
• Decide which connectors you would never grant to an always on agent.

Key ideas

Dense embedding

A fixed length vector for a query or document so that similar meanings land close together.

Cross encoder reranker

A model that reads the query and each candidate together to rescore a short list more accurately than vectors alone.

nDCG@10

A ranking score that rewards putting the most relevant results near the top of the first ten.

Steps

1. Read the NVIDIA post on the Hugging Face blog first for the model lineup and benchmarks.
2. Open the Nemotron-3.5-Embed-1B-BF16 model card and note the query and passage prefixes and the 2048 dimension output.
3. Trace the pipeline: Parse turns PDFs into text, Embed retrieves candidates, Rerank reorders them before the agent reads them.
4. Note why fewer but better ranked chunks lower agent token cost.
5. Self check: why does a reranker reduce tokens rather than add cost overall?

Steps

1. Create a Python 3.12 environment and run uv pip install "sentence-transformers[image]>5.4" "transformers==5.20.0" as the model card shows.
2. Load nvidia/Nemotron-3.5-Embed-1B-BF16 with SentenceTransformer on cuda:0 in bfloat16.
3. Embed five questions with encode_query and twenty of your own document paragraphs with encode_document.
4. Score them with model.similarity and check whether the right paragraph is in the top three for each question.
5. Add one screenshot of a table from your own documents as an image input and see if a matching question finds it.

Small angles to try

• Truncate embeddings to 512 dimensions, renormalize and compare recall.
• Serve the model with vLLM as the model card describes and compare latency.
• Run your own PDFs through Nemotron-3.5-Parse before embedding.

Key ideas

Agent harness

Everything around the model that shapes its behavior, such as prompts, tools, memory and subagents.

Propose-Probe-Compose

An editor proposes harness changes, each is run on the same task, and the contrasts between runs are combined into a better harness.

Harnessing level recursion

Saving what was learned on each task into a reusable harness that helps on later tasks.

Steps

1. Read the arXiv abstract first for the two levels of recursion.
2. Read the GitHub README sections on the eight editable components and the security warnings.
3. Skim the Oct 19 HN thread for practitioner reactions.
4. Compare this with hand tuning your own coding agent's system prompt and tools.
5. Self check: how does Harness2 improve a harness without any ground truth grader?

Steps

1. Clone https://github.com/google-research/harness2 inside a container or VM with no secrets on it.
2. Run uv venv --python 3.12 .venv and source .venv/bin/activate.
3. Install with uv pip install -e '.[render]'.
4. Run harness2-demo --output runs/demo --k 2 to see both recursion modes with deterministic editors.
5. Open the files under runs/demo and trace one proposed edit, its probe runs and the composed result.

Small angles to try

• Compare parallel and sequential modes in the demo output.
• Raise --k and see how the number of candidate harnesses grows.
• Before any model backed run, block outbound network for the executor and note what breaks.

Key ideas

Fan out agents

A large job is split into many small, independent tasks, each handled by its own agent with a narrow context.

Merge or reduce step

The separate agent reports are combined into one structure, here an atlas linking genes, cell effects, papers and possible mechanisms.

Perturb-seq atlas

A dataset recording how switching off individual genes changes gene activity in single cells.

Steps

1. Read the OpenAI post on scaling AI research for complex biological data, focusing on how the work was split and compared.
2. Open the jinlabneurogenomics/distributed_agent README and note how a query is dispatched to per perturbation agents.
3. Skim the bioRxiv preprint methods section on the single agent baseline and the blinded expert comparison.
4. List which parts of the pattern carry over to a non biology dataset you own, such as one agent per customer account or per log file.
5. Self check: why does holding model and compute constant matter when claiming that fan out beats a single agent?

Steps

1. Clone the jinlabneurogenomics/distributed_agent repository and run pixi install -e minimal from the repo root.
2. Run git lfs install and git lfs pull to fetch the bundled public data, then run pixi run prepare-manuscript to rebuild the saved analyses without any model calls.
3. Copy .env_example to .env, authenticate with codex login, and ask one question with pixi run -e minimal distributed_agents query "<question>" --out-dir runs/starter.
4. Open the saved reports under runs/starter and check how many distinct references each per gene report cites.
5. Write a short note on where the merge step lost or kept detail compared with the individual reports.

Small angles to try

• Point the same fan out idea at your own data, one agent per file or per customer, and compare against a single agent with the same budget.
• Rerun the query against the shi_holdout corpus with --corpus-release shi_holdout and compare the answers.
• Count tokens and wall time for fan out versus one agent to see what the extra coverage costs.

Key ideas

Write quorum

A write counts as committed only after enough replicas confirm it, so the slowest needed replica sets the latency.

Lock scope

Locking a whole repository serializes every push, while locking a single reference lets pushes to other branches run in parallel.

Virtual replica

A quorum member or read server backed by object storage rather than a full copy on local disk, which separates storage from compute.

Steps

1. Read the earlier GitHub post, Building Git infrastructure for agent scale development, for the two design principles.
2. Read The bridge to agent scale Git infrastructure and sketch the write path before and after the per reference change.
3. Note why a 25ms cross region hop matters more when every write holds a repository wide lock.
4. Compare the virtual write replica and the virtual read replicas and what each one is allowed to do.
5. Self check: why does the throughput ceiling rise roughly with the number of references written at the same time?

Steps

1. Create a local bare repository with git init --bare and make one clone with an initial commit pushed to it.
2. Make several more local clones to stand in for agents, each on its own branch.
3. Write a small shell loop that has every clone commit and push to its own branch at the same time, and time the run.
4. Repeat with every clone pushing to the same branch and count rejected pushes and retries.
5. Compare the two runs and relate the gap to what GitHub describes about per reference quorum.

Small angles to try

• Add an artificial delay to a server side hook in your local bare repository to imitate cross region latency.
• Scale the number of simulated agents from 2 to 32 and plot pushes per second.
• Review your own agent tooling to confirm it gives each agent its own branch instead of sharing one.

Key ideas

Query rewriting

The assistant turns a user question into its own sub queries before searching, so results differ from a plain web search.

Reranking

A model scores retrieved passages for relevance to the query, which rewards pages that look like an answer, true or not.

Citation laundering

Each time a respected source repeats a fake number, the number looks more credible to the next reader or model.

Steps

1. Read the original Peec AI post, Publish stats, get cited, for the method and traffic data.
2. Read the GIGAZINE summary for the list of places the fake figures ended up.
3. Trace the retrieval path the post describes: rewrite, gather candidates, rerank, open a few pages, answer.
4. List where your own app shows numbers that come from retrieved web pages.
5. Self check: which step in that path could check whether a statistic has a primary source?

Steps

1. Ask a chat assistant with web search three niche statistics questions in a field you know and record every cited source.
2. For each cited number, follow the link and look for a named primary source such as a survey, dataset or filing.
3. Mark each answer as primary, secondary or unsourced and count how many fall in each group.
4. If you run a website, search your own server logs for the ChatGPT-User user agent to see which of your pages get fetched.
5. Write one rule for your product, such as hiding statistics that lack a primary source link.

Small angles to try

• Repeat the same questions a week later to see whether cited sources change.
• Build a tiny local retrieval demo on your own documents, include one page of obviously fake figures, and see whether your reranker prefers it.
• Compare answers across two different assistants for the same statistics question.

Key ideas

Word error rate

The share of words a transcript gets wrong through substitutions, deletions and insertions, where lower is better.

Keyword biasing

Passing a list of names or jargon in the prompt so the model favors those spellings.

RTFx

How many seconds of audio a model processes per second of compute, so higher means faster.

Steps

1. Read the IBM Granite post on the Hugging Face blog for the three models and their roles.
2. Open the granite-speech-5.0-2b model card and read the prompt format and audio requirements.
3. Look at the Open ASR Leaderboard to see how average English WER is measured.
4. Decide which model fits your use: 2B for accuracy and translation, 470M for speed.
5. Self check: why could a model with lower average WER still do worse on your own meeting recordings?

Steps

1. Install the packages with pip install transformers torchaudio soundfile.
2. Record or pick two short clips of your own speech, mono at 16 kHz and under about two minutes each.
3. Follow the model card example for ibm-granite/granite-speech-5.0-2b: load the processor and model, put the <|audio|> token in the user message, and generate a transcript.
4. Write a reference transcript by hand and compute word error rate with any WER script.
5. Rerun with the keyword prompt format from the model card, listing names and jargon from your clip, and compare the error rate.

Small angles to try

• Serve it with vllm serve ibm-granite/granite-speech-5.0-2b --max-model-len 2048 and send audio through the OpenAI compatible API.
• Compare speed and accuracy against ibm-granite/granite-speech-5.0-470m-turboctc on the same clips.
• Try a Japanese or Spanish clip and the translation prompt.

Key ideas

Mixture of experts

Only some expert sub networks run for each token, so compute per token is lower than the total parameter count suggests.

Subagents with isolated contexts

The main model hands subtasks to fresh model instances so their working notes do not crowd its own context.

Self verification

The model pauses to check evidence and progress, then changes strategy if a path looks unproductive.

Steps

1. Read the arXiv abstract for the three claimed improvements: orchestration, verification and open exploration.
2. Open the Hugging Face model card and read how spawn_subagent and the verification flag are enabled.
3. Compare the benchmark table against Agents-A1 and note where scores went down.
4. Check the serving section for context length and parser flags.
5. Self check: what does training subagent use into the weights change compared with orchestration written in the harness?

Steps

1. Serve the model with vllm serve InternScience/Sage-A1.5 --port 8000 --max-model-len 262144 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder as shown on the model card.
2. Point an OpenAI compatible client at http://localhost:8000/v1 and give it a small multi step task on your own repository or files.
3. Rerun the same task with verification on by passing chat_template_kwargs with verification set to true and a non empty system message.
4. Compare the reasoning_content field and the final answer between the two runs.
5. Enable the spawn_subagent tool as described on the card and note how the task is split.

Small angles to try

• Lower --max-model-len if memory is tight and note how long tasks degrade.
• Serve it with SGLang using the command on the model card and compare throughput.
• Run the same task on a smaller model you already use and compare tool call accuracy.