Code415 Radar

Key ideas

Formal verification

A machine checked mathematical proof that a program satisfies a stated property for every possible input, not just the ones a test tried.

Fuzz target

A small entry function that feeds arbitrary bytes into a library so a fuzzer can hunt for crashes, here reused as the specification to prove.

Trusted base

The parts a proof assumes are correct, such as the CPU and OS model and the proof checker, which is where any remaining risk lives.

Steps

1. Read the Anthropic research post first, focusing on the sections on what was proved and on limitations.
2. Open the oss-verify-proofs README and skim the repository structure table to see how statements, proofs and binaries are laid out.
3. Open one entry in index/PROOFS.md, for example coturn FuzzStunClient, and read its Statements.v to see what exactly is claimed.
4. Read the README limitations list on sanitizers, x86 builds and non terminating runs.
5. Self check: name two kinds of bug that a proof here would still not rule out.

Steps

1. Clone the proofs repo with git clone https://github.com/theorem-labs/oss-verify-proofs and enter the folder.
2. Fetch one project's binaries with ./oft fetch coturn, which checks each binary against its SHA-256.
3. Install the checker's Python dependencies into a side folder with python3 -m pip install --require-hashes --target "$HOME/oft-pydeps" -r tools/oft/requirements.txt.
4. Run the pinned checker image from the README with docker run --rm --network none, mounting the repo and the deps folder, and add --cpuset-cpus to cap cores, then run ./oft check coturn/FuzzStunClient --parts statements,proofs inside it.
5. Watch the final step where the checker reads the Print Assumptions output and rejects any axiom not on the allowed list.
6. Note wall time and peak memory, and compare against the README figure of up to 20 GiB on 16 cores.

Small angles to try

• Pick a library you depend on from index/PROOFS.md, such as zstd or cjson, and check its proofs instead.
• Read the oss-verify enrollment steps and dry run tools/validate.py on a template project you own.
• Browse the counterexamples folder to see inputs that break a statement.

Key ideas

API version header

The anthropic-version request header pins which behavior the API uses, so old code keeps working until it opts into the new date.

Sampling parameters

Settings like temperature, top_p and top_k that change how random the next token choice is; the new version rejects them outright.

Server side compaction

The API summarizes older context once input tokens pass a trigger, replacing the SDK option that did this on the client.

Steps

1. Read the 2026-10-21 migration guide in the Claude docs first, top to bottom.
2. List which of the breaking changes touch your code: sampling params, document blocks in assistant turns, citations on user text, Admin API field renames.
3. Read the compaction before and after snippet to see how compactionControl maps to context_management.
4. Skim the Claude Code v2.1.307 release notes for the new migration command.
5. Self check: why would a proxy break after an SDK upgrade even though the API accepts both auth headers?

Steps

1. In a fresh virtual environment, install anthropic==2.0.0 with pip.
2. Send a short Messages request with a current model id from the docs and no sampling params, and confirm it succeeds.
3. Add temperature 0 to the same request and confirm it now fails with a 400.
4. Point the client at a tiny local logging proxy you run yourself and confirm the key arrives as an Authorization Bearer header, not X-Api-Key.
5. On a copy of an old project, run /claude-api api-version inside Claude Code and review the diff it proposes.

Small angles to try

• Call the API directly with curl twice, once with anthropic-version 2023-06-01 and once with 2026-10-21, sending temperature both times.
• Port a TypeScript tool runner from compactionControl to context_management with the 50,000 token minimum trigger.
• Grep your codebase for top_p, top_k and X-Api-Key before upgrading.

Key ideas

Output convergence

Different models trained on overlapping web data and similar tuning tend to land on the same answer to open questions.

Mode collapse

A model repeatedly picks one high probability answer instead of covering the range of plausible ones.

Sampling runs

Asking the same prompt several times shows whether an answer is a stable preference or a random draw.

Steps

1. Read the ModelCompare blog post first, including the exact prompt and the list of models.
2. Note how answers were grouped by hand and how refusals were counted.
3. Compare the text results with the image results and note where they diverge.
4. Skim the Hacker News thread for critiques of the method, such as sample size and prompt wording.
5. Self check: what would you change in the setup to tell model convergence apart from prompt wording effects?

Steps

1. Copy the exact prompt from the blog post into a small script.
2. Call each model three times with no system prompt and save every answer to a JSON file.
3. Write a one sentence label for each answer and group identical themes by hand.
4. Count how many answers share the most common theme and how many models gave it every time.
5. Compare your counts with the post's 29 of 35 figure.

Small angles to try

• Swap in a question from your own product domain to see if your candidate models converge there too.
• Add a system prompt asking for an unusual view and measure whether diversity improves.
• Run the same prompt against one model at several temperatures where supported.

Key ideas

KV cache

Stored attention keys and values for tokens already processed, so the model does not recompute them each turn.

Prefill

The step that processes the whole prompt before generation; editing early context forces it to rerun on everything after the edit.

Repositioning

Shifting the position encodings of retained keys so the context stays contiguous after some entries are dropped.

Steps

1. Read the Hugging Face paper page abstract first.
2. Open the AKV README and read how drop_message and reposition are added to a Chat Completions request.
3. Read the three ways a policy can be set: trigger, selector and reposition components, or a model calling a drop tool.
4. Compare this with how your own agent harness compacts context today.
5. Self check: why does summarizing history cost more prefill than dropping cache entries?

Steps

1. Clone the repo with git clone https://github.com/thu-nics/AKV.git and create an environment with uv venv --python 3.12.
2. Install it in editable mode with uv pip install -e . so the policy classes can be imported.
3. Read examples/demo.py to see how a multi turn agent sends its history and cache instructions.
4. Build a toy message list with a few fake tool responses and run it through a CachePolicy to print the resulting drop_message and reposition fields, without any server.
5. If you have access to a GPU box you own, launch the server per the README with a supported Qwen3 or GPT-OSS model and run the demo.

Small angles to try

• Change the trigger from turn based to token based and compare which messages get dropped.
• Log how many prompt cache hits your current agent loses each time it summarizes history.
• Try the model driven mode where the model calls a drop tool itself.

Key ideas

Personal agent

An assistant that carries out multi step tasks across a user's connected apps rather than only answering questions.

Human approval gate

A checkpoint where the agent pauses and asks the user before an irreversible action such as a payment or an outgoing email.

Guard agent

A second model, Sentinel in Muse's case, that reviews the first agent's external actions before they run.

Steps

1. Read the ITmedia NEWS article first, using a translator if needed.
2. List the current and planned integrations and note which ones handle money or personal data.
3. Read the Business Insider Japan piece for background on the September US launch and an early privacy slip.
4. Sketch what Mercari or Money Forward would need to expose for an agent to act safely.
5. Self check: which Muse safety measures protect the user if the main agent is tricked by content in an email?

Steps

1. Sign up for Muse with a test account and connect only a test Google Calendar.
2. Ask it to schedule a meeting and draft an email invite, and note exactly when it asks for approval.
3. Decline the approval and check that nothing was sent or created.
4. Open the action history and compare planned versus completed actions.
5. Revoke the calendar connection and confirm the agent can no longer read it.

Small angles to try

• Put a hidden instruction in a test calendar event description and see whether the agent follows it.
• Write a one page design for an approval API your own service could offer to agents.
• Compare Muse's permission screens with those of another agent you already use.

Key ideas

Agent Skill

A folder with a SKILL.md file of YAML metadata plus Markdown instructions, optionally bundling scripts, references and assets.

Progressive disclosure

Agents load only each skill's name and description at startup and read the full instructions only when a task matches.

Agentic AI Foundation

A Linux Foundation project cofounded by Anthropic, Block and OpenAI that gives MCP and now Agent Skills a neutral home.

Steps

1. Read the Anthropic announcement first to see what moved to the foundation and who maintains it.
2. Read the specification page on agentskills.io, focusing on the frontmatter table and the name field rules.
3. Note the size guidance: about 100 tokens of metadata, a body under 5000 tokens and a SKILL.md under 500 lines.
4. Self check: why must the description say both what a skill does and when to use it?

Steps

1. Create a folder named release-notes and inside it a SKILL.md whose frontmatter sets name: release-notes and a description saying what it does and when to use it.
2. Write a short Markdown body with step by step instructions, one input and output example and one edge case.
3. Install the skills-ref reference library from the agentskills GitHub repo by following its README.
4. Run skills-ref validate ./release-notes and fix any reported errors.
5. Break it on purpose: rename the name field to Release--Notes and confirm validation fails for uppercase and consecutive hyphens.
6. If you use a compatible agent, copy the folder into its skills directory and ask for release notes to see whether it activates.

Small angles to try

• Add the optional license, compatibility and metadata fields and revalidate.
• Move long details into references/REFERENCE.md and compare how much context the agent loads.
• Load the same skill in two different agents and compare behavior.

Key ideas

Resource command pattern

Commands mirror the REST API, so openai responses create maps to creating a Responses API call.

Admin key

A separate organization level key used only for admin endpoints such as usage reports, kept apart from normal API keys.

GJSON

A compact path syntax for picking fields out of JSON, used here by the transform flags.

Steps

1. Read the openai-cli README on GitHub first, especially installation, environment variables and output formats.
2. Compare one command with the matching endpoint in OpenAI's API reference to see the mapping.
3. Read the @file section to learn how files, plain text and base64 are passed.
4. Self check: which environment variable does an admin usage command need, and why is it separate from OPENAI_API_KEY?

Steps

1. Install with brew install openai/tools/openai, or with go install 'github.com/openai/openai-cli/cmd/openai@latest' and add the Go bin directory to PATH.
2. Export OPENAI_API_KEY with your own key.
3. Run the README example openai responses create --input "Say this is a test" --model gpt-5.5.
4. Repeat it with --format yaml, then with --format json plus a --transform GJSON path that extracts only the output text.
5. Pass a local text file of your own with the @file syntax and check the request is built as expected.
6. Rerun once with --debug to inspect the HTTP exchange, then delete that output since it holds request bodies.

Small angles to try

• Replace an existing curl plus jq script with the CLI and compare length.
• With an admin key, pull a day of completions usage via the openai admin:organization:usage completions command from the README.
• Use --format-error to get machine readable errors in a shell script.

Key ideas

Grid carbon intensity

Grams of CO2e per kWh of electricity, about 8 for Sweden versus about 333 for Texas in Berget's sources.

Embodied emissions

Emissions from manufacturing hardware, here about 1,000 kg CO2 per data centre GPU spread over its requests.

SCI for AI

A Green Software Foundation spec for computing a software carbon intensity score for AI workloads.

Steps

1. Read the Berget AI press release first, including the correction note at the end.
2. Recompute the ratio from the table: 3.2 divided by 0.38 is about 8.4, close to the 8.5x the release states from unrounded data.
3. Open the berget-ai co2-calculator repo README and find the inputs it takes, such as hardware, grid region and token counts.
4. Self check: why does Sweden's footprint become dominated by hardware manufacturing rather than electricity?

Steps

1. Make an empty Node project and install the calculator package named in the berget-ai co2-calculator README; the README says npm install @berget/co2-emissions-calculator while the release names @bergetai/co2-calculator, so check which one npm actually has.
2. Copy the README's calculateInference example into a script and run it once as is.
3. Change only the deploymentGrid between the available grid regions and print grams of CO2e per request for each.
4. Change the input and output token counts toward the release example of 5,000 in and 387 out and note how the totals scale.
5. Put the results in a small table and compare the region ratio with the release's figure.

Small angles to try

• Toggle includeTraining and see how much training amortization adds.
• Vary concurrency to see how batching shifts the hardware share.
• Estimate a month of your own app's traffic in two regions.

Key ideas

Active parameters

In a MoE only a few experts run per token, so 4.3B of 28.8B weights do the work each step.

Hybrid attention

The model mixes 18 linear attention layers with 7 full attention layers to keep long context cheaper.

Draft model

A small model that proposes tokens the big model verifies in one pass, speeding decoding without changing output.

Steps

1. Read the Hugging Face model card first, especially the architecture notes and benchmark table.
2. Compare the MoE rows with the dense rows and note which tasks favor which.
3. Look at the quickstart to see how the DFlash2 draft is wired in with speculative decoding flags.
4. Self check: why can a 30B MoE run nearly as fast as a 4B dense model yet still need memory for all 30B weights?

Steps

1. Request access on the model page; approval is automatic.
2. Pick the variant that fits your machine: KRAFTON/Raon-1-mini-preview-gguf for llama.cpp with the Q4_K_M quant, or KRAFTON/Raon-1-mini-preview-mlx on Apple silicon.
3. Load it with the tool's standard run command as shown in the model card's local apps section and ask a short reasoning question with thinking on.
4. Ask the same question in Korean and in English and compare answer quality.
5. Give it a simple tool calling task through an agent harness you already use, such as OpenCode, and check the tool call format.
6. Write down tokens per second and whether answers look right.

Small angles to try

• On a GPU server, try the vLLM quickstart with the dflash2 draft and compare speed with and without it.
• Run the same prompts on Qwen3.6-35B-A3B for a side by side.
• Feed a long document to probe the long context claim.

Key ideas

Speculative decoding

A cheap drafter guesses several tokens and the target model verifies them in one pass, keeping only correct ones.

Cost measurement

eagle-harness profiles your GPU and tensor parallel size once, about 10 to 20 minutes on MI355X per AMD, to tune speculation depth.

Concurrency tradeoff

At high batch sizes the GPU is already busy, so spare compute for drafting shrinks and speedups fall.

Steps

1. Read the AMD technical article first for the MI355X numbers and the benchmark setup.
2. Then read the EAGLE-4 GitHub README speedup table for B300 across SGLang, vLLM and TokenSpeed.
3. Read the license section of the repo and note the commercial terms.
4. Self check: why is the speedup at 64 concurrent requests far below the one at a single request?

Steps

1. Clone the EAGLE-4 repo and open the README speedup table and license file.
2. Put the AMD single GPU numbers next to the repo's B300 numbers in a small table, labeling each source.
3. Read the Docker serve command in the README and identify each speculative flag: num steps 12, topk 4 and 16 draft tokens.
4. Sketch your own workload's typical concurrency and estimate which column of the table applies.
5. Decide whether your use falls under the license's commercial terms.

Small angles to try

• If you have a suitable GPU, run the README Docker command and send the sample curl request.
• Compare EAGLE-4 against the built in MTP speedup numbers in the AMD article.
• Read the EAGLE-4 paper PDF in the repo for how the drafter is trained.

Key ideas

Decision model

A model that picks one answer from a fixed set and reports how likely each option is, instead of writing open text.

Predicate, choice and score questions

Predicate returns a probability that a condition holds, choice picks one of your listed values, and score rates input against ordered rubric levels.

Calibration

How well the reported probabilities match how often the model is actually right, which decides whether a confidence threshold is trustworthy.

Steps

1. Read the OpenAI Decisions API developer guide at developers.openai.com first, focusing on the request fields and the three question types.
2. Read the OpenAI announcement page for pricing, regions and the GPT-Live voice trigger example.
3. Skim the TechCrunch article for how it ranks against TypeSafe's Jev and other decision models.
4. Compare with pplx-decider from the Oct 19 radar: hosted versus open weight, and what each returns.
5. Self check: for a support ticket router, which question type would you use, and what would you do when confidence is low?

Steps

1. Collect 30 of your own short texts with known labels, for example support messages or commit messages tagged bug, feature or chore.
2. Send one POST https://api.openai.com/v1/decisions request with "model": "gpt-6-luna", the text as input, and one choice question listing your labels as choices.
3. Loop over all 30 texts, logging the chosen label, its confidence and the wall clock latency of each call.
4. Run the same 30 texts through the Responses API on gpt-6-luna with a plain classification prompt and log latency and tokens for comparison.
5. Compute accuracy for both, then sort the Decisions results by confidence and pick a threshold below which you would fall back to a full LLM call.

Small angles to try

• Use a predicate question as a guardrail, such as whether a message contains a request for personal data.
• Use a score question with 3 to 5 rubric levels to grade answer quality.
• Put several questions in one request against the same input and see how latency changes.

Key ideas

Tree-sitter AST parsing

A fast local parser turns source files into syntax trees, so functions, classes and calls can be found without any model.

Extracted versus inferred edges

Extracted edges come straight from the code structure, while inferred edges are guesses from a semantic pass and deserve less trust.

MCP server

A small process that exposes tools such as graph queries to an AI assistant over the Model Context Protocol.

Steps

1. Read the Graphify README on GitHub first, especially the install section and the Common commands block.
2. Read the 0.9.94 release notes on the releases page to see what kinds of edges and languages it handles.
3. Open a sample graph.json and GRAPH_REPORT.md after a build to see how nodes and edges are named.
4. Self check: why would a coding agent prefer a call graph over embedding search when asked what breaks if a function changes?

Steps

1. Install the CLI with uv tool install graphifyy, noting the double y in the package name, then run graphify install to register the skill with your assistant.
2. In your assistant, inside one of your own repos, run /graphify . --no-viz to build the graph and report only.
3. Open graphify-out/GRAPH_REPORT.md and check whether the main modules and their links match your mental model of the code.
4. Ask a structural question with /graphify path "ClassA" "ClassB" using two real names from your repo, and /graphify explain on one central class.
5. Edit one file and run /graphify . --update to confirm only changed files are re-extracted.

Small angles to try

• Run it on a repo with SQL stored procedures and look for the new read and write edges.
• Install the MCP extra with uv tool install "graphifyy[mcp]" and compare agent answers with and without the graph.
• Upgrade from an older version and time the forced full re-extraction.

Key ideas

Platform environment variables

Secrets a hosting platform stores and injects into your app at runtime, so a platform breach can expose them even if your code is clean.

Upstream first rotation

Issue a new key at the provider and revoke the old one there, because changing the value only on the platform leaves the leaked key valid.

Sealed variables

Railway values that cannot be read back through its UI, API or CLI, though the disclosure does not say whether they were safe.

Steps

1. Read Railway's security disclosure for Oct 21 first, including the later updates.
2. Read GitGuardian's response guide for its order of operations from investigation to cleanup.
3. List which AI provider keys your own apps use and where each is stored.
4. Self check: if an AI API key leaks from your hosting platform, which two actions come first, and why must the first happen at the provider?

Steps

1. Write an inventory of every AI provider key your own projects use, noting where each is stored: local .env files, CI secrets or a hosting platform.
2. Install the scanner with pipx install ggshield and authenticate with ggshield auth login.
3. In each of your own repos, scan the working tree with ggshield secret scan path -r . and the git history with ggshield secret scan repo ..
4. Confirm .env files are listed in .gitignore and that no committed config holds a live key; move any you find into your secret store.
5. Rotate one of your own AI keys end to end: create a new key at the provider, update your config, redeploy, revoke the old key, and note how long it took.
6. Set a spend limit or usage alert on each provider account so a stolen key shows up quickly.

Small angles to try

• If you deploy on Railway, follow GitGuardian's guide to export your own variables with railway variable list --service <service> --environment <environment> --kv > .env.railway, scan the file, then delete it.
• Split one shared AI key into per project, per environment keys so a leak has a smaller blast radius.
• Turn the rotation you just did into a short written runbook.

Key ideas

Bit complexity

Counting elementary bit operations on a multitape Turing machine or Boolean circuit, so results do not depend on machine word size.

FFT based multiplication

Large integers are split into chunks, transformed with a discrete Fourier transform, multiplied pointwise and transformed back.

Formal verification in Lean

Writing a proof in the Lean proof assistant so a computer checks every step, instead of trusting human or model review.

Steps

1. Read the first two pages of Harvey's note first for the claim, its origin and his caveats.
2. Look up a plain introduction to the Schönhage and Strassen FFT method to see where the n log n shape comes from.
3. Return to the note's sections on tensor powers and amplified savings, reading for the idea rather than every detail.
4. Self check: why does a saving of (log n)^ε with ε = 2^-182 matter to theorists but not to any program you will run?

Steps

1. In Python, time multiplication of two random integers built with random.getrandbits at bit lengths from 10 thousand to 10 million, doubling each step.
2. Divide each time by n * math.log2(n) and plot or print the ratio to see how far the built in algorithm is from n log n scaling.
3. Compute math.log2(n) ** (2**-182) for n = 2**64 and observe that the claimed saving factor is indistinguishable from 1 in floating point.
4. Pick one paragraph of Harvey's note, ask any chat model to explain it, then check the explanation against the note's own wording and list where they disagree.
5. Write down what evidence would convince you the result is correct: a Lean proof, independent expert review, or both.

Small angles to try

• Repeat the timing with an arbitrary precision library you already have installed and compare the curves.
• Ask two different models to summarize the note and compare their stated caveats.
• Search for a Lean formalisation effort over the coming weeks and note whether one appears.

Key ideas

Model Garden

Google Cloud's catalog for deploying and calling first party and partner models inside your own project.

Committed spend drawdown

Usage counts against money a company already promised to spend on the cloud, which often decides which vendor a team can use.

Zero Data Retention

A mode where the provider keeps no copy of prompts or outputs after serving the request.

Steps

1. Read the Google Cloud blog post first for model names, preview status and the security list.
2. Read OpenAI's announcement for timing and the Microsoft partnership note.
3. List which controls your team needs before using a model through a cloud: VPC Service Controls, audit logs, retention terms.
4. Self check: what changes for a team on GCP compared with calling OpenAI directly, and what stays the same?

Steps

1. Pick 10 prompts from your own real workload and save them with the answers you would accept.
2. Run them through the OpenAI Responses API on gpt-6-luna and gpt-6.1-sol, logging latency, tokens and cost per prompt.
3. Save this as a small reusable eval script so the same prompts can be rerun on Model Garden once access opens.
4. In your own Google Cloud project, browse Model Garden and Marketplace to see whether the OpenAI listing is visible to your account.
5. Check with whoever owns your cloud contract whether a Google Cloud commitment exists that this usage could draw down, and submit the interest form if it is worth it.

Small angles to try

• Add gpt-6-astra to the eval for your hardest prompts and compare cost per correct answer.
• Write down which Google Cloud controls you would require before switching, such as VPC Service Controls and audit logging.
• Run the same prompts on a model you can already reach in Model Garden to get a baseline.

Key ideas

Decision model

A model that scores a fixed list of answers in one forward pass and returns their probabilities, instead of writing a reply token by token.

Shared prefix caching

The system prompt and state are processed once and cached, so each extra question only costs a pass over its own short suffix.

Calibration

A model is calibrated when an answer it gives 80% really is right about 80% of the time, which makes thresholds for escalation meaningful.

Steps

1. Read the Baseten launch post first for the idea and the benchmark claims.
2. Read the Base Labs technical report section on how option probabilities come from a softmax over the option letter logits.
3. Read section 2.5 of the report on the "I don't know" training rule and its threshold.
4. Compare this with asking a chat model to answer "billing or support" in text and parsing the reply.
5. Self check: why does one shared prefix make ten questions about the same ticket cheap, and what threshold would send a ticket to a human?

Steps

1. Create a Baseten API key and export it as BASETEN_API_KEY.
2. Open the elephant-1 page in the Baseten Model Library and copy its curl example, which POSTs to https://inference.baseten.co/v1/systemone with model decision-models/baseten-elephant-9b-latest.
3. Write 20 short support messages of your own and label each as billing or support by hand.
4. Send each message as the state with one choice question named route whose criteria describe billing and support.
5. Record the probability of the chosen option, then compute accuracy and see whether the wrong answers had lower probabilities.
6. Pick a cutoff such as 0.6 and count how many tickets would go to a human.

Small angles to try

• Swap in mammoth-1 from the Model Library and compare accuracy and latency on the same 20 messages.
• Add a score question for urgency with three ordered levels and check whether the probabilities look sensible.
• Run the same set through a chat model with a text prompt and compare cost per decision.

Key ideas

GUI agent

A model that takes a screenshot and a goal and outputs the next action, such as a tap at given coordinates or text to type.

Image token pruning

Removing image patches that carry no content, like empty background, so the language model has fewer tokens to prefill.

Accelerator fallback

Trying the fastest hardware first and falling back to slower hardware when a self test shows wrong outputs or no speed gain.

Steps

1. Read the Opposable blog post from start to the NPU section, noting each latency number.
2. Skim the llama.cpp multimodal doc to see how a vision encoder and a language model are loaded as two files.
3. Sketch the step loop: screenshot, crop, encode, prune, prefill, decode an action, check the screen.
4. Note why the w8a16 encoder was rejected even though it was faster.
5. Self check: which of encoder, prefill and decode dominates step time on a CPU, and which trick targets each?

Steps

1. Install llama.cpp following the quick start in its GitHub README.
2. Take 5 screenshots of your own phone home screen and settings pages and copy them to the laptop.
3. Start a small vision model with llama-mtmd-cli -hf ggml-org/Qwen2-VL-2B-Instruct-GGUF, adding -c 8192 if the context is too small.
4. For each screenshot, ask where to tap to open a named setting and note the answer and the time per step.
5. Crop the status bar and navigation bar off each image with any image editor and repeat the same questions.
6. Compare the time and the answers before and after cropping.

Small angles to try

• Try ggml-org/Qwen2.5-VL-3B-Instruct-GGUF and compare speed against the 2B model.
• Downscale screenshots to half width and see where answers start to break.
• Ask for one subgoal at a time instead of the whole task and compare accuracy.

Key ideas

Slot filling

The agent keeps asking until it has every field a booking needs, such as date, time, party size and name.

Tool calling against a ledger

The model does not decide availability itself; it calls the reservation system and phrases the result for the caller.

Human in the loop FAQ

The AI drafts answers from the store's site, and staff approve or edit them before callers hear them.

Steps

1. Read the Impress Watch article, using a translator if needed, and list every action the agent can take.
2. Map each action to an API call it would need on a booking system: check, create, change, cancel.
3. Note which parts must be exact, like times and party size, and which can be free text, like the greeting.
4. Think about failure cases: a caller who changes their mind, an unclear date, a full night.
5. Self check: why should availability always come from the ledger and never from the model's own guess?

Steps

1. Create a JSON file with a fake restaurant's tables and time slots for the next three days.
2. Write a small Python text chat loop that asks for date, time, party size and name until all are filled.
3. Add functions to check, book, change and cancel that read and write only that JSON file.
4. When the slot is full, return the two nearest free times and offer them.
5. Add three FAQ answers, hours, address and parking, and route questions that match them.
6. Run ten test conversations of your own, including a cancel and a change, and check the JSON file after each.

Small angles to try

• Put a local LLM in front to turn free speech like "tomorrow at seven for four" into the slot fields.
• Add speech input and output on your own machine with any offline speech library to make it a voice prototype.
• Log every turn and write a one line summary per call, like the staff dashboard.

Key ideas

Connected apps

Per app grants that let the assistant search a given service, such as Gmail or Photos, on your behalf.

Retrieval over personal data

The assistant looks up your own emails, events or photos at question time and answers from them, instead of from its training data.

Opt in scope

Nothing is connected by default; each source is added explicitly and can be removed later.

Steps

1. Read the Google blog post for what works in the car and the limits.
2. Read the Gemini help page on Personal Intelligence for the settings path and the required activity settings.
3. List which of your own Google apps hold data you would or would not want read aloud in a car.
4. Think about how a wrong calendar match could send a driver to the wrong place.
5. Self check: which two actions are needed to make Gemini forget something it learned from a connected app?

Steps

1. Open Gemini on your phone, tap your profile picture, then Personal Intelligence, then Connected Apps.
2. Write down which apps are on and whether Keep Activity is on.
3. With only Calendar connected, ask about your next appointment and note what it found.
4. Turn Calendar off, ask again, and confirm the answer no longer uses your data.
5. Use Tools in a chat to turn Personal Intelligence off for that chat and compare an answer with it on.
6. Decide which apps to keep connected and set them.

Small angles to try

• On the web, check Settings & help, then Personal Intelligence, then Connected Apps, and compare with the phone.
• Ask a question that needs a photo, then disconnect Photos and see how the answer changes.
• Review Gemini Apps Activity to see what was saved from your test chats.

Key ideas

Plan then render

One stage decides where text and elements go, and a second stage draws the pixels, which helps with text heavy layouts.

Async job API

You create a job, get an id, and poll for the result instead of waiting on one long request.

Idempotency key

A unique id sent with the create call so a retry does not start a second, paid job.

Steps

1. Read the Canva developer community post first for the request shape and the limits.
2. Read the TechCrunch article for how the two stages and the template training work.
3. Open the Canva Connect REST reference, Generations section, for the create and get job pages.
4. Note the two error codes, model_not_available and unsafe_input, and how to handle each.
5. Self check: why does a model trained on templates handle a menu or poster better than one trained on photos?

Steps

1. Try Craft in Canva's Dream Lab tab with a prompt for a text heavy poster to see the output first.
2. Create an integration in the Canva developer portal and get a user access token with the asset:write scope.
3. Send POST /v1/image-generations with a prompt, "model": "craft_v1_2k", an aspect ratio and a fresh UUID as idempotency_key.
4. Poll GET /v1/image-generations/{jobId} every few seconds with growing waits until status is success, and time it.
5. Download the image within 24 hours and check whether all the text is spelled correctly.

Small angles to try

• Set asset_upload.type to upload so the image is saved to your Canva uploads instead.
• Generate the same poster with Craft and with the default model and compare text accuracy.
• Send the same idempotency key twice and confirm only one job is created.