In-depth articles on technology shaping what comes next.

AWS Emulator Testing: 5 Myths That Bite in Production

AWS's official emulator is real, but the myths around AWS emulator testing will burn you. Learn where fidelity breaks and what must still run on real AWS.

Two conveyor belts, one pristine and one worn, showing identical boxes with one spilled on the worn side
Green CI against an emulator and green CI against the real thing are not the same green.

The worst integration test failure I've ever dealt with was green. Every check passed in CI, the deploy went out, and twenty minutes later an on-call engineer was staring at a queue full of poison messages because the real SQS behavior differed from the stub in exactly one way nobody had thought to check. So when AWS shipped its own emulator — a Rust workspace with one crate per service, generated from the same Smithy models that drive the real SDKs — my first reaction wasn't excitement. It was: good, now let's find out where it lies. Because every emulator lies somewhere, and AWS emulator testing only works if you know where the lies are.

Myth 1: "Green tests against an AWS emulator mean it works on AWS"

This is the belief that gets people hurt, and the project's own documentation is unusually honest about it. AWS publishes per-service fidelity scores with each release and tracks gaps in a literal issues/things-that-do-not-work/ directory in the repo. That directory is the first thing you should read, before you write a single test. Early adopters have already hit the walls: Step Functions definitions using the Assign field don't work, and Lambda support for Python code is incomplete enough that a workflow which runs fine on MiniStack stalls on the official emulator. None of this is a scandal — it's the normal state of a young emulator. The scandal is treating a green suite as proof of anything beyond "your code works against the emulator's interpretation of the API."

The right mental model is coverage, not truth. The emulator covers the request shapes your SDK sends and returns plausible responses. It does not cover throttling behavior, cross-service propagation delays, or the thousand undocumented quirks of a globally distributed system. This is the same specification-implementation gap that shows up everywhere in infrastructure: the Smithy model tells you what the wire format is, but it can't tell you that real SQS occasionally delivers a message twice within the visibility timeout, or that real IAM propagation takes seconds and your just-created role will fail its first three invocations. Your tests passed because the emulator never had the opportunity to be eventual.

Myth 2: "I can validate IAM policies and security controls locally"

No, and AWS says so in plain language on the emulator's security page. Don't use it to test IAM policies or permissions boundaries, resource-based policies like bucket policies, KMS key policies, cross-account or cross-region access, or any other security control. Some services evaluate IAM for users and roles you create inside the emulator, but not the way AWS does. The emulator never denies a request signed with its own key. A request that AWS would deny can succeed locally — which means the emulator is not just useless for security testing, it's actively misleading. It will hand you a false negative with a straight face.

I've watched teams learn this the expensive way. A developer tightens a bucket policy, runs the local suite, sees green, ships it — and the nightly job against the sandbox account dies because the policy now blocks the replication role. Or the inverse and much worse: the policy is too permissive, everything passes everywhere, and you find out months later during an audit that your "private" bucket was readable by any authenticated principal. Policy logic is one of the places where cloud authentication has genuine blind spots, and no emulator is going to rescue you from them. If you want to verify policy behavior, you need real IAM evaluation. That means a real account — ideally a dedicated sandbox with SCPs locking down what it can reach — and tests that assert denials, not just successes:

import boto3
import pytest
from botocore import UNSIGNED
from botocore.config import Config
from botocore.exceptions import ClientError
def test_private_bucket_denies_anonymous_read():
# Runs only in the nightly job, against a dedicated sandbox account.
s3 = boto3.client("s3")
s3.put_object(Bucket=SANDBOX_BUCKET, Key="probe.txt", Body=b"x")
anon = boto3.client("s3", config=Config(signature_version=UNSIGNED))
with pytest.raises(ClientError) as err:
anon.get_object(Bucket=SANDBOX_BUCKET, Key="probe.txt")
assert err.value.response["Error"]["Code"] == "AccessDenied"

One more thing before you point your whole CI fleet at this tool: telemetry is on by default. AWS says it's minimized and never includes request contents or credentials, and I have no reason to doubt that, but my rule is that dev tooling doesn't phone home unless I told it to. Run it with --telemetry off or set EMULATOR_TELEMETRY=off in your CI environment and move on.

Myth 3: "Wire-protocol fidelity is the hard part"

Iceberg with a small neat tip above water and a vast chaotic mass hidden below the surface
The API shape is the tip. The behavior underneath is where your tests silently diverge.

Getting the wire format right is table stakes, and it's the part AWS was always going to nail. Their SDKs are generated from Smithy models, the emulator is generated from the same models, so every operation, every optional field, every error shape matches the documented API. One HN commenter put it well: spec-driven generation hits every variant of every endpoint, where community emulators historically implement the commonly used subset and approximate the rest. That's a genuine advantage of the official tool.

But wire fidelity is the easy part. The hard part is behavioral fidelity, and nobody has it — not AWS's emulator, not LocalStack, not moto, not Floci. Behavioral fidelity is the stuff that isn't in any model: Does a Lambda cold start change observable timing enough to trip your client's timeout? Does DynamoDB throttling surface as a retriable error at the rate your production traffic actually hits? Does CloudFormation teardown hang for twenty minutes when resources were created in a legal-but-unfortunate order? One practitioner asked whether they could point an entire CDK project at the emulator and test cdk deploy end to end, and my answer is: you can, and you should, but only to catch synthesis and API-shape errors. The deployment dynamics — rollback behavior, resource replacement ordering, drift detection — are exactly the parts an emulator is worst at, because those are emergent properties of the real control plane.

  • Wire fidelity — request/response shapes, error codes, serialization. Emulators are good at this, and the Smithy-generated one should be best in class.
  • State fidelity — CRUD round-trips, listing, filtering, pagination. Mostly good across emulators, with per-service gaps you must check.
  • Behavioral fidelity — consistency timing, throttling, partial failures, resource lifecycle quirks. Nobody has this. Plan accordingly.

Myth 4: "Emulators are drop-in — just swap the endpoint"

The marketing pitch of every emulator is "set one environment variable and go." For a quick smoke test, that's true. The server mode listens on 127.0.0.1:9297 over HTTPS with a self-signed certificate, and pointing the CLI at it looks like this:

export AWS_ENDPOINT_URL=https://localhost:9297
export AWS_ACCESS_KEY_ID=emulator
export AWS_SECRET_ACCESS_KEY=emulator
export AWS_REGION=us-east-1
# self-signed cert: don't disable verification globally, scope it
aws s3 ls --no-verify-ssl

The drop-in story falls apart at test-suite scale. The practical issues people hit within the first week: there's no fast state reset — if your suite needs a clean slate between tests (and moto users are accustomed to exactly that, one decorator and the world is fresh), your current option is restarting the process. That's fine for ten tests and painful for ten thousand. There's also no multi-account or Organizations emulation, which kills any test that models your real account structure. And if you're tempted by the embedded Rust mode — one isolated in-process instance per test, no endpoint or certificates to manage, which is a genuinely nice design — read the build-cost section first. The workspace is one crate per AWS service, a full debug build wants on the order of 100 GB of target/ space, and your first cargo test compiles the emulator alongside your own code. Their advice to share one CARGO_TARGET_DIR, cache target/ in CI, and pin to a release commit is not optional decoration. It's the difference between a five-minute and a fifty-minute CI job.

use aws_emulator_embedded::{AwsEmulatorEmbedded, EmbeddedOptions, Telemetry};
let emulator = AwsEmulatorEmbedded::new(
EmbeddedOptions::default().telemetry(Telemetry::Off),
);
let http_client = emulator.http_client();
// wire http_client into your SDK client config — no endpoint,
// no env vars, one isolated instance per test

Myth 5: "AWS's own emulator kills LocalStack, moto, and the rest"

I heard versions of this within hours of the announcement, and it misunderstands what the incumbents actually sell. LocalStack's value was never merely "we reimplemented the S3 API." It's years of per-service behavioral tuning, state management, testcontainers integration, CI ergonomics, and a community that has already filed the bug you're about to hit. moto's per-test state reset via decorators is a workflow feature, not an API feature, and the AWS emulator doesn't have an answer for it yet. Floci's fans pointed out it covers GCP and Azure too — if your estate is multi-cloud, a single-pane local environment is worth more than a higher-fidelity single-cloud one.

There's also a legitimate trust question about longevity. The first comment on the thread asked the thing everyone was thinking: is this a real product AWS will maintain, or an exercise? The repo accepts issue reports but not pull requests, which is a defensible choice for a spec-driven codebase but also means the community can't patch gaps itself. If the team gets reassigned, you're holding a CC0-licensed snapshot. My position: evaluate it like any dependency with a single maintainer — fine to adopt, reckless to bet the farm on. The same skepticism applies to the breathless "agents wrote this in Rust, the moat is gone" commentary. The moat was never the API surface; it's operational excellence, and emulating the API was always the cheap part. We saw this dynamic play out with distributed SQLite and raft too — copying the interface is weeks of work, matching the failure semantics is years.

What must still run against real AWS infrastructure

A nearly complete suspension bridge whose final span is being load-tested with hanging weights over water
Emulators build most of the bridge. The last span — security and failure behavior — gets load-tested for real.

Here's the split I run on real projects, and it hasn't failed me yet. Everything fast, deterministic, and shape-oriented runs against the emulator on every commit: serialization, error handling for documented error codes, retry logic, state machines that only depend on CRUD round-trips. That's 90% of your suite by test count and maybe 10% by risk.

The remaining 10% runs nightly against a real, isolated sandbox account, and it's the part that catches what actually pages you:

  • IAM and resource policies. Assert explicit denials, not just allowed paths. The emulator structurally cannot do this — it never denies a request signed with its own key.
  • Anything with propagation delay. New roles, DNS changes, KMS grants. If your code creates a resource and immediately uses it, only real AWS will tell you whether you need a retry loop.
  • At-least-once and idempotency behavior. Duplicate SQS deliveries, Lambda retries on timeout, partial batch failures. Emulators are polite; production is not.
  • Infrastructure lifecycle. Full cdk deploy / terraform apply and destroy cycles, especially teardown ordering and rollback paths.
  • Throttling and quota edges. If it matters at production load, it only exists at production API behavior.

The sandbox account needs guardrails of its own: service control policies capping what it can do, budget alerts, and a sweeper that nukes leftover resources, because your nightly job will leak things. That's real operational overhead, and it's exactly why the emulator is valuable — it keeps 90% of your feedback loop fast and free so you can afford to make the last 10% honest.

An emulator's job is to make your feedback loop fast. It is not to make you confident. Confidence still costs real API calls.

The official AWS emulator is a genuinely good tool — spec-generated coverage, an embedded mode with real isolation, and refreshingly honest documentation about what it can't do. Adopt it for what it is: the fastest possible way to be mostly right. Then decide, deliberately and in writing, which tests get to claim the other kind of right — the kind that only comes from infrastructure that can actually deny your request, throttle your client, and deliver your message twice.