AWS S3 403 Signature Mismatch: Pinpoint APM Was the Real Cause

Intro

While building a file upload feature to AWS S3 from a Spring Boot application, I ran into a problem I did not expect. The Access Key and Secret Key were correct, the IAM permissions were set properly — and yet the following error kept firing:

software.amazon.awssdk.services.s3.model.S3Exception: The request signature we calculated does not match the signature you provided.
Check your key and signing method.

This error usually points to wrong credentials or a permission problem. I checked everything and the config looked right. I even created a new IAM user and issued a fresh Access Key — same error.

What is Pinpoint APM?

Before walking through the problem, a short note on Pinpoint. Pinpoint is an open-source APM (Application Performance Monitoring) tool originally built at Naver. It’s used to monitor application performance and trace inter-service calls in large distributed systems.

Key characteristics:

• Distributed transaction tracing — visualize call flow across microservices
• Real-time application monitoring — response time, throughput, error rate
• Code-level visibility — identify bottlenecks down to the method
• Non-invasive install — add a Java Agent, no application code changes

How Pinpoint works

Pinpoint installs into an application via a Java Agent. You set it up on Tomcat startup like this:

-javaagent:/opt/pinpoint-agent/pinpoint-bootstrap.jar
-Dpinpoint.agentId=web-app-01
-Dpinpoint.applicationName=my-application

The important part: once the Pinpoint Agent is attached, every outbound HTTP request from the application gets tracing headers automatically added to it. This happens without the developer noticing — and AWS SDK calls to S3 are not exempt.

Investigation

How Pinpoint injects headers

Pinpoint uses bytecode instrumentation and works roughly like this:

1. Intercepts HTTP client libraries — Apache HttpClient, OkHttp, URLConnection, and others are registered as monitoring targets.
2. Automatically inserts headers — every outbound HTTP request gets tracing headers added.
3. Propagates transaction context — TraceId, SpanId, and similar fields are added so the call graph can be reconstructed across services.

So without any code changes, attaching the Pinpoint Agent on Tomcat means every outbound HTTP request from the app (including AWS S3) gets headers like:

pinpoint-flags: 0
pinpoint-host: s3.ap-northeast-2.amazonaws.com
pinpoint-pappname: myapp.tomcat
pinpoint-papptype: 1010
pinpoint-pspanid: 1234567890123456789
pinpoint-spanid: 9876543210987654321
pinpoint-traceid: myapp^1234567890123^45

This happens transparently, whether the developer asked for it or not.

The problem code

@Service
public class FileService {

    @Value("${aws.s3.bucket-name}")
    private String bucketName;

    @Value("${aws.s3.access-key}")
    private String accessKey;

    @Value("${aws.s3.secret-key}")
    private String secretKey;

    private final S3Client s3Client;

    public FileService() {
        this.s3Client = S3Client.builder()
                .region(Region.AP_NORTHEAST_2)
                .credentialsProvider(StaticCredentialsProvider.create(
                    AwsBasicCredentials.create(accessKey, secretKey)
                ))
                .build();
    }

    public String uploadFile(MultipartFile file) {
        String key = generateKey(file.getOriginalFilename());

        PutObjectRequest request = PutObjectRequest.builder()
                .bucket(bucketName)
                .key(key)
                .contentType(file.getContentType())
                .build();

        s3Client.putObject(request, RequestBody.fromBytes(file.getBytes()));
        return key;
    }

    private String generateKey(String originalFilename) {
        LocalDateTime now = LocalDateTime.now();
        String uuid = UUID.randomUUID().toString();
        return String.format("files/%d/%02d/%02d/%s_%s",
            now.getYear(), now.getMonthValue(), now.getDayOfMonth(), uuid, originalFilename);
    }
}

The clue in the logs

I turned on DEBUG logging in the AWS SDK and found something telling:

DEBUG s.a.awssdk.auth.signer.Aws4Signer - AWS4 Canonical Request: PUT
/files/2025/10/18/abc123-def4-56789-ghij-klmnopqrstuv.png

amz-sdk-invocation-id:abc12345-6789-abcd-efgh-ijklmnopqrst
amz-sdk-request:attempt=1; max=4
content-length:2048
content-type:image/png
host:my-bucket-name.s3.ap-northeast-2.amazonaws.com
pinpoint-flags:0
pinpoint-host:s3.ap-northeast-2.amazonaws.com
pinpoint-pappname:myapp.tomcat
pinpoint-papptype:1010
pinpoint-pspanid:1234567890123456789
pinpoint-spanid:9876543210987654321
pinpoint-traceid:myapp^1234567890123^45
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20251018T120000Z

The key observation: pinpoint-* headers are inside the Canonical Request.

How AWS signing works

AWS Signature Version 4 includes all headers in the signature calculation. Every header on the request feeds into the signature.

1. Build a Canonical Request — normalize HTTP method, URI, query parameters, headers, and payload.
2. Build the String to Sign — combine the algorithm, timestamp, scope, and the hash of the Canonical Request.
3. Compute the signature — HMAC-SHA256 over Secret Key and String to Sign.

The problem was that the custom headers Pinpoint was adding got folded into this process, so the signature my client produced did not match what AWS S3 expected.

What I tried first (all useless)

1. Create a new IAM user — no effect
2. Issue a new Access Key — no effect
3. Reset permissions — no effect
4. Hard-code credentials directly — no effect
5. Downgrade TLS version — no effect

The real fix: disable Pinpoint (or strip its headers)

Once I understood the cause, the fix was straightforward.

Option 1: Disable Pinpoint via JVM options

-Dpinpoint.enable=false
-Dprofiler.enable=false

Option 2: Strip Pinpoint headers at the code level

@Configuration
public class S3Config {

    @Bean
    public S3Client s3Client() {
        return S3Client.builder()
                .region(Region.AP_NORTHEAST_2)
                .credentialsProvider(StaticCredentialsProvider.create(
                    AwsBasicCredentials.create(accessKey, secretKey)
                ))
                .overrideConfiguration(ClientOverrideConfiguration.builder()
                        .addExecutionInterceptor(new RemovePinpointHeadersInterceptor())
                        .build())
                .build();
    }

    public static class RemovePinpointHeadersInterceptor implements ExecutionInterceptor {
        @Override
        public SdkHttpRequest modifyHttpRequest(
                Context.ModifyHttpRequest context,
                ExecutionAttributes executionAttributes) {

            SdkHttpRequest.Builder requestBuilder = context.httpRequest().toBuilder();

            // Remove the Pinpoint headers
            requestBuilder.removeHeader("pinpoint-flags");
            requestBuilder.removeHeader("pinpoint-host");
            requestBuilder.removeHeader("pinpoint-pappname");
            requestBuilder.removeHeader("pinpoint-papptype");
            requestBuilder.removeHeader("pinpoint-pspanid");
            requestBuilder.removeHeader("pinpoint-spanid");
            requestBuilder.removeHeader("pinpoint-traceid");

            return requestBuilder.build();
        }
    }
}

Option 3: application.properties

pinpoint.enable=false
profiler.enable=false

Takeaways

1. APM tools and AWS SDK can quietly collide

APM tools like Pinpoint inject extra headers into HTTP requests to do tracing. It’s easy to miss that those injected headers can collide with AWS’s signature mechanism.

2. Logs matter

Do not stop at the error message. DEBUG-level logs show you what the actual outbound request looks like, and that’s where the real answer usually is.

3. Dig past the obvious cause

A 403 Signature Mismatch makes you think credentials. Sometimes the cause is further down the stack.

4. Dev and prod environments diverge

You may not run APM in dev but suddenly run it in prod — and the bug only shows up there. Know the differences between environments before they bite.

5. Validate APM + cloud service compatibility

Pinpoint isn’t unique here. New Relic, Datadog, AppDynamics, and others can inject HTTP headers the same way. When mixing APM with AWS, GCP, or Azure, verify compatibility early.

Final recommendations

• Validate AWS SDK + APM compatibility up front
• Know that APM-injected headers can affect cloud service authentication
• Consider excluding APM headers for specific services
• Make it a habit to inspect actual request contents via logs
• Keep APM configuration aligned between dev and prod to avoid surprises

Hidden issues show up in places you didn’t expect. Stay open to possibilities that aren’t the obvious one — especially remember that Java Agent-based APM tools can change application behavior without the developer noticing.