The Challenges of API Logging

APIs

APIs are everywhere, powering all of the modern internet from ordering food to travel platforms, connected cars, home systems such as water heating, and more. 

APIs can also run almost anywhere, including any type of compute platforms and network infrastructure services on AWS. An additional challenge is that all log to different destinations. 

In this blog, we’ll go over the different types of compute platforms, network infrastructure services, and how they relate to your APIs and API security.

Compute Platforms

There are many types of compute platforms, each with unique default logging destinations.

Within AWS, there is…

• Lambda which consists of serverless functions and logs to CloudWatch Lambda logs..
• AppSync, which is an AWS-native graphQL API service and logs to CloudWatch and/or CloudTrail.
•  EC2 which consists of virtual machines and logs to CloudWatch and/or VPC flow logs.

An example of a VPC flow log looks like this:

3 eni-33333333333333333 123456789010 vpc-abcdefab012345678 subnet-22222222bbbbbbbbb i-01234567890123456 10.20.33.164 10.40.2.236 39812 80 6 3 IPv4 10.20.33.164 10.40.2.236 ACCEPT OK ... 3 eni-33333333333333333 123456789010 vpc-abcdefab012345678 subnet-22222222bbbbbbbbb i-01234567890123456 10.40.2.236 10.20.33.164 80 39812 6 19 IPv4 10.40.2.236 10.20.33.164 ACCEPT OK

Container Platforms

There are also container platforms such as…

• ECS, which logs to CloudTrail.
• EKS which logs to CloudWatch.
• Fargate which logs to Fargate logs.

Network Infrastructure Services

In addition to compute platforms, you can run APIs behind many types of network infrastructure services, and these also have unique default logging destinations.

• API Gateways log to CloudWatch.
• WAFs (or Web Application Firewalls) log to AWS WAF logs.

An example of a Network Address Translation (NAT) Gateway log flow could look like this:

- eni-1235b8ca123456789 10.0.0.220 203.0.113.5 10.0.0.220 203.0.113.5

- eni-1235b8ca123456789 203.0.113.5 10.0.0.220 203.0.113.5 10.0.0.220

And a Transit Gateway could look like this:

3 eni-33333333333333333 123456789010 vpc-abcdefab012345678 subnet-22222222bbbbbbbbb i-01234567890123456 10.20.33.164 10.40.2.236 39812 80 6 3 IPv4 10.20.33.164 10.40.2.236 ACCEPT OK

...

3 eni-33333333333333333 123456789010 vpc-abcdefab012345678 subnet-22222222bbbbbbbbb i-01234567890123456 10.40.2.236 10.20.33.164 80 39812 6 19 IPv4 10.40.2.236 10.20.33.164 ACCEPT OK

In addition to these infrastructures, there are the load balancers, such as ELB which logs to classic access logs in an S3 bucket, ALB which logs to access logs, and NLB which logs to NLB logs, parallel to VPC flow logs.

Application logs

Only application logs can truly get inside an API to be able to see everything. These logs give developers superior visibility, however, there are several challenges that make it difficult to implement application logs.

The main issue is that in order to collect application-level log data, there must be a way to embed the logging function (log shipper, log extractor) within the API itself. However, there are many possible ways to achieve this. FireTail makes it easy with our open-source code libraries. 

Not all logs are created equal

Many log file types have different levels of visibility which lend them different advantages and limitations.

The chart below shows different data elements that are visible in different log types or log file formats. For the purposes of examining API calls for threats and risks (see the FireTail API data breach tracker and State of API Security 2024 to understand the most common threats and breach vectors), there are 3 key elements that should be captured in log files:

1. Authentication 
2. Authorization
3. Data sent to and from the API

From that, we can clearly see that only application logs give a complete view of all the necessary data for either forensic analysis (single log / transaction / user / visit / API interaction) or to look for systematic breaches, such as scraping data from an API.

We can also see that logs come in many shapes and sizes, and go to many different locations. Therefore, centralizing logs for all API traffic, even just on one CSP like AWS, can be extremely challenging.

Takeaways

Logging is an essential part of the API security posture management process, however with all the different types of containers and network services logging to different locations, it is difficult for developers and security teams to stay on top of their logging. That’s where FireTail comes in.

Although application logs are the best, they are not always easy to implement. However, there are good solutions available, such as using a platform like FireTail that provides centralized, unified logging. This gives you a single log stream to analyze and write detection queries against. 

Recommendations

For optimal API security, it is crucial to maintain maximum visibility into your API landscape. After all, if you can’t see it, you can’t secure it. But beyond mere visibility, developers need a deeper understanding of their APIs and interactions.

Security teams need to be aware of the full request parameters, payloads, and different attack types such as manipulation of business logic or data scraping. 

They also need comprehension of the response payloads for forensic purposes, to examine the scope and scale of a data breach. This forensic analysis has real-world impact for regulatory reporting requirements and paying the costs of a data breach to identity and credit monitoring services, et cetera.

FireTail can help solve the problem of “log sprawl” by providing a consistent, central logging point. Simply put, FireTail normalizes API logs across all compute platform types, all network infrastructure types and all CSPs.

To see how FireTail can help you gain better visibility and simplify your APIs for better understanding, schedule a demo here or try our free tier out for yourself today!