Fix API Rate Limit Exhaustion in Gatsby

Gatsby builds often fail due to API rate limit exhaustion when source plugins or `gatsby-node.js` logic trigger massive concurrent requests to external REST/GraphQL endpoints. If you're using `Promise.all()` without a concurrency limit, you're essentially self-DoS-ing your own build pipeline and risking a permanent IP ban from your data providers.

The Vulnerable Pattern

const axios = require('axios');
exports.sourceNodes = async ({ actions }) => {
const { data: nodes } = await axios.get(‘https://api.external-service.com/v1/nodes’);
// VULNERABLE: Firing hundreds of requests simultaneously
// This will trigger a 429 Too Many Requests error on large datasets
const detailPromises = nodes.map(node =>
axios.get(https://api.external-service.com/v1/nodes/${node.id})
);
await Promise.all(detailPromises);
};

The Secure Implementation

The vulnerable code utilizes an unconstrained `Promise.all`, which attempts to open as many network sockets as there are items in the array. This burst behavior is a red flag for WAFs and API gateways. The secure implementation uses `p-throttle` (or `p-limit`) to wrap the asynchronous calls. This enforces a predictable request cadence, ensuring the build process stays under the provider's threshold. For Gatsby Functions (client-side), you should also implement middleware like `express-rate-limit` to prevent users from exhausting your API keys via the proxy.

const axios = require('axios');
const pThrottle = require('p-throttle');
// Define a throttle: 5 requests per 1000ms
const throttle = pThrottle({
limit: 5,
interval: 1000
});
const throttledGet = throttle(url => axios.get(url));
exports.sourceNodes = async ({ actions }) => {
const { data: nodes } = await axios.get(‘https://api.external-service.com/v1/nodes’);
// SECURE: Requests are queued and executed within rate limits
const detailPromises = nodes.map(node =>
throttledGet(https://api.external-service.com/v1/nodes/${node.id})
);
await Promise.all(detailPromises);
};