Header Modification in Automation: Why It Matters and How to do it in requestly

When you’re automating tests or running browser-based workflows, modifying HTTP headers isn’t just a trick, it’s often essential. Whether you’re simulating different user behaviors, bypassing security, or injecting custom values for tracking, header modification can make or break your automation strategy.
In this article, we’ll break down:
- Why developers modify headers in automation
- How traditional tools and frameworks handle it
- Where those tools fall short
- And most importantly, how to do it cleanly and reliably with Requestly.
Why Header Modification Matters in Automation
1. Application Behavior Control
Modern applications often exhibit different behaviors based on header content. Many organizations maintain multiple sets of backend APIs, production, staging, and mock environments and route traffic based on specific headers. For instance, a custom header like X-Environment: mock might redirect requests to a mocked API endpoint instead of the live production system, enabling safe testing without affecting real data.
2. Authentication Requirements
Basic authentication remains a cornerstone of API security, but implementing it in automation can be tricky. Safari, notably, doesn’t support basic authentication out of the box, making header modification essential for cross-browser testing. By automatically injecting authentication headers, you can ensure your tests run consistently across all browsers without manual intervention.
3. Bypassing Bot Detection
One of the most practical applications of header modification is bypassing bot detection systems. Modern websites employ sophisticated anti-bot measures like Captcha and Akamai Bot Manager. By adding specific headers to bot bypass lists, automated tests can run smoothly without triggering these protective mechanisms that would otherwise halt your test execution.
4. Geolocation Testing
Testing location-specific content traditionally required complex proxy setups or VPN configurations. Header-based geolocation mocking offers a simpler alternative. By injecting location headers, you can test how your application behaves for users in different geographical regions without the overhead of managing multiple proxy servers.
5. Whitelisting and Access Control
Similar to IP whitelisting, header-based whitelisting provides a flexible access control mechanism. This approach is particularly valuable when combined with IP geolocation, as it eliminates the need to maintain extensive proxy lists while still providing the testing flexibility you need.
6. Analytics and Tracking Separation
Organizations using automation for production validation face a common problem: automated sessions contaminating real user analytics. Custom headers provide a clean solution for filtering out automated traffic from your analytics data, ensuring your metrics accurately reflect genuine user behavior.
The Challenge with Native Framework Solutions
Selenium Limitations
While Selenium offers Chrome DevTools Protocol (CDP) support for header modification, this solution comes with significant constraints:
- Browser Limitation: Works only with Chromium-based browsers, leaving Firefox and Safari unsupported
- Static Headers: Cannot dynamically change headers per request
- No Response Modification: Response headers remain untouchable
Playwright's Approach
Playwright’s page.setExtraHTTPHeaders()
method provides broader browser support but still falls short:
- Navigation-Only: Headers apply only before navigation, not during mid-test scenarios
- Complex Dynamic Modification: Request-specific header manipulation requires advanced workarounds
Cypress Constraints
Cypress takes a different approach with cy.request()
and cy.intercept()
, offering request-level control but with notable gaps:
- No Safari Support: Limited browser compatibility
- Navigation Headers: No control over page navigation headers
- Response Limitations: Cannot modify response headers
Implementing Header Modification with Requestly
1. Install Chrome for Testing
Starting with Chrome 137, official Chrome builds no longer support the --load-extension
flag due to security concerns. There commit →
To load extensions in automation tools (Selenium, Playwright, Puppeteer, etc.), you must use Chrome for Testing or Chromium.
Regular Chrome will block extension loading via automation. Use Chrome for Testing to avoid breakages and ensure compatibility.
Download Chrome for Testing (official Chromium build) from:
https://googlechromelabs.github.io/chrome-for-testing/
2. Install Selenium WebDriver
npm install @requestly/rq-automation
In your project directory, install the Selenium WebDriver package:
npm install selenium-webdriver
3. Download the Requestly CRX Extension
getExtension()
. Make sure you have the .crx file for the Requestly Automation Extension.
4. Configure Chrome Options
Use the Selenium chrome.Options()
object to:
- Set Chrome binary path (from Chrome for Testing)
- Add the
.crx
extension
const chrome = require("selenium-webdriver/chrome");
const path = require("path");
// For Requestly NPM package ( @requestly/rq-automation )
const { getExtension } = require("@requestly/rq-automation");
const options = new chrome.Options();
options.setChromeBinaryPath(
"/Path/to/Chrome/Binary"
);
// Add Requestly CRX extension
// Without Requestly NPM package
options.addExtensions(path.resolve(__dirname, "requestly.crx"));
// With Requestly NPM package ( @requestly/rq-automation )
options.addExtensions(getExtension("crx"));
5. Build the WebDriver
const { Builder } = require("selenium-webdriver");
const driver = await new Builder()
.forBrowser("chrome")
.setChromeOptions(options)
.build();
6. Close Extension Welcome Page
When Chrome installs the Requestly extension, it may open a “welcome” (post-install) tab that interrupts automation. You can close this tab programmatically:
// give Chrome a moment for any stray tabs
await driver.sleep(500);
// close any “welcome” tab if still present
const handles = await driver.getAllWindowHandles();
const mainHandle = handles[0];
for (const handle of handles.slice(1)) {
await driver.switchTo().window(handle);
await driver.close();
}
await driver.switchTo().window(mainHandle);
// With Requestly NPM package ( @requestly/rq-automation )
import { closeWelcomePage } = require("@requestly/rq-automation");
await closeWelcomePage(driver);
6. Perform Header Modifications
To add or remove headers via Requestly automation extension, visit the automation URL with the right query parameters:
Add Request Header:
// Without Requestly NPM package
await driver.get("https://app.requestly.io/automation/add-request-header?headerName=headerValue");
// With Requestly NPM package ( @requestly/rq-automation )
await driver.get(addRequestHeaderUrl("headerName", "headerValue"));
Add Response Header:
// Without Requestly NPM package
await driver.get("https://app.requestly.io/automation/add-response-header?headerName=headerValue");
// With Requestly NPM package ( @requestly/rq-automation )
await driver.get(addResponseHeaderUrl("headerName", "headerValue"));
Remove Request Header:
// Without Requestly NPM package
await driver.get("https://app.requestly.io/automation/remove-request-header?headerName");
// With Requestly NPM package ( @requestly/rq-automation )
await driver.get(removeRequestHeaderUrl("headerName"));
Remove Response Header:
// Without Requestly NPM package
await driver.get("https://app.requestly.io/automation/remove-response-header?headerName");
// With Requestly NPM package ( @requestly/rq-automation )
await driver.get(removeResponseHeaderUrl("headerName"));
7. Test It on a Live Site
Navigate to testheaders.com to test your header modification
await driver.get("https://testheaders.com");
Example Without NPM package
const { Builder, By, until } = require("selenium-webdriver");
const chrome = require("selenium-webdriver/chrome");
const path = require("path");
(async () => {
const options = new chrome.Options();
options.setChromeBinaryPath("/Path/to/chrome-for-testing");
options.addExtensions(path.resolve(__dirname, "requestly.crx"));
const driver = await new Builder()
.forBrowser("chrome")
.setChromeOptions(options)
.build();
// give Chrome a moment for any stray tabs
await driver.sleep(500);
// close any “welcome” tab if still present
const handles = await driver.getAllWindowHandles();
const mainHandle = handles[0];
for (const handle of handles.slice(1)) {
await driver.switchTo().window(handle);
await driver.close();
}
await driver.switchTo().window(mainHandle)
try {
await driver.get("https://app.requestly.io/automation/add-request-header?x-testing=selenium");
await driver.wait(
until.elementTextContains(
driver.findElement(By.tagName("body")),
"Success"
),
1000
);
await driver.get("https://testheaders.com");
await driver.findElement(By.css("button")).click();
await driver.wait(
until.elementTextContains(
driver.findElement(By.tagName("body")),
"x-testing"
),
5000
);
console.log("Header added and verified!");
} catch (err) {
console.error("Error during automation:", err);
} finally {
await driver.quit();
}
})();
Example With NPM package
const {
addRequestHeaderUrl,
addRequestHeadersUrl,
removeRequestHeaderUrl,
removeRequestHeadersUrl,
addResponseHeaderUrl,
addResponseHeadersUrl,
removeResponseHeaderUrl,
removeResponseHeadersUrl,
importRules,
closeWelcomePage,
} = require("@requestly/rq-automation");
const { Builder } = require("selenium-webdriver");
const chrome = require("selenium-webdriver/chrome");
const { getExtension } = require("@requestly/rq-automation");
async function setupDriver() {
const options = new chrome.Options();
// Load Requestly extension
const extensionPath = getExtension("unpacked");
options.addArguments(`--load-extension=${extensionPath}`);
options.addArguments(`--disable-extensions-except=${extensionPath}`);
const driver = await new Builder()
.forBrowser("chrome")
.setChromeOptions(options)
.build();
return driver;
}
async function seleniumExample() {
const driver = await setupDriver();
await closeWelcomePage(driver);
// Add a single request header
await driver.get(addRequestHeaderUrl("X-Custom-Header", "MyValue"));
// Add multiple request headers
await driver.get(
addRequestHeadersUrl({
Authorization: "Bearer token123",
"X-Another-Header": "AnotherValue",
})
);
// Remove a request header
await driver.get(removeRequestHeaderUrl("X-Another-Header"));
// Remove multiple request headers
await driver.get(
removeRequestHeadersUrl(["Authorization", "X-Custom-Header"])
);
// Add a response header
await driver.get(addResponseHeaderUrl("Access-Control-Allow-Origin", "*"));
// Add multiple response headers
await driver.get(
addResponseHeadersUrl({
"X-Response-Header": "ResponseValue",
"X-Another-Response-Header": "AnotherResponseValue",
})
);
// Remove a response header
await driver.get(removeResponseHeaderUrl("X-Another-Response-Header"));
// Remove multiple response headers
await driver.get(
removeResponseHeadersUrl(["X-Response-Header", "X-Another-Response-Header"])
);
// Import a shared list of rules
await driver.get(importRules("YOUR_API_KEY"));
// Your test code here
await driver.get("https://example.com");
await driver.quit();
}
seleniumExample();
Conclusion
Header modification in automation is no longer a nice-to-have feature, it’s essential for comprehensive testing in modern web applications. While native framework solutions provide basic functionality, they often fall short of the flexibility and reliability required for robust test automation.
Requestly bridges this gap by offering a comprehensive, cross-browser solution that integrates seamlessly with existing test frameworks. Whether you’re dealing with authentication challenges, bot detection, geographic testing, or complex application routing, Requestly provides the tools you need to create reliable, maintainable automated tests.
Contents
- Why Header Modification Matters in Automation
- 1. Application Behavior Control
- 2. Authentication Requirements
- 3. Bypassing Bot Detection
- 4. Geolocation Testing
- 5. Whitelisting and Access Control
- 6. Analytics and Tracking Separation
- The Challenge with Native Framework Solutions
- Selenium Limitations
- Playwright's Approach
- Cypress Constraints
- Implementing Header Modification with Requestly
- 1. Install Chrome for Testing
- 2. Install Selenium WebDriver
- 3. Download the Requestly CRX Extension
- 4. Configure Chrome Options
- 5. Build the WebDriver
- 6. Close Extension Welcome Page
- 6. Close Extension Welcome Page
- 6. Perform Header Modifications
- Add Request Header:
- Add Response Header:
- Remove Request Header:
- Remove Response Header:
- 7. Test It on a Live Site
- Example Without NPM package
- Example With NPM package
- Conclusion
Subscribe for latest updates
Share this article
Related posts


