🎉 Requestly API Client if now Free Forever & Open Source. Try now.

Header Modification in Automation: Why It Matters and How to do it in requestly

Kanishk Rawat
Learn how to modify headers in automation to handle auth, geolocation, and bot detection, using Requestly for cross-browser support.
Header Modification in Automation

When you’re automating tests or running browser-based workflows, modifying HTTP headers isn’t just a trick, it’s often essential. Whether you’re simulating different user behaviors, bypassing security, or injecting custom values for tracking, header modification can make or break your automation strategy.

In this article, we’ll break down:

  • Why developers modify headers in automation
  • How traditional tools and frameworks handle it
  • Where those tools fall short
  • And most importantly, how to do it cleanly and reliably with Requestly.

Why Header Modification Matters in Automation

1. Application Behavior Control

Modern applications often exhibit different behaviors based on header content. Many organizations maintain multiple sets of backend APIs, production, staging, and mock environments and route traffic based on specific headers. For instance, a custom header like X-Environment: mock might redirect requests to a mocked API endpoint instead of the live production system, enabling safe testing without affecting real data.

2. Authentication Requirements

Basic authentication remains a cornerstone of API security, but implementing it in automation can be tricky. Safari, notably, doesn’t support basic authentication out of the box, making header modification essential for cross-browser testing. By automatically injecting authentication headers, you can ensure your tests run consistently across all browsers without manual intervention.

3. Bypassing Bot Detection

One of the most practical applications of header modification is bypassing bot detection systems. Modern websites employ sophisticated anti-bot measures like Captcha and Akamai Bot Manager. By adding specific headers to bot bypass lists, automated tests can run smoothly without triggering these protective mechanisms that would otherwise halt your test execution.

4. Geolocation Testing

Testing location-specific content traditionally required complex proxy setups or VPN configurations. Header-based geolocation mocking offers a simpler alternative. By injecting location headers, you can test how your application behaves for users in different geographical regions without the overhead of managing multiple proxy servers.

5. Whitelisting and Access Control

Similar to IP whitelisting, header-based whitelisting provides a flexible access control mechanism. This approach is particularly valuable when combined with IP geolocation, as it eliminates the need to maintain extensive proxy lists while still providing the testing flexibility you need.

6. Analytics and Tracking Separation

Organizations using automation for production validation face a common problem: automated sessions contaminating real user analytics. Custom headers provide a clean solution for filtering out automated traffic from your analytics data, ensuring your metrics accurately reflect genuine user behavior.

The Challenge with Native Framework Solutions

Selenium Limitations

While Selenium offers Chrome DevTools Protocol (CDP) support for header modification, this solution comes with significant constraints:

  • Browser Limitation: Works only with Chromium-based browsers, leaving Firefox and Safari unsupported
  • Static Headers: Cannot dynamically change headers per request
  • No Response Modification: Response headers remain untouchable

Playwright's Approach

Playwright’s page.setExtraHTTPHeaders() method provides broader browser support but still falls short:

  • Navigation-Only: Headers apply only before navigation, not during mid-test scenarios
  • Complex Dynamic Modification: Request-specific header manipulation requires advanced workarounds

Cypress Constraints

Cypress takes a different approach with cy.request() and cy.intercept(), offering request-level control but with notable gaps:

  • No Safari Support: Limited browser compatibility
  • Navigation Headers: No control over page navigation headers
  • Response Limitations: Cannot modify response headers

Implementing Header Modification with Requestly

1. Install Chrome for Testing

Starting with Chrome 137, official Chrome builds no longer support the --load-extension flag due to security concerns. There commit →
To load extensions in automation tools (Selenium, Playwright, Puppeteer, etc.), you must use Chrome for Testing or Chromium.
Regular Chrome will block extension loading via automation. Use Chrome for Testing to avoid breakages and ensure compatibility.

Download Chrome for Testing (official Chromium build) from:
https://googlechromelabs.github.io/chrome-for-testing/

2. Install Selenium WebDriver

We also offer an NPM package, @requestly/rq-automation, to make this easier. Learn more →

npm install @requestly/rq-automation

In your project directory, install the Selenium WebDriver package:

				
					npm install selenium-webdriver
				
			

3. Download the Requestly CRX Extension

If you're using our NPM package, you don’t need to download it. Simply call the function getExtension().

Make sure you have the .crx file for the Requestly Automation Extension.

4. Configure Chrome Options

Use the Selenium chrome.Options() object to:

  • Set Chrome binary path (from Chrome for Testing)
  • Add the .crx extension
				
					const chrome = require("selenium-webdriver/chrome");
const path = require("path");
// For Requestly NPM package ( @requestly/rq-automation )
const { getExtension } = require("@requestly/rq-automation");

const options = new chrome.Options();
options.setChromeBinaryPath(
  "/Path/to/Chrome/Binary"
);

// Add Requestly CRX extension

// Without Requestly NPM package
options.addExtensions(path.resolve(__dirname, "requestly.crx"));

// With Requestly NPM package ( @requestly/rq-automation )
options.addExtensions(getExtension("crx"));
				
			

5. Build the WebDriver

				
					const { Builder } = require("selenium-webdriver");

const driver = await new Builder()
  .forBrowser("chrome")
  .setChromeOptions(options)
  .build();
				
			

6. Close Extension Welcome Page

When Chrome installs the Requestly extension, it may open a “welcome” (post-install) tab that interrupts automation. You can close this tab programmatically:

				
					  // give Chrome a moment for any stray tabs
await driver.sleep(500);

// close any “welcome” tab if still present
const handles = await driver.getAllWindowHandles();
const mainHandle = handles[0];
for (const handle of handles.slice(1)) {
  await driver.switchTo().window(handle);
  await driver.close();
}
await driver.switchTo().window(mainHandle);

// With Requestly NPM package ( @requestly/rq-automation )
import { closeWelcomePage } = require("@requestly/rq-automation");
await closeWelcomePage(driver);

				
			

6. Perform Header Modifications

To add or remove headers via Requestly automation extension, visit the automation URL with the right query parameters:

Add Request Header:

				
					// Without Requestly NPM package
await driver.get("https://app.requestly.io/automation/add-request-header?headerName=headerValue");

// With Requestly NPM package ( @requestly/rq-automation )
await driver.get(addRequestHeaderUrl("headerName", "headerValue"));

				
			

Add Response Header:

				
					// Without Requestly NPM package
await driver.get("https://app.requestly.io/automation/add-response-header?headerName=headerValue");

// With Requestly NPM package ( @requestly/rq-automation )
await driver.get(addResponseHeaderUrl("headerName", "headerValue"));

				
			

Remove Request Header:

				
					// Without Requestly NPM package
await driver.get("https://app.requestly.io/automation/remove-request-header?headerName");

// With Requestly NPM package ( @requestly/rq-automation )
await driver.get(removeRequestHeaderUrl("headerName"));

				
			

Remove Response Header:

				
					// Without Requestly NPM package
await driver.get("https://app.requestly.io/automation/remove-response-header?headerName");

// With Requestly NPM package ( @requestly/rq-automation )
await driver.get(removeResponseHeaderUrl("headerName"));


				
			

7. Test It on a Live Site

Navigate to testheaders.com to test your header modification

				
					await driver.get("https://testheaders.com");
				
			

Example Without NPM package

				
					const { Builder, By, until } = require("selenium-webdriver");
const chrome = require("selenium-webdriver/chrome");
const path = require("path");

(async () => {
  const options = new chrome.Options();

  options.setChromeBinaryPath("/Path/to/chrome-for-testing");
  options.addExtensions(path.resolve(__dirname, "requestly.crx"));

  const driver = await new Builder()
    .forBrowser("chrome")
    .setChromeOptions(options)
    .build();
  
  // give Chrome a moment for any stray tabs
  await driver.sleep(500);

  // close any “welcome” tab if still present
  const handles = await driver.getAllWindowHandles();
  const mainHandle = handles[0];
  for (const handle of handles.slice(1)) {
    await driver.switchTo().window(handle);
    await driver.close();
  }
  await driver.switchTo().window(mainHandle)

  try {
    await driver.get("https://app.requestly.io/automation/add-request-header?x-testing=selenium");

    await driver.wait(
      until.elementTextContains(
        driver.findElement(By.tagName("body")),
        "Success"
      ),
      1000
    );

    await driver.get("https://testheaders.com");

    await driver.findElement(By.css("button")).click();

    await driver.wait(
      until.elementTextContains(
        driver.findElement(By.tagName("body")),
        "x-testing"
      ),
      5000
    );

    console.log("Header added and verified!");

  } catch (err) {
    console.error("Error during automation:", err);
  } finally {
    await driver.quit();
  }
})();

				
			

Example With NPM package

				
					const {
  addRequestHeaderUrl,
  addRequestHeadersUrl,
  removeRequestHeaderUrl,
  removeRequestHeadersUrl,
  addResponseHeaderUrl,
  addResponseHeadersUrl,
  removeResponseHeaderUrl,
  removeResponseHeadersUrl,
  importRules,
  closeWelcomePage,
} = require("@requestly/rq-automation");
const { Builder } = require("selenium-webdriver");
const chrome = require("selenium-webdriver/chrome");
const { getExtension } = require("@requestly/rq-automation");

async function setupDriver() {
  const options = new chrome.Options();

  // Load Requestly extension
  const extensionPath = getExtension("unpacked");
  options.addArguments(`--load-extension=${extensionPath}`);
  options.addArguments(`--disable-extensions-except=${extensionPath}`);

  const driver = await new Builder()
    .forBrowser("chrome")
    .setChromeOptions(options)
    .build();

  return driver;
}

async function seleniumExample() {
  const driver = await setupDriver();
  await closeWelcomePage(driver);

  // Add a single request header
  await driver.get(addRequestHeaderUrl("X-Custom-Header", "MyValue"));

  // Add multiple request headers
  await driver.get(
    addRequestHeadersUrl({
      Authorization: "Bearer token123",
      "X-Another-Header": "AnotherValue",
    })
  );

  // Remove a request header
  await driver.get(removeRequestHeaderUrl("X-Another-Header"));

  // Remove multiple request headers
  await driver.get(
    removeRequestHeadersUrl(["Authorization", "X-Custom-Header"])
  );

  // Add a response header
  await driver.get(addResponseHeaderUrl("Access-Control-Allow-Origin", "*"));

  // Add multiple response headers
  await driver.get(
    addResponseHeadersUrl({
      "X-Response-Header": "ResponseValue",
      "X-Another-Response-Header": "AnotherResponseValue",
    })
  );

  // Remove a response header
  await driver.get(removeResponseHeaderUrl("X-Another-Response-Header"));

  // Remove multiple response headers
  await driver.get(
    removeResponseHeadersUrl(["X-Response-Header", "X-Another-Response-Header"])
  );

  // Import a shared list of rules
  await driver.get(importRules("YOUR_API_KEY"));

  // Your test code here
  await driver.get("https://example.com");

  await driver.quit();
}

seleniumExample();

				
			

Conclusion

Header modification in automation is no longer a nice-to-have feature, it’s essential for comprehensive testing in modern web applications. While native framework solutions provide basic functionality, they often fall short of the flexibility and reliability required for robust test automation.

Requestly bridges this gap by offering a comprehensive, cross-browser solution that integrates seamlessly with existing test frameworks. Whether you’re dealing with authentication challenges, bot detection, geographic testing, or complex application routing, Requestly provides the tools you need to create reliable, maintainable automated tests.

Written by
Kanishk Rawat
Kanishk Rawat, a tech enthusiast since childhood, has mastered programming through dedication. Whether solo or in a team, he thrives on challenges, crafting innovative solutions .

Related posts