Indexing is the process of analyzing and collecting information about your website’s pages so they can be processed more efficiently in the future. The LiteSpeed Cache plugin includes a built-in crawler that can automatically visit your website’s pages and warm the cache, helping visitors receive cached content faster.
Important: The LiteSpeed Cache crawler is not enabled by default. If you would like to use it, please contact support through the 24/7 Helpdesk available in the mybox control panel.
Note: Running the crawler increases server resource usage. For websites with significant traffic or a large number of pages, we recommend using it only when sufficient server resources are available.
Table of Contents
Contents
- Accessing the Crawler Settings
- Delay
- Run Duration
- Interval Between Runs
- Crawl Interval
- Threads
- Timeout
- Server Load Limit
Accessing the Crawler Settings
- Log in to your WordPress administration panel.
- Navigate to:
LiteSpeed Cache → Crawler
- Open the:
General Settings
tab.
This section contains the primary settings used to control the crawler’s behavior.
Delay
The crawler visits pages by sending requests to your website. If these requests are sent too quickly, they can increase server load.
The Delay setting determines how long the crawler waits between requests.
The default value is:
10000 microseconds (0.01 seconds)
Increasing this value reduces server load but also increases the total crawl time.
Run Duration
This setting determines how long the crawler should run before taking a break.
For example:
60 seconds
means the crawler will work continuously for one minute and then pause.
After the pause, it will resume exactly where it stopped until the entire website has been crawled.
Interval Between Runs
This setting determines the length of the pause between crawler sessions.
For example:
120 seconds
means that after each crawl session, the crawler will wait two minutes before continuing.
This helps prevent excessive server load during large crawls.
Crawl Interval
Once the crawler has completed scanning all pages on the website, it stops.
The Crawl Interval setting determines how long the crawler should wait before starting a completely new crawl cycle.
This allows cached content to be refreshed automatically at regular intervals.
Threads
Threads determine how many crawler processes run simultaneously.
Examples:
- 1 thread = one page processed at a time
- 4 threads = four pages processed simultaneously
Using more threads allows the crawler to complete its work faster but increases resource consumption.
For most websites:
2–4 threads
is a reasonable starting point.
Timeout
The Timeout setting specifies how long the crawler should wait for a page to load before considering the request unsuccessful.
The recommended value is:
30 seconds
Increasing this value may help on slower websites but can also prolong crawl times if pages become unresponsive.
Server Load Limit
This setting protects the server from becoming overloaded.
The crawler monitors the current Linux system load and automatically pauses if the specified limit is reached.
General guidelines:
- A load value of 0 indicates an idle server.
- Each active process increases the load.
- When the configured limit is reached, the crawler stops until server load decreases.
This prevents the crawler from consuming all available server resources and affecting normal website operation.
Recommended Starting Configuration
For a typical WordPress website:
| Setting | Recommended Value |
|---|---|
| Delay | 10000 |
| Run Duration | 60 seconds |
| Interval Between Runs | 120 seconds |
| Crawl Interval | Default |
| Threads | 2–4 |
| Timeout | 30 seconds |
| Server Load Limit | Default or according to server capacity |
Summary
The LiteSpeed Cache crawler can improve cache efficiency by automatically visiting pages and generating cached versions before visitors access them. However, because it increases resource usage, it should be configured carefully.
The most important settings to monitor are:
- Delay – controls request frequency
- Threads – controls crawl speed
- Timeout – controls page response limits
- Server Load Limit – prevents excessive resource consumption
A balanced configuration helps keep your cache warm while maintaining stable server performance.