How to increase the memory allocation of the Exploratory Server.

It describes how to increase the memory allocation of the Exploratory Server. If you are getting an error like "Processing might have failed due to the large data size." when you try to use the parameter of published content or schedule it even if you have enough memory on the server, this may fix this problem.

There are 2 places you can consider updating the memory allocation. One is the R runtime and the other one is the Web Server runtime. You can try updating one by one, not both at a time because, In most of cases, the problem comes from either one.

How to increase the memory allocation of the R runtime.

1. Update the config file.

Open the docker-compose.yml in the Exploratory Server installation folder with a text editor such as vi. Find the rserve section, and add the following parameters to the rserve section.

shm_size: Size of the /dev/shm partition.
mem_limit: Maximum memory size
memswap_limit: Maximum memory swap size
mem_reservation: Reserved (minimum) memory size

For details about the parameters, please refer to the Docker documentation. The values of each parameter will vary depending on the system running Docker. If there are no other processes running on the system besides Docker, a rough guideline is to subtract 2GB (needed for other components) from the system's total memory size. For example, if the system has 16GB of memory, this would be approximately 14GB. Below is an example where the memory and swap sizes are set to 14GB each, and the reserved memory size is set to 1GB.

 rserve:
   image: r-exploratory:10.2.1
            :
            :
   shm_size: '14gb'
   mem_limit: '14gb'    
   memswap_limit: '14gb'
   mem_reservation: '1gb'

2. Restart Exploratory Server

Restart the Exploratory Server to make the change effective.

docker-compose down
docker-compose up -d

How to increase the memory allocation of the Web Server runtime.

Please prepare 2 terminal windows connected to the server where the Exploratory Server is installed.

1. Run a Docker container based on the default exploratory Docker image.

On a terminal window, run the following command to start a Docker container based on the default exploratory Docker image. In this example, we use the Docker image from Exploratory Server version 7.0.9. Please replace the version number in the command with the one you are using.

docker run -it exploratory:7.0.9 /bin/bash

2. Update the config file.

2.1 Install a text editor

Since we need to update the configuration file in the next step, we need to install a text editor. Here is an example of a Vim editor installation.

apt-get update
apt-get -y --no-install-recommends install vim

2.2 Update the config file.

Open the config file datablog/pm2_single.json with an editor like the following.

vim datablog/pm2_single.json

Add the --max-old-space-size=4096" parameter to the node_args line like the following. The number 4096 means 4GB in megabytes. If you have larger memory on your server, you can specify the larger value here.

Before:

"node_args": [ "--harmony" ],

After:

"node_args": [ "--harmony", "--max-old-space-size=4096" ],

Save the change to the file.

3. Save the change as a different Docker image.

On another terminal window, run the following command to find the Docker container ID that we are running on the other terminal window.

docker ps

The above command should give the table that looks like the following as the output.

CONTAINER ID        IMAGE                  COMMAND         PORTS       
aad6f8d956a9        exploratory:7.0.9      "/bin/bash"    6311/tcp

Find the Container ID for the Docker container from the table. In this case, it's aad6f8d956a9. If you see multiple lines of output, look for a line that has exploratory: for the IMAGE and /bin/bash for the COMMAND.

Run the following command to save this container with the modification above as a separate Docker image. Here, let's give the new image the customized version number 7.0.9.1.

docker commit  -c 'CMD ["./start.sh"]' -c 'EXPOSE 3000' aad6f8d956a9  exploratory:7.0.9.1

4. Verify that the new Docker image has your changes

Start a Docker container with the new Docker image you just created.

docker run -it exploratory:7.0.9.1 /bin/bash

Show the config file content and make sure the change is there.

cat datablog/pm2_single.json

5. Use the customized exploratory Docker image from the Exploratory Server

Locate the line in the docker-compose.yml file that specifies the Docker image version for the exploratory Docker container and update the version to the one that you just created.

Before:

...
  exploratory:
    image: exploratory:7.0.9
...

After:

...
  exploratory:
    image: exploratory:7.0.9.1
...

6. Restart Exploratory Server

Restart the Exploratory Server to make the change effective.

docker-compose down
docker-compose up -d