Use a Local File Server for Bulk Operations

On this page

If you need a location to store files for the IMPORT process, but do not have access to (or cannot use) cloud storage providers, you can run a local file server. You can then use this file server by leveraging support for our HTTP Export Storage API.

This is especially useful for:

• Implementing a compatibility layer in front of custom or proprietary storage providers for which CockroachDB does not yet have built-in support
• Using on-premises storage

HTTP export storage API

A CockroachDB IMPORT process that requires reading or writing external files can use the HTTP Export Storage API by prefacing the address with http, e.g., http://fileserver/mnt/cockroach-exports.

This API uses the GET, PUT and DELETE methods. This behaves like you would expect typical HTTP requests to work. After a PUT request to some path, a subsequent GET request should return the content sent in the PUT request body, at least until a DELETE request is received for that path.

Examples

You can use any file server software that supports GET, PUT and DELETE methods, but we've included code samples for common ones:

• Using PHP with IMPORT
• Using Python with IMPORT
• Using Ruby with IMPORT
• Using nginx as a file server

Using PHP with IMPORT

The PHP language has an HTTP server built in. You can serve local files using the commands below. For more information about how to import these locally served files, see the documentation for the IMPORT statement.

$ cd /path/to/data
$ php -S 127.0.0.1:3000 # files available at e.g., 'http://localhost:3000/data.sql'

Using Python with IMPORT

The Python language has an HTTP server included in the standard library. You can serve local files using the commands below. For more information about how to import these locally served files, see the documentation for the IMPORT statement.

$ cd /path/to/data
$ python -m SimpleHTTPServer 3000 # files available at e.g., 'http://localhost:3000/data.sql'

If you use Python 3, try:

$ cd /path/to/data
$ python -m http.server 3000

Using Ruby with IMPORT

The Ruby language has an HTTP server included in the standard library. You can serve local files using the commands below. For more information about how to import these locally served files, see the documentation for the IMPORT statement.

$ cd /path/to/data
$ ruby -run -ehttpd . -p3000 # files available at e.g., 'http://localhost:3000/data.sql'

Using nginx as a file server

1. Install nginx with the webdav module (often included in -full or similarly named packages in various distributions).

2. In the nginx.conf file, add a dav_methods PUT DELETE directive. For example:

   events {
       worker_connections  1024;
   }
   http {
     server {
       listen 20150;
       location / {
         dav_methods  PUT DELETE;
         root /mnt/cockroach-exports;
         sendfile           on;
         sendfile_max_chunk 1m;
       }
     }
   }
   

See also

• IMPORT
• Use Cloud Storage for Bulk Operations

Was this helpful?